AI Benchmark Digest — 2026-07-12

        July 12, 2026

AI Benchmark Digest — 2026-07-12

AI Benchmark Digest — 2026-07-12
View on AI Benchmark Hub
Daily
New Benchmarks (1)

OpenCompass Language - Chat (Score (%)): Doubao-Seed-2-0-Pro-260215 (high) leads with 91.7 across 23 models.

New Scores From Top-10 Models (10)

Claude Fable 5 on Lean AI Formalization Leaderboard: 4.0 Solved Problems (#13/35)
Claude Fable 5 on TaxCalcBench: 26.0 Correct Returns - Strict (%) (#5/9)
Claude Fable 5 on TrackingAI IQ Test (Mensa Norway): 60.0 Mensa Norway Score (%) (#20/31)
Claude Fable 5 on TrackingAI IQ Test (Vision): 78.57 IQ Test Score (%) (#3/19)
Claude Opus 4.8 on TaxCalcBench: 16.0 Correct Returns - Strict (%) (#7/9)
GPT-5.6 Sol on Design Arena (3D): 1370.0 Elo (#2/125)
GPT-5.6 Sol on Design Arena (Game Dev): 1378.0 Elo (#2/138)
GPT-5.6 Sol on TaxCalcBench: 26.0 Correct Returns - Strict (%) (#4/9)
GPT-5.6 Sol on VoxelBench: 2306.0 Rating (#1/45)
Seed 2.1 Pro on BabyVision: 73.7 Score (%) (#4/24)

New #1 Leaders (6)

BabyVision (Score (%)): GPT-5.5 (83.6) beat Seed 2.0 Pro (High) (60.6) by 23.0.
Design Arena (UI Components) (Elo): GPT-5.6 Sol (1413.0) beat Claude Fable 5 (1406.0) by 7.0.
GSO-Bench (Opt@1 (%)): Claude Opus 4.8 (47.06) beat Claude Opus 4.7 (44.12) by 2.94.
Open Universal Arabic ASR Leaderboard (Average WER (lower is better)): cohere-transcribe-arabic-07-2026 (25.87) beat omniASR_LLM_7B (28.32) by 2.45.
AMA-Bench - Embodied AI (Average Score (%)): VeraKV (Qwen3-32B) (58.09) beat GPT-5.2 (56.44) by 1.65.
Open Universal Arabic ASR Leaderboard - CER (Average CER (lower is better)): cohere-transcribe-arabic-07-2026 (11.8) beat Qwen3-ASR-1.7B (12.33) by 0.53.

Weekly
New Benchmarks (15)

Shader Benchmark (Avg judge score (0-500, render fails count 0)): gpt-5.5-high leads with 242.2 across 3 models.
  Shader Benchmark (nbardy) — generating shader programs from natural-language prompts, graded by an LLM judge panel with render failures counting as zero; average judge score (0-500).
KINA (Accuracy (%)): Gemini-3.1-Pro-Preview leads with 53.17 across 42 models.
  KINA (Knowledge Index of Noah's Ark) — 899-item knowledge benchmark across 261 fine-grained disciplines with representativeness-aware sampling and tournament-audited annotation; overall accuracy.
Office Comprehension Bench (Domain Q&A Accuracy (%)): GPT-5.5 Thinking leads with 59.35 across 3 models.
  Office Comprehension Bench (Microsoft) — Domain Q&A track: expert-level multi-step reasoning over native Word/Excel/PowerPoint files across 12 professional domains; assertion-level accuracy via LLM-judge ensemble.
OmniGameArena - ObstacleRun2D (Normalized game score (%)): GPT-5.5 leads with 47.3 across 12 models.
  The \
OmniGameArena - ObstacleRun3D (Normalized game score (%)): Claude Sonnet 4.6 leads with 20.0 across 12 models.
  The \
OmniGameArena - LastStand (Normalized game score (%)): GPT-5.5 leads with 41.6 across 12 models.
  The \
OmniGameArena - MonsterShoot (Normalized game score (%)): Gemini 3.1 Pro Preview leads with 71.0 across 12 models.
  The \
OmniGameArena - SceneEscape (Normalized game score (%)): GPT-5.5 leads with 72.0 across 12 models.
  The \
OmniGameArena - CueChase (Normalized game score (%)): Claude Opus 4.6 leads with 84.0 across 12 models.
  The \
OmniGameArena - SoloCraft (Normalized game score (%)): GPT-5.5 leads with 25.2 across 12 models.
  The \
OmniGameArena - SharedFloor (Normalized game score (%)): GPT-5.5 leads with 36.8 across 10 models.
  The \
OmniGameArena - HandoffRun (Normalized game score (%)): GPT-5.5 leads with 18.4 across 10 models.
  The \
Poker Arena (Tournament win rate (%)): Claude Opus 4.6 leads with 19.4 across 7 models.
  Poker Arena (arXiv) — strategic reasoning, risk assessment and bluffing measured through head-to-head LLM poker tournaments; tournament win rate (%).
ClawArena (Composite Reliability Score): Claude Fable-5 leads with 70.05 across 20 models.
  ClawArena (clawarena.cc) — reliability of LLM subagent orchestration under a Claw-style harness; Composite Reliability Score across scripted multi-agent runs.
OpenCompass Language - Chat (Score (%)): Doubao-Seed-2-0-Pro-260215 (high) leads with 91.7 across 23 models.

New Models (131)

Claude Fable 5 — ELO 3083, #6/1474, above Gemini 3 Deep Think, below GPT-5.4 Pro
Senior SWE-Bench: 29.1 (#1/10)
KernelBench Hub - Mega: 18.7 (#2/12)
TrackingAI IQ Test (Vision): 78.57 (#3/19)
KernelBench Hub - Hard: 24.0 (#4/13)
TaxCalcBench: 26.0 (#5/9)
Lean AI Formalization Leaderboard: 4.0 (#13/35)
TrackingAI IQ Test (Mensa Norway): 60.0 (#20/31)

Claude Opus 4.8 — ELO 2770, #8/1474, above GPT-5.4 Pro, below Muse Spark 1.1
GSO-Bench: 47.06 (#1/30)
TaxCalcBench: 16.0 (#7/9)

Muse Spark 1.1 — ELO 2766, #9/1474, above Claude Opus 4.8, below GPT-5.6 Sol
SEAL - MultiNRC: 65.59 (#1/44)
SEAL - VisualToolBench (VTB): 44.77 (#1/21)
SEAL - SWE-Bench Pro (Public Dataset): 61.5 (#1/25)
SEAL - SWE-Bench Pro (Private Dataset): 51.5 (#1/14)
SEAL - Professional Reasoning Benchmark - Finance: 55.01 (#1/29)
SEAL - Professional Reasoning Benchmark - Legal: 57.05 (#1/29)
SEAL - MCP Atlas: 88.1 (#1/24)
Vals AI TaxEval v2: 79.72 (#1/127)
Vals AI MedScribe: 88.89 (#1/70)
Vals AI Harvey Legal Agent Bench: 20.0 (#1/19)

GPT-5.6 Sol — ELO 2718, #10/1474, above Muse Spark 1.1, below Seed 2.1 Pro
KernelBench Hub - Hard: 30.0 (#1/13)
FrontierMath - Tiers 1-3 (v2): 89.12 (#1/36)
Benchmarks.bio - scBench-Long: 42.5 (#1/17)
Design Arena (UI Components): 1413.0 (#1/128)
Epoch AI - ECI: 161.61 (#1/424)
OTIS Mock AIME 2024-25: 100.0 (#1/153)
Chess Puzzles (Epoch AI): 64.0 (#1/55)
Epoch AI - Critpt: 32.3 (#1/70)
AA GPQA Diamond: 94.14 (#1/537)
Tau3 Banking: 32.99 (#1/150)

Seed 2.1 Pro — ELO 2674, #11/1474, above GPT-5.6 Sol, below GPT-5.5
BabyVision: 73.7 (#4/24)
Agents' Last Exam: 19.5 (#11/20)

GPT-5.5 — ELO 2608, #12/1474, above Seed 2.1 Pro, below Qwen 3.7 Max
BabyVision: 83.6 (#1/24)
TaxCalcBench: 24.0 (#6/9)

Qwen 3.7 Max — ELO 2577, #13/1474, above GPT-5.5, below Claude Opus 4.6
Agent Arena - Tool Hallucination: 1.02 (#14/32)
Agent Arena - Bash Recovery: 7.91 (#20/32)
Agent Arena - Steerability: 5.2 (#22/32)
Agent Arena - Confirmed Success: 1.43 (#29/32)
Agent Arena: 0.36 (#30/32)
Agent Arena - Praise vs Complaint: 0.52 (#31/32)

Gemini 3.1 Pro (Preview) — ELO 2507, #15/1474, above Claude Opus 4.6, below Qwen 3.6 Max
TaxCalcBench: 2.0 (#9/9)
Agents' Last Exam: 15.8 (#12/20)

Claude Sonnet 5 — ELO 2441, #19/1474, above Gemini 3.5 Flash, below Muse Spark
You're Absolutely Right!: 3.875 (#4/21)
KernelBench Hub - Hard: 23.0 (#5/13)
GSO-Bench: 37.25 (#5/30)
KernelBench Hub - Mega: 4.0 (#6/12)
EQ-Bench Longform Writing: 78.3 (#7/117)
Benchmarks.bio - scBench-Long: 14.29 (#8/17)
TaxCalcBench: 6.0 (#8/9)
Kaggle Game Arena Poker (Heads Up): 2.21 (#9/20)
Bullshit Benchmark: 65.5 (#14/158)
SimpleBench: 60.6 (#19/86)

Gemini 3 Pro — ELO 2412, #22/1474, above GPT-5.3 Codex, below GPT-5 Pro
Chatbot Arena (Image Edit): 1388.0 (#7/52)
Chatbot Arena (Text-to-Image): 1245.0 (#8/74)

Grok 4.5 — ELO 2403, #24/1474, above GPT-5 Pro, below GLM-5.2
SWE-Marathon: 29.0 (#1/12)
Vals AI Harvey Legal Agent Bench: 12.92 (#2/19)
RuneBench: 5682.0 (#3/30)
Lynchmark: 83.33 (#4/10)
Chatbot Arena (Code): 1566.0 (#4/96)
Vals AI SWE-bench Verified: 86.6 (#4/69)
VoxelBench: 1775.0 (#5/45)
Senior SWE-Bench: 17.2 (#5/10)
Vals AI MedScribe: 86.88 (#5/70)
Vals AI GPQA: 92.93 (#5/121)

GLM-5.2 — ELO 2398, #25/1474, above Grok 4.5, below GPT-5.2 Pro
Design Arena (Website): 1346.0 (#1/152)
Design Arena (3D): 1371.0 (#1/125)
Design Arena (Game Dev): 1354.0 (#3/138)
Agent Security League - Functional Correctness: 82.5 (#4/17)
Design Arena (UI Components): 1335.0 (#5/128)
Design Arena (Data Viz): 1318.0 (#6/132)
Agent Security League - Security Correctness: 12.0 (#12/17)
Riemann-bench: 10.4 (#16/20)
SWE-rebench: 51.09 (#29/91)
LisanBench: 0.0161 (#89/150)

GPT-5.6 Terra — ELO 2380, #27/1474, above GPT-5.2 Pro, below GPT-5.4
Benchmarks.bio - scBench-Long: 25.4 (#2/17)
AA CritPt: 30.0 (#2/443)
Vals AI ProofBench: 71.0 (#3/49)
Vals AI Excel Modeling: 67.61 (#3/21)
FrontierMath - Tiers 1-3 (v2): 85.96 (#4/36)
OTIS Mock AIME 2024-25: 99.72 (#4/153)
Epoch AI - Critpt: 30.0 (#4/70)
Tau3 Banking: 31.75 (#4/150)
Gert Labs Rankings: 66.95 (#4/82)
Vals AI Legal Research Bench: 40.87 (#4/18)

GPT-5 — ELO 2356, #29/1474, above GPT-5.4, below GPT-5 (2025-08-07)
Epoch AI - Scicode: 42.94 (#47/69)

Gemini 3.1 Flash — ELO 2324, #31/1474, above GPT-5 (2025-08-07), below Qwen 3.7 Plus
Chatbot Arena (Text-to-Image): 1261.0 (#5/74)
Chatbot Arena (Image Edit): 1385.0 (#9/52)

Qwen 3.7 Plus — ELO 2317, #32/1474, above Gemini 3.1 Flash, below Qwen3-VL-235B-A22B-Thinking-FP8
Agent Arena - Tool Hallucination: 0.63 (#4/32)
Design Arena (Website): 1287.0 (#21/152)
Agent Arena - Steerability: 5.2 (#23/32)
Agent Arena - Bash Recovery: 5.9 (#24/32)
Agent Arena - Confirmed Success: 1.87 (#27/32)
Agent Arena - Praise vs Complaint: 1.63 (#27/32)
Agent Arena: 0.43 (#29/32)

Nex N2 Pro — ELO 2290, #36/1474, above GPT-5.2 Codex, below O3 Pro
Wolfram LLM Benchmarking Project: 61.3 (#47/500)

GPT-5.6 Luna — ELO 2265, #41/1474, above Gemini 3 Flash, below Gemini 2.5 Pro
Vals AI IOI: 72.92 (#2/58)
Vals AI Terminal-Bench 2.1: 79.03 (#3/41)
Benchmarks.bio - scBench-Long: 19.05 (#5/17)
OTIS Mock AIME 2024-25: 98.33 (#6/153)
Vals AI Vibe Code Bench: 77.06 (#6/72)
GDP.pdf: 22.7 (#6/17)
FrontierMath - Tiers 1-3 (v2): 82.11 (#7/36)
AA Long Context Reasoning: 74.0 (#7/465)
FrontierMath - Tier 4 (v2): 60.98 (#8/38)
AA GDPval: 1591.81 (#8/148)

Gemini 2.5 Pro — ELO 2263, #42/1474, above GPT-5.6 Luna, below Claude Opus 4.5 (20251101)
MCPMark: 15.75 (#29/39)

Kimi K2.7 Code — ELO 2238, #46/1474, above O3 Pro (2025-06-10), below DeepSeek V4 Pro
Design Arena (SVG): 1237.0 (#24/99)
SimpleBench: 57.9 (#24/86)

DeepSeek V4 Pro — ELO 2230, #47/1474, above Kimi K2.7 Code, below MiniMax-M3
Agent Arena - Tool Hallucination: 0.99 (#11/32)
Agent Arena - Confirmed Success: 2.34 (#24/32)
Agent Arena - Steerability: 4.61 (#24/32)
Agent Arena - Bash Recovery: 4.94 (#25/32)
Agent Arena - Praise vs Complaint: 3.13 (#26/32)
Agent Arena: 0.83 (#28/32)
SWE-rebench: 42.7 (#45/91)

MiniMax-M3 — ELO 2228, #48/1474, above DeepSeek V4 Pro, below GPT-5.1 Codex
Agent Arena - Tool Hallucination: 0.99 (#12/32)
Agent Arena - Confirmed Success: 7.66 (#14/32)
Agent Arena - Praise vs Complaint: 9.93 (#16/32)
Agent Arena - Steerability: 5.24 (#21/32)
Agent Arena - Bash Recovery: 6.46 (#22/32)
Agent Arena: 3.08 (#23/32)
SimpleBench: 45.8 (#43/86)
Epoch AI - ECI: 144.9 (#182/424)

MiMo-V2.5-Pro — ELO 2204, #58/1474, above Grok 4.20 Multi-Agent, below DeepSeek V4 Flash
Agent Arena - Tool Hallucination: 0.63 (#5/32)
Agent Arena - Praise vs Complaint: 10.28 (#15/32)
Agent Arena - Confirmed Success: 6.18 (#19/32)
Agent Arena - Steerability: 5.48 (#20/32)
Agent Arena: 3.71 (#22/32)
Agent Arena - Bash Recovery: 2.77 (#30/32)
SWE-rebench: 42.36 (#46/91)

DeepSeek V4 Flash — ELO 2201, #59/1474, above MiMo-V2.5-Pro, below Claude Sonnet 4.6
Agent Arena - Tool Hallucination: 0.6 (#3/32)
Agent Arena - Confirmed Success: 9.37 (#8/32)
Agent Arena - Praise vs Complaint: 10.65 (#14/32)
Agent Arena - Steerability: 7.33 (#16/32)
Agent Arena: 4.65 (#21/32)
Agent Arena - Bash Recovery: 4.71 (#27/32)
SWE-rebench: 38.36 (#51/91)

Kimi K2.6 — ELO 2171, #63/1474, above Seed 2.0 Pro, below GLM-5 FP8
BabyVision: 68.5 (#6/24)
Agent Arena - Steerability: 7.33 (#15/32)
Agent Arena - Bash Recovery: 8.23 (#19/32)
Agent Arena - Praise vs Complaint: 3.89 (#24/32)
Agent Arena: 1.61 (#25/32)
Agent Arena - Confirmed Success: 2.34 (#25/32)
Agent Arena - Tool Hallucination: 1.24 (#30/32)

GLM-5.1 — ELO 2136, #73/1474, above O3 (2025-04-16), below GPT-5.1 Codex Max
Agent Arena: 1.57 (#26/32)
Agent Arena - Confirmed Success: 1.67 (#28/32)
Agent Arena - Praise vs Complaint: 0.8 (#28/32)
Agent Arena - Bash Recovery: 4.07 (#28/32)
Agent Arena - Tool Hallucination: 1.24 (#28/32)
SimpleBench: 55.1 (#28/86)
Agent Arena - Steerability: 0.05 (#32/32)

Grok 4.3 — ELO 2127, #77/1474, above Qwen 3.6 Plus, below DeepSeek-V4-Flash-FP8
Agent Arena: 15.57 (#1/32)
Agent Arena - Bash Recovery: 43.1 (#1/32)
Agent Arena - Praise vs Complaint: 16.01 (#3/32)
Agent Arena - Confirmed Success: 11.3 (#5/32)
Agent Arena - Tool Hallucination: 0.91 (#10/32)
Agent Arena - Steerability: 8.34 (#13/32)
RuneBench: 1609.0 (#20/30)

Claude Sonnet 4.5 — ELO 2091, #93/1474, above GLM-5-Turbo, below Kimi K2.5
MCPMark: 32.09 (#9/39)

Qwen 3 Max — ELO 2090, #95/1474, above Kimi K2.5, below GLM-5
MCPMark: 17.72 (#25/39)

MiMo-V2-Pro — ELO 2068, #98/1474, above NVIDIA-Nemotron-3-Super-120B-A12B-FP8, below plutus-8B-instruct
AI Chess Leaderboard (Reasoning): 756.0 (#101/292)
AI Chess Leaderboard (Continuation): 591.0 (#112/237)

Qwen 3.6 27B — ELO 2060, #102/1474, above Gemini 3.1 Flash Lite, below Step 3.7 Flash
SWE-rebench: 36.55 (#58/91)

Grok 4 — ELO 2058, #104/1474, above Step 3.7 Flash, below Qwen 3 VL 235B A22B Instruct
MCPMark: 31.69 (#10/39)

GLM-5V Turbo — ELO 2050, #107/1474, above Grok 4.20, below LongCat-Flash-Thinking-2601
Chatbot Arena (Code): 1402.0 (#44/96)
Chatbot Arena (Text): 1440.0 (#66/374)

Claude Sonnet 4 — ELO 2036, #109/1474, above LongCat-Flash-Thinking-2601, below Step 3.5 Flash
MCPMark: 28.15 (#15/39)

Grok 4 Fast — ELO 2031, #111/1474, above Step 3.5 Flash, below Qwen 3 Max (2025-09-23)
MCPMark: 24.02 (#20/39)

Claude Opus 4.1 — ELO 2024, #113/1474, above Qwen 3 Max (2025-09-23), below Gemini 2.5 Flash (Preview 05-20)
MCPMark: 29.92 (#12/39)

GPT-5 Mini — ELO 2016, #119/1474, above GPT-5 Chat, below Kimi K2
Epoch AI - Scicode: 39.24 (#55/69)

O4 Mini — ELO 2014, #121/1474, above Kimi K2, below Claude Opus 4
MCPMark: 17.32 (#26/39)

MiMo-V2.5 — ELO 2007, #125/1474, above KAT Coder Pro V2, below GPT-5.1 ChatGPT
AA GDPval: 1145.01 (#50/148)
GDPval-AA: 1145.0 (#54/153)
Tau3 Banking: 6.6 (#102/150)

Mercury 2 — ELO 1980, #139/1474, above Qwen 3 Max (Preview), below Fino1-14B
Design Arena (SVG): 1024.0 (#93/99)

Kimi K2 0905 — ELO 1978, #142/1474, above Qwen 3 VL 32B Instruct, below Gemma 4 26B A4B (IT)
MCPMark: 21.85 (#21/39)

GPT-5.4 Mini — ELO 1972, #148/1474, above Gemini 2.5 Pro (Preview 03-25), below Gemini 2.5 Flash (Preview 09-2025)
BenchLM: 65.0 (#33/79)

MiniMax-M2.7 — ELO 1967, #151/1474, above Qwen 3.5 122B A10B, below MiniMax M1 80k
Agent Arena - Confirmed Success: 17.12 (#1/32)
Agent Arena - Steerability: 16.42 (#2/32)
Agent Arena: 10.18 (#5/32)
Agent Arena - Praise vs Complaint: 14.49 (#5/32)
Agent Arena - Tool Hallucination: 0.89 (#9/32)
Agent Arena - Bash Recovery: 3.74 (#29/32)

GPT-4o ChatGPT — ELO 1964, #155/1474, above Gemini 2.5 Pro (Preview 05-06), below GPT-4.1 (2025-04-14)
LiveBench Table Join: 45.3 (#1/73)
LiveBench AMPS Hard: 48.0 (#2/73)
LiveBench Zebra Puzzle: 46.0 (#2/73)
LiveBench Connections: 51.33 (#3/73)
LiveBench Math Comp: 44.79 (#5/73)
LiveBench Story Generation: 79.92 (#6/73)
LiveBench Olympiad: 63.78 (#9/73)
LiveBench Plot Unscrambling: 42.53 (#9/73)
LiveBench Summarize: 71.42 (#10/73)
LiveBench Typos: 56.0 (#10/73)

Gemini 2.5 Flash — ELO 1955, #161/1474, above Step3 VL 10B, below GPT-4o
MCPMark: 9.06 (#31/39)

Hy3 — ELO 1946, #168/1474, above Command A+, below Qwen 3 VL 32B
AI Chess Leaderboard (Reasoning): 1540.0 (#15/292)
AI Chess Leaderboard (Continuation): 1021.0 (#38/237)
Gert Labs Rankings: 41.68 (#56/82)
Design Arena (Game Dev): 1202.0 (#61/138)
SvelteBench: 88.9 (#66/145)
Design Arena (Website): 1198.0 (#74/152)

GLM-4.5 — ELO 1932, #173/1474, above O3 Mini (2025-01-31), below MiniMax-M2.1
MCPMark: 15.55 (#30/39)

Claude 3.5 Sonnet (20240620) — ELO 1925, #177/1474, above GLM-4.6, below Gemini 3.1 Flash Lite (Preview)
LiveBench AMPS Hard: 49.0 (#1/73)
LiveBench Plot Unscrambling: 55.14 (#1/73)
LiveBench Zebra Puzzle: 48.0 (#1/73)
LiveBench Olympiad: 67.46 (#2/73)
LiveBench Table Reformat: 68.0 (#2/73)
LiveBench Typos: 66.0 (#2/73)
LiveBench Table Join: 40.22 (#3/73)
LiveBench Connections: 49.67 (#4/73)
LiveBench Math Comp: 44.79 (#6/73)
LiveBench Paraphrase: 72.22 (#8/73)

DeepSeek V3.1 Terminus — ELO 1924, #181/1474, above Ring-2.6-1T, below Mistral-Large-3-675B-Instruct-2512
MCPMark: 16.54 (#28/39)

GPT-4.1 — ELO 1923, #183/1474, above Mistral-Large-3-675B-Instruct-2512, below MiMo-V2-Omni
MCPMark: 8.07 (#33/39)

Qwen 3 Coder Plus — ELO 1904, #196/1474, above Hermes-4-14B, below Arcee Trinity Large
MCPMark: 24.8 (#19/39)

qwen3.5-flash — ELO 1879, #217/1474, above MiniMax-M2.5, below Qwen 3 Next 80B A3B Instruct
AI Chess Leaderboard (Continuation): 437.0 (#182/237)

GPT-4o (2024-05-13) — ELO 1871, #219/1474, above Qwen 3 Next 80B A3B Instruct, below InternVL3-38B
LiveBench Connections: 53.0 (#2/73)
LiveBench Typos: 66.0 (#3/73)
LiveBench AMPS Hard: 45.0 (#4/73)
LiveBench Olympiad: 65.06 (#4/73)
LiveBench Summarize: 74.62 (#6/73)
LiveBench Plot Unscrambling: 42.81 (#8/73)
LiveBench Paraphrase: 70.8 (#10/73)
LiveBench Zebra Puzzle: 40.0 (#10/73)
LiveBench Math Comp: 39.58 (#11/73)
LiveBench Table Join: 29.22 (#12/73)

Gemini 1.5 Pro (Preview 0801) — ELO 1869, #222/1474, above DeepSeek R1 0528, below Qwen 3 Next 80B A3B
LiveBench Summarize: 83.13 (#2/73)
LiveBench Table Reformat: 68.0 (#3/73)
LiveBench Paraphrase: 78.63 (#4/73)
LiveBench Simplify: 76.5 (#5/73)
LiveBench Typos: 62.0 (#5/73)
LiveBench AMPS Hard: 43.0 (#6/73)
LiveBench Plot Unscrambling: 41.87 (#10/73)
LiveBench Olympiad: 61.88 (#11/73)
LiveBench Story Generation: 77.08 (#11/73)
LiveBench Math Comp: 37.5 (#12/73)

Gemini 1.5 Pro (Preview 0827) — ELO 1866, #227/1474, above GLM 4.5 Air, below Claude Haiku 4.5 (20251001)
LiveBench Math Comp: 54.17 (#1/73)
LiveBench Olympiad: 68.66 (#1/73)
LiveBench AMPS Hard: 46.0 (#3/73)
LiveBench Simplify: 77.8 (#3/73)
LiveBench Table Reformat: 68.0 (#4/73)
LiveBench Paraphrase: 77.58 (#5/73)
LiveBench Typos: 62.0 (#6/73)
LiveBench Plot Unscrambling: 43.44 (#7/73)
LiveBench Summarize: 73.4 (#7/73)
LiveBench Connections: 42.5 (#8/73)

GPT-4 Turbo — ELO 1858, #231/1474, above GPT-5.4 Nano, below nemotron-3-super-120B-a12B
LiveBench Connections: 44.17 (#5/73)
LiveBench Math Comp: 44.79 (#7/73)
LiveBench Simplify: 74.23 (#7/73)
LiveBench Story Generation: 79.92 (#7/73)
LiveBench AMPS Hard: 42.0 (#8/73)
LiveBench Table Join: 35.96 (#8/73)
LiveBench Olympiad: 60.19 (#13/73)
LiveBench Table Reformat: 64.0 (#13/73)
LiveBench Typos: 52.0 (#13/73)
LiveBench Plot Unscrambling: 39.62 (#14/73)

Qwen3 Coder Next — ELO 1857, #233/1474, above nemotron-3-super-120B-a12B, below Trinity Large
Tau3 Banking: 5.15 (#119/150)

Qwen 3.6 35B A3B — ELO 1855, #236/1474, above Grok 3 Mini Beta, below GPT-4o (2024-11-20)
SWE-rebench: 33.82 (#65/91)

GPT-4o (2024-08-06) — ELO 1839, #245/1474, above GPT-OSS-120B, below GPT-4
LiveBench Connections: 58.0 (#1/73)
LiveBench Olympiad: 65.96 (#3/73)
LiveBench Math Comp: 47.92 (#4/73)
LiveBench Table Join: 38.68 (#4/73)
LiveBench Plot Unscrambling: 45.11 (#5/73)
LiveBench Story Generation: 80.08 (#5/73)
LiveBench Summarize: 75.78 (#5/73)
LiveBench AMPS Hard: 43.0 (#7/73)
LiveBench Typos: 60.0 (#8/73)
LiveBench Zebra Puzzle: 42.0 (#8/73)

GPT-4 Preview (0125) — ELO 1833, #247/1474, above GPT-4, below Mistral Medium 3.5
LiveBench Table Reformat: 70.0 (#1/73)
LiveBench Zebra Puzzle: 44.0 (#4/73)
LiveBench Plot Unscrambling: 43.49 (#6/73)
LiveBench Connections: 43.17 (#7/73)
LiveBench Table Join: 36.18 (#7/73)
LiveBench AMPS Hard: 37.0 (#14/73)
LiveBench Math Comp: 36.46 (#18/73)
LiveBench Summarize: 65.68 (#19/73)
LiveBench Typos: 44.0 (#21/73)
LiveBench Olympiad: 54.79 (#22/73)

Gemini 1.5 Flash (Preview 0827) — ELO 1832, #249/1474, above Mistral Medium 3.5, below Awqward2.5-32B-Instruct
LiveBench Simplify: 84.75 (#1/73)
LiveBench Story Generation: 81.33 (#1/73)
LiveBench Paraphrase: 73.58 (#6/73)
LiveBench Summarize: 72.77 (#9/73)
LiveBench Olympiad: 63.4 (#10/73)
LiveBench Table Reformat: 64.0 (#11/73)
LiveBench Math Comp: 36.46 (#16/73)
LiveBench Plot Unscrambling: 34.63 (#18/73)
LiveBench Connections: 30.5 (#19/73)
LiveBench Table Join: 21.6 (#20/73)

Grok Code Fast 1 — ELO 1780, #279/1474, above Qwen Max, below nova-2-lite-v1
MCPMark: 20.47 (#23/39)

DeepSeek V3 Chat — ELO 1774, #288/1474, above Mistral Large 3, below Qwen 2.5 72B Instruct
MCPMark: 16.73 (#27/39)

Llama 3.1 70B Instruct — ELO 1764, #297/1474, above Mistral Small 3.2, below QVQ-72B-Preview
LiveBench Paraphrase: 82.25 (#1/73)
LiveBench Summarize: 83.85 (#1/73)
LiveBench Table Reformat: 68.0 (#5/73)
LiveBench Simplify: 74.8 (#6/73)
LiveBench Connections: 41.67 (#9/73)
LiveBench Table Join: 30.88 (#9/73)
LiveBench Math Comp: 41.67 (#10/73)
LiveBench Olympiad: 60.06 (#14/73)
LiveBench Story Generation: 75.42 (#14/73)
LiveBench AMPS Hard: 35.0 (#15/73)

GPT-4.1 Mini — ELO 1756, #303/1474, above Laguna XS.2, below Qwentile2.5-32B-Instruct
MCPMark: 3.94 (#38/39)

Llama 3.1 405B Instruct — ELO 1724, #336/1474, above Palmyra-X-004, below Nova Pro (v1)
LiveBench Paraphrase: 81.13 (#2/73)
LiveBench Simplify: 78.33 (#2/73)
LiveBench Plot Unscrambling: 47.38 (#3/73)
LiveBench Summarize: 78.93 (#3/73)
LiveBench Connections: 44.17 (#6/73)
LiveBench Table Join: 36.52 (#6/73)
LiveBench Olympiad: 64.15 (#8/73)
LiveBench Typos: 58.0 (#9/73)
LiveBench AMPS Hard: 38.0 (#12/73)
LiveBench Math Comp: 37.5 (#13/73)

Claude 3 Opus (20240229) — ELO 1713, #345/1474, above GPT-4o (Mar 2025), below GLM-4.7 Flash
LiveBench Typos: 68.0 (#1/73)
LiveBench Plot Unscrambling: 50.16 (#2/73)
LiveBench Table Join: 40.96 (#2/73)
LiveBench Math Comp: 41.67 (#9/73)
LiveBench Table Reformat: 64.0 (#9/73)
LiveBench AMPS Hard: 38.0 (#11/73)
LiveBench Simplify: 72.55 (#11/73)
LiveBench Summarize: 70.48 (#11/73)
LiveBench Connections: 37.0 (#12/73)
LiveBench Paraphrase: 69.38 (#13/73)

Mistral Large 2 (Jul) — ELO 1702, #357/1474, above Llama 3.3 70B Instruct, below qwen2.5-reinstruct-alternate-lumen-14B
LiveBench Olympiad: 64.41 (#6/73)
LiveBench Story Generation: 78.5 (#8/73)
LiveBench Simplify: 73.13 (#9/73)
LiveBench Plot Unscrambling: 40.05 (#12/73)
LiveBench Typos: 50.0 (#14/73)
LiveBench Summarize: 68.15 (#15/73)
LiveBench Table Reformat: 64.0 (#17/73)
LiveBench Table Join: 21.82 (#19/73)
LiveBench Connections: 29.33 (#20/73)
LiveBench Paraphrase: 67.6 (#20/73)

GPT-4 (0613) — ELO 1698, #363/1474, above Palmyra X V3 (72B), below Gemini 2.5 Flash Lite
LiveBench Plot Unscrambling: 45.54 (#4/73)
LiveBench Typos: 64.0 (#4/73)
LiveBench Zebra Puzzle: 42.0 (#7/73)
LiveBench Summarize: 73.33 (#8/73)
LiveBench Connections: 39.17 (#10/73)
LiveBench Story Generation: 76.08 (#12/73)
LiveBench Simplify: 70.97 (#14/73)
LiveBench Olympiad: 58.7 (#18/73)
LiveBench Table Join: 20.08 (#21/73)
LiveBench Paraphrase: 66.77 (#23/73)

DeepSeek Coder V2 — ELO 1696, #367/1474, above c4ai-command-a-03-2025, below InternVL3-8B
LiveBench Math Comp: 50.0 (#3/73)
LiveBench AMPS Hard: 43.0 (#5/73)
LiveBench Olympiad: 64.61 (#5/73)
LiveBench Zebra Puzzle: 42.0 (#5/73)
LiveBench Table Join: 30.76 (#10/73)
LiveBench Story Generation: 75.33 (#15/73)
LiveBench Summarize: 64.7 (#21/73)
LiveBench Typos: 42.0 (#22/73)
LiveBench Plot Unscrambling: 31.3 (#23/73)
LiveBench Connections: 25.83 (#25/73)

Qwen 2 72B Instruct — ELO 1689, #376/1474, above gemma-3-12B-pt, below Qwen 3 VL 8B Instruct
LiveBench Story Generation: 80.17 (#4/73)
LiveBench Zebra Puzzle: 40.0 (#13/73)
LiveBench Math Comp: 37.5 (#14/73)
LiveBench Connections: 34.0 (#15/73)
LiveBench Olympiad: 59.81 (#16/73)
LiveBench Paraphrase: 68.73 (#16/73)
LiveBench AMPS Hard: 33.0 (#20/73)
LiveBench Plot Unscrambling: 33.64 (#21/73)
LiveBench Summarize: 62.02 (#30/73)
LiveBench Simplify: 62.17 (#31/73)

Qwen 2.5 Max — ELO 1652, #417/1474, above Qwen 2.5 14B, below Qwen2.5-7B-Instruct-Uncensored
AI Chess Leaderboard (Reasoning): 667.0 (#138/292)
AI Chess Leaderboard (Continuation): 414.0 (#194/237)

Gemma 3 12B (IT) — ELO 1639, #425/1474, above Laguna M.1, below Mistral Small 3
GDPval-AA: 144.0 (#139/153)
AA GDPval: -143.64 (#146/148)

Phi-3.5-MoE-instruct — ELO 1622, #444/1474, above Granite 4.0 H Small, below Qwen 2 VL 7B
LiveBench AMPS Hard: 32.0 (#21/73)
LiveBench Paraphrase: 66.3 (#24/73)
LiveBench Table Reformat: 58.0 (#24/73)
LiveBench Math Comp: 26.04 (#26/73)
LiveBench Simplify: 63.28 (#28/73)
LiveBench Zebra Puzzle: 34.0 (#31/73)
LiveBench Typos: 26.0 (#35/73)
LiveBench Olympiad: 41.87 (#36/73)
LiveBench Summarize: 51.6 (#44/73)
LiveBench Connections: 11.0 (#45/73)

Gemma 2 27B (IT) — ELO 1598, #468/1474, above Darkest-muse-v1, below gemma-4-E4B-it
LiveBench Zebra Puzzle: 42.0 (#6/73)
LiveBench Paraphrase: 71.13 (#9/73)
LiveBench Table Reformat: 64.0 (#12/73)
LiveBench AMPS Hard: 33.0 (#19/73)
LiveBench Summarize: 64.42 (#22/73)
LiveBench Olympiad: 53.83 (#24/73)
LiveBench Typos: 42.0 (#24/73)
LiveBench Connections: 25.17 (#26/73)
LiveBench Plot Unscrambling: 30.03 (#26/73)
LiveBench Simplify: 64.52 (#26/73)

DeepSeek V2.5 — ELO 1593, #474/1474, above recoilme-gemma-2-9B-v0.2, below Granite 4.1 8B
LiveBench Math Comp: 53.12 (#2/73)
LiveBench Olympiad: 64.29 (#7/73)
LiveBench AMPS Hard: 40.0 (#10/73)
LiveBench Story Generation: 77.17 (#10/73)
LiveBench Table Reformat: 64.0 (#10/73)
LiveBench Paraphrase: 68.72 (#17/73)
LiveBench Plot Unscrambling: 35.38 (#17/73)
LiveBench Table Join: 24.34 (#17/73)
LiveBench Summarize: 65.12 (#20/73)
LiveBench Connections: 28.17 (#22/73)

Llama 3 70B Instruct — ELO 1580, #490/1474, above Smaug-Llama-3-70B-Instruct-32K, below Qwen2.5-7B-Instruct-abliterated
LiveBench Table Join: 25.24 (#16/73)
LiveBench Connections: 31.17 (#17/73)
LiveBench Simplify: 67.13 (#18/73)
LiveBench Olympiad: 54.11 (#23/73)
LiveBench Plot Unscrambling: 31.15 (#24/73)
LiveBench Story Generation: 69.75 (#25/73)
LiveBench Typos: 40.0 (#26/73)
LiveBench Table Reformat: 52.0 (#29/73)
LiveBench Summarize: 59.25 (#33/73)
LiveBench AMPS Hard: 22.0 (#35/73)

Gemini 1.5 Flash (0514) — ELO 1572, #509/1474, above JT-MINI, below openbuddy-llama3-70B-v21.2-32k
LiveBench Olympiad: 61.67 (#12/73)
LiveBench Zebra Puzzle: 38.0 (#16/73)
LiveBench Typos: 46.0 (#17/73)
LiveBench Table Join: 24.08 (#18/73)
LiveBench Simplify: 66.77 (#20/73)
LiveBench AMPS Hard: 30.0 (#25/73)
LiveBench Table Reformat: 54.0 (#25/73)
LiveBench Paraphrase: 66.1 (#27/73)
LiveBench Plot Unscrambling: 29.41 (#27/73)
LiveBench Math Comp: 25.0 (#28/73)

GPT-4o Mini (2024-07-18) — ELO 1568, #515/1474, above Sonar Pro, below Llama 3 70B
LiveBench AMPS Hard: 41.0 (#9/73)
LiveBench Connections: 36.5 (#14/73)
LiveBench Typos: 46.0 (#18/73)
LiveBench Math Comp: 35.42 (#19/73)
LiveBench Table Reformat: 60.0 (#21/73)
LiveBench Zebra Puzzle: 36.0 (#21/73)
LiveBench Paraphrase: 67.0 (#22/73)
LiveBench Simplify: 65.53 (#23/73)
LiveBench Story Generation: 71.0 (#23/73)
LiveBench Olympiad: 48.32 (#26/73)

Gemini 1.5 Pro (0514) — ELO 1560, #527/1474, above oxyge1-33B, below romulus-mistral-nemo-12B-simpo
LiveBench Table Join: 38.42 (#5/73)
LiveBench Table Reformat: 66.0 (#7/73)
LiveBench Typos: 56.0 (#11/73)
LiveBench Plot Unscrambling: 39.92 (#13/73)
LiveBench Simplify: 71.05 (#13/73)
LiveBench Paraphrase: 69.27 (#14/73)
LiveBench AMPS Hard: 34.0 (#17/73)
LiveBench Math Comp: 36.46 (#17/73)
LiveBench Olympiad: 56.8 (#20/73)
LiveBench Summarize: 63.8 (#24/73)

Mistral Large — ELO 1540, #564/1474, above Gemma-2-9B-It-SPPO-Iter1, below Tiger-Gemma-9B-v2
LiveBench Zebra Puzzle: 40.0 (#11/73)
LiveBench Summarize: 69.23 (#13/73)
LiveBench Table Reformat: 64.0 (#16/73)
LiveBench Simplify: 67.27 (#17/73)
LiveBench Plot Unscrambling: 33.9 (#20/73)
LiveBench Story Generation: 72.92 (#20/73)
LiveBench Connections: 28.33 (#21/73)
LiveBench Math Comp: 28.12 (#22/73)
LiveBench Table Join: 15.66 (#25/73)
LiveBench Olympiad: 46.49 (#28/73)

Qwen 1.5 110B Chat — ELO 1511, #607/1474, above Qwen 1.5 32B, below Llama-VARCO-8B-Instruct
LiveBench AMPS Hard: 23.0 (#33/73)
LiveBench Connections: 20.33 (#34/73)
LiveBench Table Join: 10.36 (#35/73)
LiveBench Olympiad: 39.16 (#38/73)
LiveBench Summarize: 55.52 (#38/73)
LiveBench Paraphrase: 59.52 (#41/73)
LiveBench Story Generation: 59.25 (#42/73)
LiveBench Zebra Puzzle: 28.0 (#44/73)
LiveBench Plot Unscrambling: 15.34 (#46/73)
LiveBench Math Comp: 16.67 (#47/73)

Mixtral 8x22B Instruct (v0.1) — ELO 1507, #614/1474, above DeepHermes-3-Mistral-24B-Preview, below Palmyra Med
LiveBench Simplify: 66.87 (#19/73)
LiveBench Table Join: 19.0 (#23/73)
LiveBench Summarize: 63.62 (#26/73)
LiveBench Connections: 25.17 (#27/73)
LiveBench Plot Unscrambling: 28.27 (#28/73)
LiveBench AMPS Hard: 27.0 (#29/73)
LiveBench Story Generation: 68.67 (#29/73)
LiveBench Zebra Puzzle: 34.0 (#29/73)
LiveBench Olympiad: 43.41 (#34/73)
LiveBench Typos: 26.0 (#34/73)

Phi-3-medium-4k-instruct — ELO 1480, #664/1474, above c4ai-command-r-plus, below Yi-34Bx2-MoE-60B-DPO
LiveBench Math Comp: 27.08 (#24/73)
LiveBench AMPS Hard: 29.0 (#26/73)
LiveBench Zebra Puzzle: 30.0 (#39/73)
LiveBench Connections: 14.0 (#40/73)
LiveBench Olympiad: 37.99 (#40/73)
LiveBench Simplify: 55.78 (#42/73)
LiveBench Table Reformat: 36.0 (#43/73)
LiveBench Plot Unscrambling: 15.73 (#44/73)
LiveBench Paraphrase: 56.47 (#47/73)
LiveBench Table Join: 4.88 (#47/73)

Ling-2.6-flash — ELO 1475, #680/1474, above Redemption_Wind_24B, below Smaug-34B-v0.1
Tau3 Banking: 2.89 (#137/150)

Claude 3 Opus — ELO 1468, #691/1474, above Mixtral-8x22B-v0.1, below Command-R+ (08-2024)
BabyVision: 81.2 (#2/24)

Command-R+ (08-2024) — ELO 1468, #692/1474, above Claude 3 Opus, below Bielik-11B-v2.1-Instruct
LiveBench Zebra Puzzle: 38.0 (#15/73)
LiveBench Summarize: 65.87 (#18/73)
LiveBench Typos: 44.0 (#20/73)
LiveBench Table Reformat: 52.0 (#27/73)
LiveBench Connections: 23.67 (#28/73)
LiveBench Simplify: 62.85 (#29/73)
LiveBench Math Comp: 23.96 (#30/73)
LiveBench Paraphrase: 64.43 (#30/73)
LiveBench Plot Unscrambling: 25.19 (#31/73)
LiveBench Table Join: 11.76 (#31/73)

Phi-3-small-8k-instruct — ELO 1455, #715/1474, above Llama 3 8B Instruct V0.8, below NeuralLLaMa-3-8B-ORPO-v0.3
LiveBench Zebra Puzzle: 40.0 (#12/73)
LiveBench Connections: 19.5 (#35/73)
LiveBench Story Generation: 64.08 (#36/73)
LiveBench Table Reformat: 42.0 (#38/73)
LiveBench Math Comp: 18.75 (#41/73)
LiveBench Summarize: 53.22 (#41/73)
LiveBench Plot Unscrambling: 15.89 (#42/73)
LiveBench Olympiad: 36.44 (#43/73)
LiveBench Paraphrase: 57.93 (#45/73)
LiveBench AMPS Hard: 16.0 (#47/73)

Phi-3-medium-128k-instruct — ELO 1452, #717/1474, above NeuralLLaMa-3-8B-ORPO-v0.3, below Magnum-Picaro-0.7-v2-12B
LiveBench Math Comp: 25.0 (#29/73)
LiveBench Table Reformat: 44.0 (#34/73)
LiveBench Paraphrase: 62.07 (#36/73)
LiveBench Story Generation: 63.75 (#37/73)
LiveBench AMPS Hard: 21.0 (#39/73)
LiveBench Simplify: 51.7 (#45/73)
LiveBench Connections: 9.5 (#48/73)
LiveBench Plot Unscrambling: 14.77 (#49/73)
LiveBench Typos: 14.0 (#49/73)
LiveBench Table Join: 4.36 (#50/73)

c4ai-command-r-08-2024 — ELO 1444, #735/1474, above SILMA-9B-Instruct-v1.0, below Lexora-Lite-3B
LiveBench Paraphrase: 69.05 (#15/73)
LiveBench Zebra Puzzle: 34.0 (#24/73)
LiveBench Story Generation: 69.58 (#26/73)
LiveBench Olympiad: 47.29 (#27/73)
LiveBench Simplify: 63.77 (#27/73)
LiveBench Summarize: 60.42 (#31/73)
LiveBench Plot Unscrambling: 18.16 (#35/73)
LiveBench Table Reformat: 42.0 (#36/73)
LiveBench Math Comp: 18.75 (#38/73)
LiveBench Typos: 20.0 (#39/73)

Claude 3 Sonnet (20240229) — ELO 1440, #746/1474, above calme-2.2-qwen2-7B, below calme-2.1-qwen2-7B
LiveBench Typos: 54.0 (#12/73)
LiveBench Summarize: 67.23 (#17/73)
LiveBench Plot Unscrambling: 34.08 (#19/73)
LiveBench Table Reformat: 62.0 (#19/73)
LiveBench Zebra Puzzle: 36.0 (#19/73)
LiveBench Connections: 26.17 (#24/73)
LiveBench Table Join: 15.68 (#24/73)
LiveBench Olympiad: 50.24 (#25/73)
LiveBench Simplify: 64.55 (#25/73)
LiveBench Paraphrase: 65.82 (#28/73)

GPT-4.1 Nano — ELO 1439, #748/1474, above calme-2.1-qwen2-7B, below WizardCoder-Python-34B-V1.0
MCPMark: 0.0 (#39/39)

GPT-3.5 Turbo (0125) — ELO 1433, #756/1474, above Llama-3-Instruct-8B-SimPO-ExPO, below dolphin-2.9.1-yi-1.5-9B
LiveBench Table Join: 19.64 (#22/73)
LiveBench Typos: 42.0 (#25/73)
LiveBench Table Reformat: 54.0 (#26/73)
LiveBench Zebra Puzzle: 34.0 (#27/73)
LiveBench Paraphrase: 65.77 (#29/73)
LiveBench Olympiad: 44.26 (#31/73)
LiveBench Story Generation: 67.08 (#31/73)
LiveBench AMPS Hard: 23.0 (#32/73)
LiveBench Plot Unscrambling: 17.32 (#37/73)
LiveBench Simplify: 57.03 (#39/73)

GPT-3.5 Turbo — ELO 1426, #772/1474, above Samantha-Qwen-2-7B, below Nova Micro
SimpleBench: 8.0 (#86/86)

Llama 3 8B Instruct — ELO 1423, #779/1474, above Llama 3.2 11B Instruct, below gemma-2-9B-it-DPO
LiveBench Connections: 20.33 (#33/73)
LiveBench Paraphrase: 61.88 (#37/73)
LiveBench Zebra Puzzle: 30.0 (#37/73)
LiveBench Summarize: 54.27 (#39/73)
LiveBench Simplify: 56.17 (#41/73)
LiveBench Typos: 20.0 (#41/73)
LiveBench Plot Unscrambling: 15.82 (#43/73)
LiveBench Olympiad: 36.41 (#44/73)
LiveBench Table Reformat: 34.0 (#46/73)
LiveBench Math Comp: 14.58 (#50/73)

Qwen 1.5 72B Chat — ELO 1421, #784/1474, above Gemma-Ko-Merge, below EEVE-Korean-Instruct-10.8B-v1.0
LiveBench Story Generation: 69.08 (#28/73)
LiveBench Olympiad: 43.77 (#32/73)
LiveBench Table Join: 10.94 (#32/73)
LiveBench Zebra Puzzle: 34.0 (#32/73)
LiveBench Plot Unscrambling: 17.94 (#36/73)
LiveBench Summarize: 56.83 (#36/73)
LiveBench Simplify: 56.23 (#40/73)
LiveBench AMPS Hard: 19.0 (#42/73)
LiveBench Table Reformat: 36.0 (#44/73)
LiveBench Math Comp: 17.71 (#45/73)

Hermes 3 - Llama-3.1 70B — ELO 1420, #787/1474, above openbuddy-llama3.1-8B-v22.3-131k, below Nous-Hermes-2-Mixtral-8x7B-DPO
LiveBench Typos: 62.0 (#7/73)
LiveBench Table Reformat: 66.0 (#8/73)
LiveBench Table Join: 26.34 (#15/73)
LiveBench Connections: 32.0 (#16/73)
LiveBench Plot Unscrambling: 37.32 (#16/73)
LiveBench Math Comp: 20.83 (#34/73)
LiveBench Olympiad: 43.12 (#35/73)
LiveBench Zebra Puzzle: 32.0 (#35/73)
LiveBench Simplify: 58.65 (#37/73)
LiveBench AMPS Hard: 21.0 (#38/73)

Kimi K2 (0711) — ELO 1416, #796/1474, above c4ai-command-r-v01, below granite-3.1-8B-instruct
MCPMark: 19.09 (#24/39)

Gemma 2 9B (IT) — ELO 1410, #815/1474, above Gemma 3 4B, below Qwen 1.5 14B
LiveBench Paraphrase: 68.05 (#19/73)
LiveBench AMPS Hard: 28.0 (#27/73)
LiveBench Typos: 38.0 (#27/73)
LiveBench Connections: 23.0 (#29/73)
LiveBench Table Reformat: 50.0 (#30/73)
LiveBench Simplify: 61.77 (#32/73)
LiveBench Plot Unscrambling: 21.93 (#34/73)
LiveBench Summarize: 59.05 (#35/73)
LiveBench Table Join: 5.18 (#46/73)
LiveBench Story Generation: 57.33 (#49/73)

Llama 3.1 8B Instruct — ELO 1402, #835/1474, above Llama-3.2-Rabbit-Ko-3B-Instruct, below Phi-3.5-mini-instruct
LiveBench Paraphrase: 64.03 (#31/73)
LiveBench Table Reformat: 50.0 (#31/73)
LiveBench Typos: 30.0 (#31/73)
LiveBench Simplify: 57.52 (#38/73)
LiveBench Plot Unscrambling: 16.98 (#39/73)
LiveBench Summarize: 54.0 (#40/73)
LiveBench AMPS Hard: 20.0 (#41/73)
LiveBench Connections: 13.17 (#42/73)
LiveBench Math Comp: 17.71 (#44/73)
LiveBench Olympiad: 35.4 (#45/73)

Phi-3.5-mini-instruct — ELO 1402, #836/1474, above Llama 3.1 8B Instruct, below Jamba-v0.1
LiveBench Zebra Puzzle: 34.0 (#30/73)
LiveBench Story Generation: 65.5 (#32/73)
LiveBench Summarize: 59.93 (#32/73)
LiveBench AMPS Hard: 21.0 (#40/73)
LiveBench Simplify: 53.2 (#44/73)
LiveBench Table Join: 5.28 (#45/73)
LiveBench Math Comp: 16.67 (#46/73)
LiveBench Table Reformat: 34.0 (#47/73)
LiveBench Paraphrase: 54.57 (#50/73)
LiveBench Connections: 5.17 (#53/73)

Claude 3 Haiku (20240307) — ELO 1400, #845/1474, above OLMo 3 7B, below dzakwan-MoE-4x7B-Beta
LiveBench Summarize: 69.58 (#12/73)
LiveBench Typos: 48.0 (#15/73)
LiveBench Table Reformat: 62.0 (#18/73)
LiveBench Simplify: 65.1 (#24/73)
LiveBench Table Join: 12.62 (#28/73)
LiveBench Olympiad: 45.38 (#30/73)
LiveBench Plot Unscrambling: 23.72 (#32/73)
LiveBench Paraphrase: 63.52 (#33/73)
LiveBench Zebra Puzzle: 32.0 (#33/73)
LiveBench Math Comp: 19.79 (#36/73)

Mixtral 8x7B Instruct (v0.1) — ELO 1395, #857/1474, above Mengzi3-8B-Chat, below Llama-3-Instruct-8B-SimPO-SPPO-Iter3-merge
LiveBench Connections: 15.0 (#39/73)
LiveBench Plot Unscrambling: 16.28 (#41/73)
LiveBench AMPS Hard: 16.0 (#46/73)
LiveBench Math Comp: 15.62 (#48/73)
LiveBench Table Reformat: 30.0 (#50/73)
LiveBench Olympiad: 30.5 (#55/73)
LiveBench Typos: 10.0 (#57/73)
LiveBench Simplify: 42.75 (#58/73)
LiveBench Table Join: 2.38 (#58/73)
LiveBench Story Generation: 51.08 (#59/73)

Phi-3-small-128k-instruct — ELO 1392, #866/1474, above openbuddy-llama3-8B-v21.1-8k, below Beyonder-4x7B-v3
LiveBench AMPS Hard: 28.0 (#28/73)
LiveBench Story Generation: 64.67 (#35/73)
LiveBench Math Comp: 19.79 (#37/73)
LiveBench Olympiad: 39.12 (#39/73)
LiveBench Table Reformat: 40.0 (#39/73)
LiveBench Typos: 20.0 (#42/73)
LiveBench Zebra Puzzle: 28.0 (#43/73)
LiveBench Connections: 12.83 (#44/73)
LiveBench Simplify: 51.08 (#46/73)
LiveBench Table Join: 3.78 (#52/73)

Qwen 2.5 Turbo — ELO 1389, #881/1474, above Granite 4.0 H 1B, below Llama 3.2 3B Mix Skill
AI Chess Leaderboard (Continuation): 438.0 (#178/237)
AI Chess Leaderboard (Reasoning): 574.0 (#201/292)

DeepSeek Coder V2 Lite Instruct — ELO 1383, #901/1474, above LFM2 2.6B, below Claude Instant
LiveBench AMPS Hard: 33.0 (#18/73)
LiveBench Math Comp: 33.33 (#20/73)
LiveBench Table Reformat: 52.0 (#28/73)
LiveBench Table Join: 9.0 (#37/73)
LiveBench Typos: 20.0 (#40/73)
LiveBench Olympiad: 36.98 (#41/73)
LiveBench Story Generation: 58.25 (#45/73)
LiveBench Zebra Puzzle: 20.0 (#55/73)
LiveBench Paraphrase: 52.85 (#57/73)
LiveBench Simplify: 44.2 (#57/73)

Phi-3 Mini 4K Instruct — ELO 1381, #908/1474, above Llama-3-Base-8B-SFT-RDPO, below Impish_Mind_8B
LiveBench Table Reformat: 38.0 (#42/73)
LiveBench Zebra Puzzle: 28.0 (#42/73)
LiveBench Story Generation: 58.67 (#43/73)
LiveBench Simplify: 50.67 (#48/73)
LiveBench Table Join: 4.64 (#48/73)
LiveBench AMPS Hard: 14.0 (#49/73)
LiveBench Math Comp: 15.62 (#49/73)
LiveBench Olympiad: 32.89 (#50/73)
LiveBench Paraphrase: 53.98 (#53/73)
LiveBench Plot Unscrambling: 13.18 (#55/73)

Mistral Small (2402) — ELO 1375, #930/1474, above Gemini 1.5 Pro (001), below WestLake-7B-v2
LiveBench Math Comp: 27.08 (#23/73)
LiveBench Story Generation: 70.67 (#24/73)
LiveBench Summarize: 63.75 (#25/73)
LiveBench Plot Unscrambling: 26.5 (#29/73)
LiveBench Connections: 21.67 (#31/73)
LiveBench Table Reformat: 44.0 (#33/73)
LiveBench Paraphrase: 62.07 (#35/73)
LiveBench Simplify: 59.15 (#36/73)
LiveBench Olympiad: 41.35 (#37/73)
LiveBench Table Join: 7.64 (#41/73)

Phi-3 Mini 128K Instruct — ELO 1367, #950/1474, above c4ai-command-r7B-12-2024, below Llama-3-Swallow-8B-Instruct-v0.1
LiveBench Table Reformat: 44.0 (#35/73)
LiveBench Zebra Puzzle: 30.0 (#40/73)
LiveBench Table Join: 6.06 (#43/73)
LiveBench Summarize: 49.35 (#47/73)
LiveBench AMPS Hard: 14.0 (#48/73)
LiveBench Paraphrase: 54.15 (#52/73)
LiveBench Simplify: 48.7 (#52/73)
LiveBench Story Generation: 53.42 (#55/73)
LiveBench Olympiad: 29.9 (#57/73)
LiveBench Typos: 10.0 (#58/73)

Qwen 2 7B Instruct — ELO 1355, #978/1474, above Qwen 3 0.6B, below openchat-3.5-0106
LiveBench AMPS Hard: 25.0 (#31/73)
LiveBench Math Comp: 23.96 (#32/73)
LiveBench Table Reformat: 36.0 (#45/73)
LiveBench Connections: 10.0 (#46/73)
LiveBench Plot Unscrambling: 14.62 (#50/73)
LiveBench Olympiad: 31.65 (#52/73)
LiveBench Zebra Puzzle: 24.0 (#52/73)
LiveBench Summarize: 45.02 (#53/73)
LiveBench Table Join: 2.24 (#59/73)
LiveBench Simplify: 40.0 (#63/73)

Command-R+ — ELO 1344, #1008/1474, above internlm2-chat-7B-ExPO, below ConfigurableSOLAR-10.7B
LiveBench Story Generation: 80.83 (#2/73)
LiveBench Zebra Puzzle: 46.0 (#3/73)
LiveBench Simplify: 72.63 (#10/73)
LiveBench Paraphrase: 70.22 (#11/73)
LiveBench Summarize: 62.35 (#28/73)
LiveBench Olympiad: 45.81 (#29/73)
LiveBench Connections: 22.33 (#30/73)
LiveBench Plot Unscrambling: 25.43 (#30/73)
LiveBench Table Join: 11.8 (#30/73)
LiveBench Typos: 24.0 (#36/73)

Nanbeige4.1-3B — ELO 1341, #1012/1474, above neural-chat-7B-v3-3, below Qwen-14B-Chat
GDPval-AA: 141.0 (#141/153)
AA GDPval: -141.11 (#145/148)
Tau3 Banking: 0.0 (#149/150)

zephyr-7B-alpha — ELO 1337, #1020/1474, above Qwen 2.5 3B, below SeaLLM-7B-v2
LiveBench Paraphrase: 57.98 (#44/73)
LiveBench Story Generation: 58.17 (#46/73)
LiveBench Summarize: 47.32 (#50/73)
LiveBench Table Join: 4.2 (#51/73)
LiveBench Simplify: 47.7 (#54/73)
LiveBench Connections: 4.0 (#56/73)
LiveBench Plot Unscrambling: 11.59 (#59/73)
LiveBench Zebra Puzzle: 20.0 (#61/73)
LiveBench Olympiad: 19.62 (#62/73)
LiveBench Table Reformat: 12.0 (#62/73)

OpenHermes-2.5-Mistral-7B — ELO 1327, #1040/1474, above falcon-11B, below CodeNinja-1.0-OpenChat-7B
LiveBench Story Generation: 64.92 (#33/73)
LiveBench Olympiad: 36.8 (#42/73)
LiveBench Table Join: 6.76 (#42/73)
LiveBench Typos: 18.0 (#45/73)
LiveBench Plot Unscrambling: 15.11 (#47/73)
LiveBench Zebra Puzzle: 26.0 (#47/73)
LiveBench AMPS Hard: 11.0 (#52/73)
LiveBench Summarize: 45.68 (#52/73)
LiveBench Table Reformat: 28.0 (#52/73)
LiveBench Math Comp: 13.54 (#54/73)

Starling-LM-7B-beta — ELO 1319, #1057/1474, above Qwen 2 VL 7B Instruct, below llama-3-Korean-8B
LiveBench Connections: 4.5 (#54/73)
LiveBench Plot Unscrambling: 13.29 (#54/73)
LiveBench Zebra Puzzle: 22.0 (#54/73)
LiveBench Math Comp: 11.46 (#56/73)
LiveBench AMPS Hard: 9.0 (#57/73)
LiveBench Olympiad: 24.13 (#61/73)
LiveBench Paraphrase: 47.48 (#64/73)
LiveBench Summarize: 31.08 (#66/73)
LiveBench Simplify: 31.8 (#67/73)
LiveBench Story Generation: 42.92 (#67/73)

Command R — ELO 1314, #1065/1474, above Gemma 3n E4B, below free-solar-evo-v0.13
LiveBench Zebra Puzzle: 38.0 (#14/73)
LiveBench Table Join: 9.08 (#36/73)
LiveBench Typos: 22.0 (#38/73)
LiveBench Story Generation: 63.5 (#39/73)
LiveBench Paraphrase: 59.85 (#40/73)
LiveBench Table Reformat: 38.0 (#40/73)
LiveBench Math Comp: 17.71 (#43/73)
LiveBench Simplify: 53.83 (#43/73)
LiveBench Summarize: 51.47 (#45/73)
LiveBench Connections: 7.67 (#49/73)

Phi-4 Mini Instruct — ELO 1291, #1099/1474, above Claude Instant 1.1, below Llama 3 8B 4bit UltraChat Ita
Tau3 Banking: 1.03 (#144/150)

Gemma 1.1 7B (IT) — ELO 1249, #1181/1474, above Yi-1.5-6B-Chat, below Yi-9B-200K
LiveBench Connections: 7.17 (#50/73)
LiveBench Typos: 12.0 (#51/73)
LiveBench Olympiad: 31.48 (#53/73)
LiveBench AMPS Hard: 10.0 (#54/73)
LiveBench Plot Unscrambling: 12.78 (#57/73)
LiveBench Table Join: 2.52 (#57/73)
LiveBench Zebra Puzzle: 20.0 (#57/73)
LiveBench Paraphrase: 51.35 (#59/73)
LiveBench Summarize: 40.72 (#59/73)
LiveBench Simplify: 41.78 (#60/73)

Mistral 7B Instruct (v0.3) — ELO 1230, #1206/1474, above Yi-1.5-9B-32K, below aya-23-8B
LiveBench Plot Unscrambling: 15.39 (#45/73)
LiveBench Olympiad: 33.48 (#48/73)
LiveBench Paraphrase: 56.02 (#48/73)
LiveBench Typos: 14.0 (#48/73)
LiveBench Simplify: 50.03 (#49/73)
LiveBench Summarize: 47.75 (#49/73)
LiveBench Connections: 6.17 (#52/73)
LiveBench Story Generation: 55.67 (#52/73)
LiveBench Table Reformat: 18.0 (#58/73)
LiveBench AMPS Hard: 5.0 (#61/73)

Mistral 7B Instruct (v0.2) — ELO 1223, #1216/1474, above granite-3.0-3B-a800m-instruct, below Qwen 2 1.5B Instruct
LiveBench Summarize: 51.75 (#43/73)
LiveBench Story Generation: 58.42 (#44/73)
LiveBench Olympiad: 34.87 (#46/73)
LiveBench Connections: 6.17 (#51/73)
LiveBench Paraphrase: 54.52 (#51/73)
LiveBench Plot Unscrambling: 13.0 (#56/73)
LiveBench Simplify: 41.92 (#59/73)
LiveBench AMPS Hard: 7.0 (#60/73)
LiveBench Math Comp: 9.38 (#60/73)
LiveBench Typos: 8.0 (#60/73)

Qwen 2 1.5B Instruct — ELO 1221, #1217/1474, above Mistral 7B Instruct (v0.2), below deepseek-moe-16B-chat
LiveBench Table Join: 2.02 (#60/73)
LiveBench AMPS Hard: 5.0 (#62/73)
LiveBench Math Comp: 8.33 (#62/73)
LiveBench Table Reformat: 8.0 (#64/73)
LiveBench Zebra Puzzle: 16.0 (#65/73)
LiveBench Olympiad: 16.49 (#67/73)
LiveBench Story Generation: 40.42 (#68/73)
LiveBench Paraphrase: 26.2 (#70/73)
LiveBench Connections: 0.0 (#71/73)
LiveBench Plot Unscrambling: 9.14 (#71/73)

Qwen 1.5 4B Chat — ELO 1185, #1245/1474, above AryaBhatta-GemmaOrca-Merged, below trurl-2-13B-academic
LiveBench Zebra Puzzle: 26.0 (#48/73)
LiveBench Table Join: 3.38 (#54/73)
LiveBench Math Comp: 10.42 (#58/73)
LiveBench Typos: 8.0 (#62/73)
LiveBench AMPS Hard: 4.0 (#63/73)
LiveBench Table Reformat: 6.0 (#66/73)
LiveBench Olympiad: 15.15 (#68/73)
LiveBench Paraphrase: 31.62 (#68/73)
LiveBench Connections: 0.0 (#69/73)
LiveBench Simplify: 22.65 (#69/73)

Qwen 1.5 7B Chat — ELO 1183, #1248/1474, above granite-3.0-2B-instruct, below zephyr-7B-beta
LiveBench Zebra Puzzle: 22.0 (#53/73)
LiveBench AMPS Hard: 10.0 (#55/73)
LiveBench Table Join: 2.7 (#56/73)
LiveBench Connections: 3.5 (#57/73)
LiveBench Summarize: 41.77 (#57/73)
LiveBench Olympiad: 28.59 (#59/73)
LiveBench Paraphrase: 50.65 (#61/73)
LiveBench Plot Unscrambling: 11.04 (#62/73)
LiveBench Story Generation: 46.67 (#64/73)
LiveBench Math Comp: 7.29 (#65/73)

zephyr-7B-beta — ELO 1183, #1249/1474, above Qwen 1.5 7B Chat, below internlm2-base-7B
LiveBench Simplify: 49.62 (#51/73)
LiveBench Math Comp: 11.46 (#57/73)
LiveBench Paraphrase: 51.77 (#58/73)
LiveBench Story Generation: 52.17 (#58/73)
LiveBench Table Reformat: 14.0 (#59/73)
LiveBench Summarize: 39.72 (#60/73)
LiveBench Plot Unscrambling: 10.85 (#64/73)
LiveBench Table Join: 1.24 (#64/73)
LiveBench AMPS Hard: 4.0 (#65/73)
LiveBench Olympiad: 18.23 (#65/73)

Llama 2 7B Chat — ELO 1131, #1306/1474, above Llama-3-Base-8B-SFT-CPO, below tulu-2-13B
LiveBench Paraphrase: 55.12 (#49/73)
LiveBench Typos: 10.0 (#56/73)
LiveBench Simplify: 40.27 (#61/73)
LiveBench Summarize: 38.07 (#62/73)
LiveBench Story Generation: 46.08 (#65/73)
LiveBench Connections: 0.0 (#66/73)
LiveBench Plot Unscrambling: 10.58 (#66/73)
LiveBench Table Join: 0.0 (#67/73)
LiveBench Olympiad: 14.35 (#69/73)
LiveBench Zebra Puzzle: 10.0 (#70/73)

Yi 6B (Chat) — ELO 1130, #1308/1474, above tulu-2-13B, below tulu-2-7B
LiveBench Table Reformat: 12.0 (#61/73)
LiveBench Olympiad: 19.42 (#63/73)
LiveBench Table Join: 1.14 (#65/73)
LiveBench Plot Unscrambling: 10.07 (#67/73)
LiveBench AMPS Hard: 2.0 (#68/73)
LiveBench Summarize: 25.7 (#68/73)
LiveBench Typos: 4.0 (#68/73)
LiveBench Story Generation: 39.5 (#69/73)
LiveBench Zebra Puzzle: 12.0 (#69/73)
LiveBench Math Comp: 4.17 (#71/73)

vicuna-7B-v1.5 — ELO 1074, #1338/1474, above SmolLM-360M, below Teuken-7B-Instruct-v0.4
LiveBench Typos: 14.0 (#50/73)
LiveBench Story Generation: 50.67 (#60/73)
LiveBench Simplify: 40.22 (#62/73)
LiveBench Connections: 1.0 (#63/73)
LiveBench Math Comp: 8.33 (#63/73)
LiveBench Plot Unscrambling: 10.98 (#63/73)
LiveBench Summarize: 32.77 (#64/73)
LiveBench Paraphrase: 43.37 (#67/73)
LiveBench Zebra Puzzle: 14.0 (#67/73)
LiveBench AMPS Hard: 1.0 (#69/73)

vicuna-7B-v1.5-16k — ELO 1036, #1363/1474, above gpt-sw3-40B, below SmolLM-1.7B
LiveBench Story Generation: 53.5 (#54/73)
LiveBench Typos: 12.0 (#55/73)
LiveBench Zebra Puzzle: 20.0 (#60/73)
LiveBench Paraphrase: 49.82 (#62/73)
LiveBench Table Join: 1.8 (#62/73)
LiveBench Connections: 1.0 (#64/73)
LiveBench Olympiad: 18.84 (#64/73)
LiveBench Plot Unscrambling: 10.75 (#65/73)
LiveBench Summarize: 32.07 (#65/73)
LiveBench Table Reformat: 8.0 (#65/73)

Qwen 1.5 1.8B Chat — ELO 965, #1394/1474, above Llama 3.2 1B Instruct, below trurl-2-7B
LiveBench Connections: 0.0 (#68/73)
LiveBench Plot Unscrambling: 9.47 (#69/73)
LiveBench Table Reformat: 4.0 (#69/73)
LiveBench Math Comp: 4.17 (#70/73)
LiveBench Simplify: 21.57 (#70/73)
LiveBench Table Join: 0.0 (#70/73)
LiveBench Paraphrase: 23.4 (#71/73)
LiveBench Typos: 0.0 (#71/73)
LiveBench Zebra Puzzle: 10.0 (#71/73)
LiveBench Summarize: 23.48 (#72/73)

Qwen 2 0.5B Instruct — ELO 958, #1399/1474, above INTELLECT-1-Instruct, below ALMA-13B-R
LiveBench Math Comp: 9.38 (#61/73)
LiveBench AMPS Hard: 3.0 (#67/73)
LiveBench Simplify: 23.37 (#68/73)
LiveBench Paraphrase: 30.9 (#69/73)
LiveBench Connections: 0.0 (#70/73)
LiveBench Story Generation: 28.08 (#71/73)
LiveBench Summarize: 24.17 (#71/73)
LiveBench Table Join: 0.0 (#71/73)
LiveBench Olympiad: 9.66 (#72/73)
LiveBench Typos: 0.0 (#72/73)

Qwen 1.5 0.5B Chat — ELO 814, #1448/1474, above DBRX Instruct, below falcon-7B
LiveBench Connections: 0.0 (#67/73)
LiveBench Table Join: 0.0 (#69/73)
LiveBench Summarize: 24.95 (#70/73)
LiveBench Typos: 0.0 (#70/73)
LiveBench Olympiad: 10.16 (#71/73)
LiveBench AMPS Hard: 0.0 (#72/73)
LiveBench Math Comp: 3.12 (#72/73)
LiveBench Plot Unscrambling: 8.63 (#72/73)
LiveBench Story Generation: 25.83 (#72/73)
LiveBench Table Reformat: 0.0 (#72/73)

New Scores From Top-10 Models (121)

Claude Fable 5 on KernelBench Hub - Hard: 24.0 Best % of Hardware Roofline (#4/13)
Claude Fable 5 on KernelBench Hub - Mega: 18.7 Best Speedup vs Reference (x) (#2/12)
Claude Fable 5 on Lean AI Formalization Leaderboard: 4.0 Solved Problems (#13/35)
Claude Fable 5 on TaxCalcBench: 26.0 Correct Returns - Strict (%) (#5/9)
Claude Fable 5 on TrackingAI IQ Test (Mensa Norway): 60.0 Mensa Norway Score (%) (#20/31)
Claude Fable 5 on TrackingAI IQ Test (Vision): 78.57 IQ Test Score (%) (#3/19)
Claude Opus 4.8 on TaxCalcBench: 16.0 Correct Returns - Strict (%) (#7/9)
GPT-5.6 Sol on AA GDPval: 1747.82 ELO (#2/148)
GPT-5.6 Sol on AA GPQA Diamond: 94.14 Accuracy (%) (#1/537)
GPT-5.6 Sol on AA Humanity's Last Exam: 47.17 Accuracy (%) (#2/533)
GPT-5.6 Sol on AA IFBench: 72.65 Accuracy (%) (#47/449)
GPT-5.6 Sol on AA Long Context Reasoning: 73.67 Accuracy (%) (#12/465)
GPT-5.6 Sol on AA MMMU-Pro: 83.41 Accuracy (%) (#3/223)
GPT-5.6 Sol on AA Omniscience: 21.7 Score (#7/442)
GPT-5.6 Sol on AA Omniscience - Business: 50.6 Accuracy (%) (#3/442)
GPT-5.6 Sol on AA Omniscience - Health: 50.5 Accuracy (%) (#2/442)
GPT-5.6 Sol on AA Omniscience - Humanities & Social Sciences: 58.0 Accuracy (%) (#2/442)
GPT-5.6 Sol on AA Omniscience - Law: 53.5 Accuracy (%) (#8/442)
GPT-5.6 Sol on AA Omniscience - Science, Engineering & Mathematics: 54.1 Accuracy (%) (#2/442)
GPT-5.6 Sol on AA Omniscience - Software Engineering (SWE): 84.5 Accuracy (%) (#2/442)
GPT-5.6 Sol on AA Omniscience - Software Engineering (SWE) - C: 88.0 Accuracy (%) (#6/442)
GPT-5.6 Sol on AA Omniscience - Software Engineering (SWE) - Dart: 76.0 Accuracy (%) (#7/442)
GPT-5.6 Sol on AA Omniscience - Software Engineering (SWE) - Go: 82.0 Accuracy (%) (#8/442)
GPT-5.6 Sol on AA Omniscience - Software Engineering (SWE) - Java: 72.0 Accuracy (%) (#3/442)
GPT-5.6 Sol on AA Omniscience - Software Engineering (SWE) - JavaScript: 89.09 Accuracy (%) (#2/442)
GPT-5.6 Sol on AA Omniscience - Software Engineering (SWE) - Julia: 56.0 Accuracy (%) (#34/442)
GPT-5.6 Sol on AA Omniscience - Software Engineering (SWE) - Kotlin: 88.0 Accuracy (%) (#3/442)
GPT-5.6 Sol on AA Omniscience - Software Engineering (SWE) - Python: 88.5 Accuracy (%) (#5/442)
GPT-5.6 Sol on AA Omniscience - Software Engineering (SWE) - R: 66.0 Accuracy (%) (#12/442)
GPT-5.6 Sol on AA Omniscience - Software Engineering (SWE) - Rust: 84.0 Accuracy (%) (#8/442)
GPT-5.6 Sol on AA Omniscience - Software Engineering (SWE) - Swift: 84.0 Accuracy (%) (#6/442)
GPT-5.6 Sol on AA SciCode: 56.13 Accuracy (%) (#8/531)
GPT-5.6 Sol on AA TAU-2 Bench: 85.09 Accuracy (%) (#99/439)
GPT-5.6 Sol on AA-Omniscience Accuracy: 58.53 Accuracy (%) (#2/442)
GPT-5.6 Sol on AI Chess Leaderboard (Continuation): 1213.0 Elo (#25/237)
GPT-5.6 Sol on AI Chess Leaderboard (Reasoning): 1611.0 Elo (#11/292)
GPT-5.6 Sol on ARC-AGI-1: 96.5 Accuracy (%) (#7/163)
GPT-5.6 Sol on ARC-AGI-2: 92.5 Accuracy (%) (#2/165)
GPT-5.6 Sol on Artificial Analysis Intelligence Index: 58.89 Intelligence Index (#2/536)
GPT-5.6 Sol on BridgeBench Debugging: 80.1 Score (#24/44)
GPT-5.6 Sol on BridgeBench Hallucination: 76.8 Score (#9/65)
GPT-5.6 Sol on BridgeBench Refactoring: 64.7 Score (#18/35)
GPT-5.6 Sol on BridgeBench Security: 70.6 Score (#14/35)
GPT-5.6 Sol on BridgeBench UI: 77.6 Score (#11/44)
GPT-5.6 Sol on Bullshit Benchmark: 27.3 BS Detection Rate (%) (#59/158)
GPT-5.6 Sol on Chess Puzzles (Epoch AI): 64.0 Accuracy (%) (#1/55)
GPT-5.6 Sol on Chess Puzzles (Epoch AI): 64.0 Accuracy (%) (#1/55)
GPT-5.6 Sol on CursorBench 3.1: 67.2 Score (%) (#3/35)
GPT-5.6 Sol on Design Arena (3D): 1370.0 Elo (#2/125)
GPT-5.6 Sol on Design Arena (Data Viz): 1344.0 Elo (#2/132)
GPT-5.6 Sol on Design Arena (Game Dev): 1378.0 Elo (#2/138)
GPT-5.6 Sol on Design Arena (UI Components): 1413.0 Elo (#1/128)
GPT-5.6 Sol on Design Arena (Website): 1344.0 Elo (#2/152)
GPT-5.6 Sol on Epoch AI - Cursorbench: 67.2 Score (#4/28)
GPT-5.6 Sol on Epoch AI - Scicode: 56.13 Score (#7/69)
GPT-5.6 Sol on FrontierMath - Tier 4 (v2): 82.93 Accuracy (%, 41 private v2 problems) (#2/38)
GPT-5.6 Sol on FrontierMath - Tier 4 (v2): 82.93 Accuracy (%, 41 private v2 problems) (#2/38)
GPT-5.6 Sol on GDPval-AA: 1748.0 Elo (#2/153)
GPT-5.6 Sol on Gert Labs Rankings: 73.04 GScore (%) (#1/82)
GPT-5.6 Sol on KernelBench Hub - Mega: 1.1 Best Speedup vs Reference (x) (#11/12)
GPT-5.6 Sol on LisanBench: 0.0721 Mean Path Length / Current Maximum (#45/150)
GPT-5.6 Sol on Mercor APEX: 40.0 Best Pass@1 (%) (#4/51)
GPT-5.6 Sol on Mercor APEX: 40.0 Best Pass@1 (%) (#4/51)
GPT-5.6 Sol on OTIS Mock AIME 2024-25: 100.0 Accuracy (%) (#1/153)
GPT-5.6 Sol on PM-LLM-Benchmark: 36.3 Score (#17/158)
GPT-5.6 Sol on PinchBench: 87.0 Success Rate (%) (#22/58)
GPT-5.6 Sol on Ramp SWE-Bench: 82.28 Resolved (%) (#4/24)
GPT-5.6 Sol on Senior SWE-Bench: 24.4 Tasteful Solve Rate (pass@1, %) (#3/10)
GPT-5.6 Sol on SimpleBench: 64.8 Score (AVG@5) (#12/86)
GPT-5.6 Sol on SimpleQA Verified: 71.6 Accuracy (%) (#3/63)
GPT-5.6 Sol on SvelteBench: 94.4 Average pass@1 (%) (#26/145)
GPT-5.6 Sol on TaxCalcBench: 26.0 Correct Returns - Strict (%) (#4/9)
GPT-5.6 Sol on Vals AI (Vals Index): 72.63 Accuracy (%) (#2/35)
GPT-5.6 Sol on Vals AI Code Migration: 52.92 Accuracy (%) (#2/28)
GPT-5.6 Sol on Vals AI CorpFin v2: 64.38 Accuracy (%) (#31/122)
GPT-5.6 Sol on Vals AI Finance Agent v2: 53.76 Accuracy (%) (#6/33)
GPT-5.6 Sol on Vals AI GPQA: 95.2 Accuracy (%) (#2/121)
GPT-5.6 Sol on Vals AI Harvey Legal Agent Bench: 2.5 Accuracy (%) (#12/19)
GPT-5.6 Sol on Vals AI LegalBench: 86.97 Accuracy (%) (#4/124)
GPT-5.6 Sol on Vals AI LiveCodeBench: 82.6 Accuracy (%) (#38/127)
GPT-5.6 Sol on Vals AI MMLU-Pro: 89.1 Accuracy (%) (#10/120)
GPT-5.6 Sol on Vals AI MMMU: 88.84 Accuracy (%) (#2/82)
GPT-5.6 Sol on Vals AI MedCode: 43.97 Accuracy (%) (#23/72)
GPT-5.6 Sol on Vals AI MedScribe: 85.23 Accuracy (%) (#12/70)
GPT-5.6 Sol on Vals AI MortgageTax: 67.29 Accuracy (%) (#23/85)
GPT-5.6 Sol on Vals AI Multimodal Index: 72.19 Accuracy (%) (#2/26)
GPT-5.6 Sol on Vals AI ProofBench: 77.0 Accuracy (%) (#1/49)
GPT-5.6 Sol on Vals AI Public Benefits Bench: 66.51 Accuracy (%) (#2/16)
GPT-5.6 Sol on Vals AI SAGE: 52.56 Accuracy (%) (#4/67)
GPT-5.6 Sol on Vals AI TaxEval v2: 74.78 Accuracy (%) (#20/127)
GPT-5.6 Sol on Vals AI Vibe Code Bench: 80.5 Accuracy (%) (#4/72)
GPT-5.6 Sol on Vending-Bench 2: 9619.37 Money Balance ($) (#2/54)
GPT-5.6 Sol on YC-Bench: 685.9 Net Worth ($K) (#16/29)
GPT-5.6 Sol on You're Absolutely Right!: 2.875 Average anti-sycophancy score (1-5) (#8/21)
Muse Spark 1.1 on Chatbot Arena (Code): 1540.0 Elo (#10/96)
Muse Spark 1.1 on Chatbot Arena (Text): 1490.0 Elo (#6/374)
Muse Spark 1.1 on SEAL - Fortress: 12.4 Score (#54/56)
Muse Spark 1.1 on SEAL - MultiChallenge: 75.3 Score (#2/30)
Muse Spark 1.1 on SvelteBench: 92.2 Average pass@1 (%) (#36/145)
Muse Spark 1.1 on Vals AI (Vals Index): 68.41 Accuracy (%) (#5/35)
Muse Spark 1.1 on Vals AI Code Migration: 31.11 Accuracy (%) (#12/28)
Muse Spark 1.1 on Vals AI CorpFin v2: 71.29 Accuracy (%) (#2/122)
Muse Spark 1.1 on Vals AI Excel Modeling: 56.36 Accuracy (%) (#11/21)
Muse Spark 1.1 on Vals AI Finance Agent v2: 57.21 Accuracy (%) (#2/33)
Muse Spark 1.1 on Vals AI GPQA: 91.16 Accuracy (%) (#13/121)
Muse Spark 1.1 on Vals AI Legal Research Bench: 37.98 Accuracy (%) (#7/18)
Muse Spark 1.1 on Vals AI LegalBench: 84.98 Accuracy (%) (#16/124)
Muse Spark 1.1 on Vals AI LiveCodeBench: 85.91 Accuracy (%) (#18/127)
Muse Spark 1.1 on Vals AI MMLU-Pro: 88.73 Accuracy (%) (#11/120)
Muse Spark 1.1 on Vals AI MMMU: 86.59 Accuracy (%) (#11/82)
Muse Spark 1.1 on Vals AI MortgageTax: 66.14 Accuracy (%) (#30/85)
Muse Spark 1.1 on Vals AI Multimodal Index: 66.74 Accuracy (%) (#7/26)
Muse Spark 1.1 on Vals AI ProgramBench: 16.16 Raw Pass Rate (%) (#25/29)
Muse Spark 1.1 on Vals AI ProofBench: 39.0 Accuracy (%) (#13/49)
Muse Spark 1.1 on Vals AI Public Benefits Bench: 61.3 Accuracy (%) (#9/16)
Muse Spark 1.1 on Vals AI SAGE: 45.58 Accuracy (%) (#22/67)
Muse Spark 1.1 on Vals AI SWE-bench Verified: 82.0 Resolved (%) (#8/69)
Muse Spark 1.1 on Vals AI Terminal-Bench 2.1: 69.29 Accuracy (%) (#11/41)
Muse Spark 1.1 on Vals AI Vibe Code Bench: 72.16 Accuracy (%) (#7/72)
Seed 2.1 Pro on Agents' Last Exam: 19.5 Pass Rate (%) (#11/20)
Seed 2.1 Pro on BabyVision: 73.7 Score (%) (#4/24)

New #1 Leaders (66)

PredictionArena (Kalshi) (Account Value ($)): Claude Opus 4.6 (221482.25) beat GLM-5 (17851.95) by 203630.3.
RuneBench (Total Peak XP Rate (XP/min)): GPT-5.6 Sol (7400.0) beat GPT-5.5 (6370.0) by 1030.0.
VoxelBench (Rating): GPT-5.6 Sol (2306.0) beat Claude Fable 5 (Max) (2223.0) by 83.0.
WDCD R3 Pressure Integrity (Score (%)): Gemini 3.1 Pro (Preview) (138.0) beat Grok 4 (113.0) by 25.0.
BabyVision (Score (%)): GPT-5.5 (83.6) beat Seed 2.0 Pro (High) (60.6) by 23.0.
SEAL - VisualToolBench (VTB) (Score): Muse Spark 1.1 (44.77) beat GPT-5.4 (29.17) by 15.6.
Vals AI IOI (Accuracy (%)): GPT-5.6 Sol (86.67) beat Claude Fable 5 (72.25) by 14.42.
Benchmarks.bio - scBench-Long (Pass Rate (%)): GPT-5.6 Sol (42.5) beat Claude Opus 4.8 (28.57) by 13.93.
LiveBench Simplify (Score): Gemini 1.5 Flash (Preview 0827) (84.75) beat Gemini 3.1 Pro (Preview) (High) (71.05) by 13.7.
TabArena All Tasks (Score (%)): TabFM (default) (89.1) beat TabPFN-3 (default) (77.4) by 11.7.
Opus Magnum Bench (Human-normalized score (%)): GPT-5.6 Sol (xHigh) (70.86) beat Claude Fable 5 (High) (60.15) by 10.71.
Vals AI Harvey Legal Agent Bench (Accuracy (%)): Muse Spark 1.1 (20.0) beat Claude Fable 5 (11.25) by 8.75.
SEAL - SWE Atlas - Codebase QnA (Score): Claude Opus 4.8 (Claude Code) xhigh (57.26) beat Claude Opus 4.8 (Claude Code) (48.79) by 8.47.
Vals AI CyberBench (Accuracy (%)): GPT-5.6 Sol (88.14) beat GPT-5.5 (80.51) by 7.63.
WolfBench (Average Score (%)): GPT-5.6 Sol (85.0) beat GPT-5.5 (79.0) by 6.0.
Vals AI Terminal-Bench 2.1 (Accuracy (%)): GPT-5.6 Sol (85.77) beat Claude Fable 5 (80.52) by 5.25.
Senior SWE-Bench (Tasteful Solve Rate (pass@1, %)): Claude Fable 5 (29.1) beat Claude Opus 4.8 (24.0) by 5.1.
SEAL - Professional Reasoning Benchmark - Legal (Score): Muse Spark 1.1 (57.05) beat Muse Spark (52.29) by 4.76.
PIIMB PII Masking - OpenPII F2 (OpenPII F2 (%)): OpenMed privacy-filter-nemotron-v2 (97.63) beat hivetrace gliner-guard-omni (93.05) by 4.58.
SEAL - MCP Atlas (Score): Muse Spark 1.1 (88.1) beat Gemini 3.5 Flash (High) (83.6) by 4.5.
PIIMB PII Masking - Avg F1 (Average F1 (%)): OpenMed privacy-filter-multilingual-v2 (92.38) beat OpenMed OpenMed-PII-SuperClinical-Large-434M-v1 (87.95) by 4.43.
SEAL - SWE-Bench Pro (Private Dataset) (Score): Muse Spark 1.1 (51.5) beat Claude Opus 4.6 (Thinking) (47.1) by 4.4.
Vals AI Legal Research Bench (Accuracy (%)): GPT-5.6 Sol (48.08) beat Claude Opus 4.8 (43.75) by 4.33.
KernelBench Hub - Hard (Best % of Hardware Roofline): GPT-5.6 Sol (30.0) beat GLM-5.2 (26.0) by 4.0.
AA Omniscience - Software Engineering (SWE) - Kotlin (Accuracy (%)): GPT-5.6 Sol (xHigh) (94.0) beat GPT-5.3 Codex (xHigh) (90.0) by 4.0.
AA Omniscience - Software Engineering (SWE) - PHP (Accuracy (%)): GPT-5.6 Sol (96.0) beat GPT-5.5 (High) (92.0) by 4.0.
AA CritPt (Accuracy (%)): GPT-5.6 Sol (32.29) beat Claude Fable 5 (Adaptive Reasoning, Max Effort, Opus 4.8 Fallback) (28.57) by 3.72.
Agents' Last Exam (Pass Rate (%)): GPT-5 Sol (30.6) beat Claude Opus 4.8 (27.0) by 3.6.
AA Terminal-Bench Hard (Accuracy (%)): GPT-5.6 Sol (65.91) beat Claude Fable 5 (Adaptive Reasoning, Max Effort, Opus 4.8 Fallback) (62.88) by 3.03.
SWE-Marathon (Pass@1 (%)): Grok 4.5 (29.0) beat Claude Opus 4.8 (26.0) by 3.0.
PIIMB PII Masking - Gretel F2 (Gretel F2 (%)): OpenMed privacy-filter-nemotron-v2 (98.28) beat OpenMed OpenMed-PII-SuperClinical-Small-44M-v1 (95.29) by 2.99.
Vals AI Excel Modeling (Accuracy (%)): GPT-5.6 Sol (72.34) beat Claude Opus 4.8 (69.37) by 2.97.
PIIMB PII Masking - Avg F2 (Average F2 (%)): OpenMed privacy-filter-multilingual-v2 (91.68) beat OpenMed OpenMed-PII-SuperClinical-Large-434M-v1 (88.74) by 2.94.
GSO-Bench (Opt@1 (%)): Claude Opus 4.8 (47.06) beat Claude Opus 4.7 (44.12) by 2.94.
DeepSWE (Pass@1 (%)): GPT-5 Sol (72.7) beat Claude Fable 5 (69.9) by 2.8.
Open Universal Arabic ASR Leaderboard (Average WER (lower is better)): cohere-transcribe-arabic-07-2026 (25.87) beat omniASR_LLM_7B (28.32) by 2.45.
SEAL - SWE-Bench Pro (Public Dataset) (Score): Muse Spark 1.1 (61.5) beat GPT-5.4 (xHigh) (59.1) by 2.4.
PM-LLM-Benchmark (Score): GPT-5.6 Sol (xHigh) (40.2) beat Kimi K2.6 (37.8) by 2.4.
LiveBench Paraphrase (Score): Llama 3.1 70B Instruct (82.25) beat Gemini 3.1 Pro (Preview) (High) (79.87) by 2.38.
LiveBench Story Generation (Score): Gemini 1.5 Flash (Preview 0827) (81.33) beat GPT-5.4 (xHigh) (79.02) by 2.31.
Vals AI TaxEval v2 (Accuracy (%)): Muse Spark 1.1 (79.72) beat Muse Spark (77.68) by 2.04.
AA Omniscience - Software Engineering (SWE) - HTML (Accuracy (%)): GPT-5.6 Sol (92.0) beat GPT-5.5 (High) (90.0) by 2.0.
BenchLM (Overall Score): Claude Fable 5 (91.0) beat Claude Mythos 5 (89.0) by 2.0.
AA Omniscience - Health (Accuracy (%)): GPT-5.6 Sol (High) (50.7) beat Grok Build 0.1 (48.9) by 1.8.
Epoch AI - Critpt (Score): GPT-5.6 Sol (32.3) beat GPT-5.5 Pro (xHigh) (30.57) by 1.73.
SEAL - Professional Reasoning Benchmark - Finance (Score): Muse Spark 1.1 (55.01) beat Claude Opus 4.6 (Non-reasoning) (53.28) by 1.73.
Tau3 Banking (Success Rate (%)): GPT-5.6 Sol (32.99) beat GPT-5.5 (xHigh) (31.34) by 1.65.
AMA-Bench - Embodied AI (Average Score (%)): VeraKV (Qwen3-32B) (58.09) beat GPT-5.2 (56.44) by 1.65.
FrontierMath - Tiers 1-3 (v2) (Accuracy (%, 285 private v2 problems)): GPT-5.6 Sol (89.12) beat GPT-5.5 Pro (xHigh) (87.72) by 1.4.
TrackingAI IQ Test (IQ Test Score (%)): Claude Fable 5 (97.3) beat GPT-5 Pro (96.08) by 1.22.
Vals AI SWE-bench Verified (Resolved (%)): GPT-5.6 Sol (96.2) beat Claude Fable 5 (95.0) by 1.2.
AA Omniscience - Software Engineering (SWE) - TypeScript (Accuracy (%)): GPT-5.6 Sol (92.22) beat GPT-5.5 (xHigh) (91.11) by 1.11.
WorldScore - Static (Static Score): World Dreamer (84.52) beat EvoPhys-World (83.45) by 1.07.
Agent Arena - Confirmed Success (Confirmed Success (%)): MiniMax-M2.7 (17.12) beat Claude Fable 5 (High) (16.12) by 1.0.
WeirdML (Average Score): GPT-5.6 Sol (High) (88.76) beat Claude Fable 5 (High) (87.85) by 0.91.
Vals AI ProgramBench (Raw Pass Rate (%)): GPT-5.6 Sol (77.64) beat Claude Fable 5 (76.8) by 0.84.
WorldScore (Average Static/Dynamic Score): World Dreamer (79.44) beat EvoPhys-World (78.62) by 0.82.
AutomationBench (Pass Rate (%)): GPT-5.6 Sol (18.1) beat Claude Fable 5.0 (Max) (17.4) by 0.7.
GDP.pdf (Strict Pass Rate (%)): GPT-5.6 Sol (30.7) beat Claude Fable 5 (30.0) by 0.7.
Epoch AI - ECI (ECI Score): GPT-5.6 Sol (161.61) beat GPT-5.5 Pro (xHigh) (161.01) by 0.6.
Open Universal Arabic ASR Leaderboard - CER (Average CER (lower is better)): cohere-transcribe-arabic-07-2026 (11.8) beat Qwen3-ASR-1.7B (12.33) by 0.53.
SEAL - MultiNRC (Score): Muse Spark 1.1 (65.59) beat GPT-5 Pro (65.2) by 0.39.
WorldCupBench - Brier Skill (100 - Brier Total): DeepSeek V4 Pro (85.1919) beat MiMo-V2.5-Pro (84.82) by 0.37.
Vals AI MedScribe (Accuracy (%)): Muse Spark 1.1 (88.89) beat Claude Fable 5 (88.52) by 0.37.
WorldScore - Dynamic (Dynamic Score): World Dreamer (74.35) beat Inspatio-World (74.1) by 0.25.
Agent Arena - Tool Hallucination (Tool Hallucination (%)): Gemini 3 Flash (0.1) beat DeepSeek V4 Pro (0.12) by 0.02.

                                Don't miss what's next. Subscribe to Mikhail Doroshenko:

            Email address (required)

                    ← Newer

                AI Benchmark Digest — 2026-07-13

                    Older →

                AI Benchmark Digest — 2026-07-11