AI Benchmark Digest — 2026-07-04

View on AI Benchmark Hub

Daily

New Benchmarks (292)

• HarmVideoBench (Macro Avg. (self-reported)): HarmVideoBench (ours) leads with 84.4 across 21 models. Large vision-language models (LVLMs) have recently shown immense potential in automated content moderation, sparking growing interest in developing harmful-video benchmarks. However, we identify two primary limitations in existing works: 1) The multi-layered characteristics of harmful videos are overlooked. Existing benchmarks predominantly formulate evaluation as a binary classification task, fai
• LibEvoBench (SEUS (self-reported)): GPT-5.4 leads with 86.0 across 13 models. Large software projects often depend on older versions of libraries, even as APIs continue to evolve across releases. This creates a challenge for LLMs: they must maintain knowledge of multiple API versions, not merely the latest or most common one. However, current LLMs are trained on temporally mixed corpora and lack explicit mechanisms for such version-specific reasoning, leading to anachronist
• Age of LLM (Points per match (self-reported)): GPT-5.5 leads with 3.0 across 15 models. We introduce Age of LLM, a turn-based 1v1 benchmark in which two LLMs face off on a 13x7 grid to destroy the enemy base. Three stressors are deliberate: fog of war, full diplomacy (messages, ceasefires, ultimatums; uranium kept secret), and a reliability dimension where every turn must follow a strict JSON schema and an illegal action is silently discarded. The engine is private and each match use
• AGORA (Overall (self-reported)): Gemini 3.1 Pro Preview leads with 59.39 across 8 models. Large language models are increasingly deployed as agents that reason over documents rather than answer from parametric knowledge. We study archive-grounded reasoning: locating sparse evidence across a large, messy collection of workplace files, reconciling inconsistent terminology, units, and time conventions, and computing an answer. Existing benchmarks address only parts of this setting and non
• BehaviorBench (Game Behav. Sim. (W ↓) (self-reported)): GPT-5.4 leads with 31.4 across 20 models. Foundation models have been increasingly applied to behavioral science domains such as psychology, sociology, and economics. While these models show promise in individual tasks such as survey response prediction and human-subject experiment simulation, there remains no systematic understanding of how well they perform across diverse behavioral science tasks, contexts, and populations. We introduce
• MedBench v5 (CCR-Agent (self-reported)): Claude Opus 4.7 leads with 96.66 across 10 models. Existing medical AI benchmarks lack process visibility, atomic skill evaluation, and integrated hallucination detection. We introduce MedBench v5, a redesigned benchmark for clinical multimodal models (language, vision-language, and agent systems) that moves from static QA to dynamic, process-oriented evaluation. MedBench v5 features: (1) a dual-dimensional framework combining Clinical Cognitive R
• Qwen-AgentWorld (Avg. (self-reported)): Qwen-AgentWorld-397B-A17B leads with 58.71 across 18 models. A world model predicts environment dynamics based on current observations and actions, serving as a core cognitive mechanism for reasoning and planning. In this work, we investigate how world modeling based on language models can further push the boundaries of general agents. (i) We first focus on building foundation models for agentic environment simulation. We introduce Qwen-AgentWorld-35B-A3B a
• AgentCIBench (Leakage (self-reported)): Gemini 3.1 Pro Preview leads with 98.3 across 15 models. Computer-use agents (CUAs) now act on a user's behalf across personal applications such as email, calendars, and to-do lists. This cross-application access is useful, but it also creates a privacy risk that has been largely overlooked: when an agent works in one context, it can pull in information from another that is inappropriate in that context. Hence, we introduce AgentCIBench, an evaluation h
• GUI vs. CLI (Avg. (self-reported)): GPT-5.4 leads with 59.1 across 9 models. Computer-use agents can execute software tasks through either graphical interfaces or programmatic command interfaces, but existing evaluations confound interaction modality with differences in tasks, initial states, verifiers, and permitted actions. We introduce a matched execution-layer benchmark of 440 desktop tasks across 18 applications and 12 workflow categories, where screen-only GUI agents
• MuPPET (Multi-Party (self-reported)): Qwen3 8B leads with 70.37 across 7 models. LLM agents are increasingly deployed in multi-party environments, handling sensitive personal data on behalf of individual users, for instance in group chats. When such an agent discloses private information, it reaches every group member at once. This risk is structurally harder to control than in one-to-one settings, as every piece of private information must be appropriate for every recipient i
• BLUEX v2 (Score (self-reported)): Gemini 3.1 Pro Preview leads with 9.1 across 10 models. Although Large Language Models (LLMs) excel in many tasks, their assessment in Portuguese has received less attention, particularly for open-ended, discursive tasks that demand deeper reasoning and generation capabilities. While the original BLUEX benchmark addressed the scarcity of Portuguese evaluation datasets through multiple-choice questions from Brazilian university entrance exams, it did no
• MMGist (Macro ↑ (self-reported)): Gemini 3.1 Pro Preview leads with 66.8 across 27 models. We conduct a systematic study of 18 widely used vision-language benchmarks and identify three major issues: 1) many items do not rely on visual cues and therefore fail to effectively measure multimodal understanding; 2) many items are already close to performance saturation for current LVLMs, which limits their discriminative power; 3) a small number of anomalous items affect the reliability of ev
• PlanBench-XL (Accuracy (%) (self-reported)): Gemini 3.1 Pro Preview leads with 77.06 across 10 models. LLM agents increasingly operate in large tool ecosystems, where real-world tasks require discovering relevant tools, inferring implicit sub-goals, and adapting to dynamic environments over long horizons. However, existing benchmarks rarely evaluate planning under retrieval-limited tool visibility. To address this gap, we introduce PlanBench-XL, an interactive benchmark of 327 retail tasks over 1,6
• Benchmarking Large Language Models for Graphem (Direct (self-reported)): Qwen3 8B leads with 100.15 across 37 models. Grapheme-to-phoneme (G2P) conversion is essential for controllable and robust text-to-speech, and large language models (LLMs), with broad linguistic knowledge, offer a promising approach. We benchmarked over 30 LLMs on Japanese G2P, comparing them with conventional morphological analyzers on 3000 manually annotated sentences. We evaluated two prompting strategies: a parse mode, where the LLM perf
• Inverse Turing Bench (Accuracy (self-reported)): GPTZero-W-only leads with 89.41 across 17 models. As AI systems integrate into online spaces, differentiating them from humans in conversations is increasingly important. We present Inverse Turing Bench, a benchmark that evaluates LLMs and other models on their ability to differentiate humans and AI in multi-turn text. The benchmark provides a collection of paired dialogue transcripts, wherein one dialogue is between two humans and the other is b
• CheXpercept (Stage 1 (End-to-End) (self-reported)): Qwen3.6 27B leads with 92.2 across 14 models. The evaluation of vision-language models (VLMs) for chest X-ray (CXR) analysis has largely been limited to disease-presence classification without visual grounding. Such evaluations fail to verify the expert-level lesion perception necessary to ensure the clinical reliability of VLMs. To address these limitations, we introduce CheXpercept, a sequential, multi-level perception benchmark that mirror
• CulMind (S (self-reported)): Gemini 3 Flash Preview leads with 50.7 across 14 models. Evaluating Multimodal Large Language Models (MLLMs) in Chinese Cultural Heritage (CCH) requires fine-grained reasoning over visual, textual, stylistic, and historical clues. However, existing CCH benchmarks mainly emphasize final-answer accuracy, while the accuracy and completeness of reasoning processes remain underexplored. To address this gap, we introduce CulMind and CulMind-R: a high-quality
• MedLayXPlain (S (self-reported)): Gemini-2.5-Flash +Thinking leads with 70.6 across 32 models. Medical Vision-Language Models (Med-VLMs) achieve strong expert-level performance, yet their ability to generate patient-accessible descriptions remains underexplored. With the 21st Century Cures Act now mandating immediate patient access to diagnostic imaging results, evaluating whether Med-VLMs can bridge this Expert-Lay Gap is both urgent and clinically consequential for patient education and s
• The Metanym Game (T (self-reported)): Claude Opus 4.5 leads with 7.0 across 12 models. The metanym game is a competitive word game for LLMs that measures structural intelligence against established cognitive-science constructs. No content is given in advance; the contestants create all of it -- a new kind of analogy test, analogical production falsifiable sentence by sentence, with no fixed test set to leak into training (contamination-resistant by construction). In the council-of-p
• Trip+ (Plan Avg. (self-reported)): Gemini 3.1 Pro Preview leads with 73.31 across 18 models. Interactive travel planning has become a popular use case for language models. Agents are deployed to manage evolving preferences and unexpected disruptions over multiple turns. Such settings require models to make complex, profile-conditioned planning decisions. However, existing benchmarks often evaluate feasibility, personalization, or interaction in relatively isolated settings. We therefore i
• BIM-Edit (Final (self-reported)): Gemini 3.0 Flash leads with 49.48 across 7 models. Large language models (LLMs) are increasingly applied to computer-aided design (CAD) to generate design artifacts from textual instructions. In engineering practice, this requires more than creating new geometry, models must also understand existing scenes, edit them correctly, and preserve semantics and relations. However, many CAD benchmarks focus on creating new models rather than editing exist
• CombEval (Avg. (self-reported)): GPT-5.5 leads with 93.6 across 11 models. We present CombEval, a dynamic benchmark for evaluating combinatorial counting in large language models. CombEval represents each problem as a typed Cofola specification over entities, combinatorial objects, object dependencies, and constraints, enabling controlled generation of natural-language counting problems with exact solver-verified answers. Unlike static collections, CombEval supports syst
• JamSet/JamBench (Task 1a SCS (self-reported)): GPT-5.4 leads with 46.0 across 9 models. Current AI-driven game development has made substantial progress in asset generation, gameplay design, and web-based game coding, yet project-level code engineering on professional game engines remains largely unexplored due to the absence of large-scale datasets and deterministic evaluation methods. We present JamSet and JamBench, the first project-level game code framework dataset and benchmark
• ORAgentBench (Pass Rate All (self-reported)): GPT-5.4 leads with 35.51 across 14 models. Large language models are increasingly deployed as autonomous agents for multi-step tasks in executable environments, yet their ability to perform realistic operations research (OR) work remains unclear. Existing OR evaluations often decouple modeling from solving, rely on pre-formalized or text-only instances, and rarely test the full workflow from operational artifacts to validated decisions. In
• ROSE (Avg. (self-reported)): Human leads with 98.8 across 10 models. Multimodal large language models (MLLMs) are increasingly expected to act on visual information, yet the same scene may require different actions under different task contexts. How reliably can a model turn the same visual evidence into the action required by the current context? To answer this question, we introduce \\textsc{ROSE} (\\textbf{R}eference-conditioned \\textbf{O}ddity and \\textbf{S}y
• Are LLMs Ready to Assist Physicians? PhysAssis (en_mrs (self-reported)): GLM 5 leads with 69.4 across 14 models. The most plausible near-term role of medical LLMs is to assist rather than replace physicians, yet current evaluations often test isolated capabilities: clinical knowledge, EHR system interaction, or patient communication. Physician assistance instead requires coordinating these capabilities within the same interaction, where physicians issue underspecified requests, patients describe symptoms amb
• Crosby micro1 RedlineBench (Turn-weighted score (self-reported)): GPT-5.5 leads with 50.5 across 4 models. Multi-turn contract-redlining benchmark for SaaS MSA negotiations, using document-native redlines, attorney-authored golden responses, and rubric-based turn-level and behavioral evaluation. (Source: benchmarklist.com, self-reported.)
• LaViSA (PT-Acc (self-reported)): Gemini 3.1 Pro Preview leads with 88.9 across 10 models. Structural ambiguity arises when a single sentence admits multiple valid interpretations due to its syntactic structure, posing a fundamental challenge for language understanding. Visual scenes serve as useful cues for resolving such ambiguity, and Vision and Language Models (VLMs) need to be capable of deriving possible semantic interpretations from visual scenes. We introduce Language and Vision
• The Wrong Kind of Right (MAR Disability (self-reported)): GPT-5.4 leads with 48.8 across 26 models. Warning: This paper studies stereotypes and biases, and contains potentially disturbing examples, used for illustration purposes only. Our findings should not be interpreted as an argument against alignment. Instead, this paper highlights the need for principled approaches to more advanced alignment. Alignment aims to ensure that large language models (LLMs) behave safely and reliably, including b
• Agentic Skills Evaluation Framework (Overall Score w/ (self-reported)): Claude Opus 4.8 leads with 92.7 across 19 models. Agent skills -- structured, reusable knowledge artifacts that augment LLM agent capabilities -- have been rapidly adopted in industry, yet their cross-domain impact and use across commercial and open-source models remain under-studied, and no reusable methodology exists for evaluating an individual skill. In this work, we present an evaluation framework that lets a skill author construct realistic
• CEO-Bench (Best run API cost (self-reported)): GLM 5.1 leads with 0.0 across 10 models. Language model agents are becoming proficient executors at isolated, short-horizon tasks such as software engineering and customer service. Yet real-world challenges require a combination of sophisticated skills that remain largely untested in agents: (1) navigating long horizons amid uncertainty; (2) acquiring information in noisy environments; (3) adapting to a changing world; (4) orchestrating
• EComAgentBench (Acc. (self-reported)): Opus 4.6 leads with 57.1 across 7 models. As LLM-based shopping agents enter production, existing benchmarks fail to capture how a shopper's requirements arrive: stated implicitly in the query, recorded in a profile, or revealed only when the right question is asked. Benchmarks that expose full intent upfront and grade only the final choice can neither pose this long-horizon challenge nor explain which requirement an agent missed. To addr
• ReproRepo (Issue Match EM@10 (self-reported)): GPT-5.5 leads with 25.4 across 4 models. Reproducing research results from papers and released code is central to scientific progress. Existing works have introduced benchmarks to evaluate whether LLM agents can assist with reproducibility, but they are difficult to scale due to their reliance on substantial manual effort for data curation and evaluation. We introduce ReproRepo, a scalable framework for reproducibility evaluation that le
• GRACE (Avg Step (self-reported)): Gemini 3.1 Pro Preview leads with 81.46 across 13 models. Many reasoning tasks require models to reason over input context, from document-grounded question answering to rule-based deduction. Chain-of-Thought (CoT) prompting produces traces that appear transparent, yet individual steps can silently deviate from the source evidence, even when the final answer is correct. Existing methods detect hallucinations at the response level but fail to identify wher
• SearchGEO (ASR↓ (%) (self-reported)): Gemini 3 Flash Preview leads with 31.4 across 13 models. Large language model (LLM)-based search agents synthesize open-web content into actionable recommendations on behalf of users, creating a risk that attacker-published pages are transformed into endorsed claims. We introduce SearchGEO, a controlled evaluation framework for measuring endorsement corruption in LLM-based web-search agents, combining a web-evidence manipulation pipeline, a five-mode at
• UXBench (Automated Lift (self-reported)): GPT-5.4 leads with 21.6 across 8 models. Large language models (LLMs) are increasingly deployed as UX judges that inspect interfaces, diagnose usability problems, and propose repairs. Yet no controlled benchmark measures whether the resulting critiques are reliable and actionable across heterogeneous product surfaces. We introduce UXBench, a benchmark for evaluating LLMs as interaction-grounded UX judges. UXBench comprises local-first ru
• MedCTA (Outcome Accuracy (self-reported)): GPT-5.4 leads with 31.54 across 18 models. MedCTA evaluates medical tool agents on clinician-validated, step-implicit tasks grounded in multimodal clinical inputs, including radiology images, pathology slides, and reports. The benchmark contains 107 real-world clinical tasks with clinician-verified executable trajectories over five deployed tools and measures tool selection, argument validity, execution stability, trajectory fidelity, and
• FrontierCode (Main Score (self-reported)): Claude Fable 5 x-high leads with 46.3 across 16 models. Cognition benchmark for production-quality coding agents measuring whether maintainers would merge model PRs. It uses 150 maintainer-authored open source tasks with nested Extended, Main, and Diamond subsets scored by blockers and quality rubrics.
• Dr. DocBench (Overall (self-reported)): GPT-5.5 leads with 61.94 across 12 models. Document parsing and recognition are fundamental capabilities for vision-language models (VLMs) and document processing systems. However, existing Optical Character Recognition (OCR) and document parsing benchmarks are increasingly limited in coverage and difficulty: many focus on common document genres or uniformly sampled pages where modern parsers already perform strongly, while offering limite
• SmartHome-Bench (Overall (self-reported)): HomeFlow-RL-8B leads with 87.03 across 14 models. Large language model agents are moving beyond text-only interaction toward physical-world control, with smart homes as a representative domain. Real domestic interaction requires understanding ambiguous intents, operating in dynamic environments, and performing multi-turn reasoning. However, existing methods struggle to generate high-quality training data for smart home agents. We propose HomeFlow
• TukaBench (ASR ↑ African Languages (self-reported)): GPT 3.5* leads with 38.1 across 13 models. Safety evaluation of Large Language Models (LLMs) remains heavily English-centric, leaving Low-Resource Languages (LRLs), particularly African ones, critically underexplored. We introduce TUKABENCH, a jailbreak benchmark for seven African languages that extends JailbreakBench (JBB) beyond direct translation through four settings: human translation of JBB prompts, English adaptation to African cont
• Sandboxed Coding Agents are Competitive Omni-m (OmniGAIA Avg. (self-reported)): GPT-5.4 x-high leads with 75.0 across 12 models. As multimodal LLMs increasingly target video and audio, it is often assumed that such tasks require native omnimodal models. We show that this is not always the case: coding agents with only text+image access and a sandboxed tool-use interface can match, and in several settings outperform, SOTA native omnimodal models and predefined multimodal agent scaffolds across multiple audio-video benchmarks
• When Safe Skills Collide (Full chain (self-reported)): Haiku 4.5 leads with 9.0 across 9 models. LLM agents increasingly rely on community-contributed skills that expand an agent's operational capability set. We study a core safety problem in agentic AI systems: whether individually safe skills can compose into unsafe installed skill sets. We present SkillReact, a compositional security measurement framework with three components: a deterministic static-composition benchmark, a two-rater LLM-
• BilliardPhys-Bench (Total (Weighted) (self-reported)): GPT-5.5 leads with 73.58 across 13 models. Current multimodal models handle static image recognition well, but intuitive physical reasoning remains a weakness. Predicting how objects will move and interact from a single image is still difficult for these systems. We present BilliardPhys-Bench, a benchmark for physical reasoning in synthetic billiards environments. Its procedural engine generates randomized scenarios with friction and elast
• MineExplorer (Overall TSR (self-reported)): Claude Opus 4.6 leads with 41.08 across 18 models. Multimodal large language models (MLLMs) have shown strong capabilities in perception, reasoning, and action generation. However, their ability to sustain exploration in dynamic open worlds remains unclear. Existing embodied and game-based benchmarks often compress interaction into short-horizon tasks or entangle success with domain-specific game mechanics. In this paper, we introduce MineExplorer
• RealityTest (Text disclosure probability (self-reported)): Claude Haiku 4.5 leads with 92.3 across 17 models. AI systems are increasingly deployed in conversational settings where users may be uncertain whether they are speaking with a human or an AI. Despite mounting regulatory attention to this known safety risk, existing evaluations of AI disclosure are typically English-only, based on machine-generated questions, and restricted to text. We present RealityTest to comprehensively test whether AI systems
• SpatialAct (Succ. Rate (self-reported)): Gemini 3.1 Pro Preview leads with 20.6 across 7 models. Humans can effortlessly perceive spatial layouts, form cognitive representations, reason about spatial relations, and translate such reasoning into actions in everyday 3D environments. Although recent vision-language models (VLMs) have shown promising performance on observation-conditioned spatial perception and reasoning tasks, it remains unclear whether they can build coherent spatial understand
• StemBind (F Overall (self-reported)): Qwen3.5 Plus 2026-04-20 leads with 42.2 across 24 models. Multimodal large language models (MLLMs) often know the rule but pick the wrong answer: on abstract visual reasoning (AVR) tasks, a model can describe what it sees and name the underlying pattern, yet still fail to choose the matching candidate. Existing AVR benchmarks cannot detect this because they collapse perception, rule induction, and answer selection into a single right-or-wrong signal. We
• ActTraitBench (G_KD (Global Knowledge-Decision Gap) (self-reported)): MiniMax M2.5 leads with 2.17 across 15 models. While Large Language Models (LLMs) can convincingly simulate personas in explicit self-reports, they often deviate in implicit behavioral decisions, revealing a substantial Knowledge-Decision Gap ($G_{\\text{KD}}$). Existing benchmarks struggle to measure this asymmetry due to limited construct validity, multi-dimensional entanglement, and distributional biases in LLM-based evaluation. To address
• CardioLens (F1 Score (Random) (self-reported)): QoQMed-7B leads with 58.72 across 24 models. Multimodal Large Language Models (MLLMs) have shown strong performance on public medical benchmarks, yet existing evaluations often remain weak proxies for clinical use, relying on isolated inputs and simplified recognition-style tasks. We introduce CardioLens, a leakage-resistant evaluation testbed for multi-sequence Cardiovascular Magnetic Resonance (CMR), constructed from private hospital archi
• Causal Sensitivity Score (CSS) (CSS (self-reported)): Grok 4.20 leads with 47.3 across 6 models. Two clinical AI systems can score nearly identically on coverage-based rubrics yet behave radically differently when their patient inputs change: one updates its recommendations to match the new clinical signal, while the other produces the same output regardless. We introduce the Causal Sensitivity Score (CSS), a pre-registered interventional metric that mutates oncology tumor-board cases along f
• Cookie-Bench (React Overall (self-reported)): Opus 4.7 leads with 83.3 across 13 models. Front-end web code has become a core product surface for every frontier LLM release, yet evaluating these interactive applications at development speed remains costly because human-judged leaderboards like Arena do not scale. Existing automated proxies typically lean on reference implementations, test suites, or rigid checklists, and tend to miss the reasoned synthesis a human reviewer performs ov
• FinVerBench (Accuracy (self-reported)): Claude Sonnet 4 leads with 100.0 across 15 models. We introduce FinVerBench, a benchmark and validity study for financial statement verification: determining whether a set of corporate financial statements is numerically consistent from the information shown to the model. FinVerBench is built from SEC 10-K XBRL filings for 43 S&P 500 companies and defines a four-category error taxonomy covering arithmetic, cross-statement linkage, year-over-year,
• NICE (Weighted All (self-reported)): Gemini 3.1 Pro Preview leads with 78.1 across 6 models. As large language models (LLMs) are increasingly applied in social contexts such as emotional companionship and customer service, measuring their social intelligence has become critical to the quality and safety of human-AI interaction. However, existing social intelligence benchmarks lack a unified framework that organizes social abilities into a unified structure, and therefore cannot enable fin
• OmniMatBench (Avg. Score (self-reported)): Claude Opus 4.7 leads with 37.2 across 13 models. As multimodal language models play an increasingly important role in scientific research, materials science offers a critical testbed due to its interdisciplinary, multimodal, and application-driven nature. However, existing materials benchmarks mainly focus on property prediction, knowledge QA, or characterization understanding, leaving the broader reasoning process from materials knowledge to ap
• PassBench (AS Score (self-reported)): Eager leads with 100.0 across 11 models. Modern tensor compilers such as TorchInductor deliver substantial speedups on mainstream models, yet face a systematic performance ceiling on long-tail workloads -- our profiling shows that 43% of real-world subgraphs experience end-to-end slowdowns under default compilation. While LLMs offer a path toward automated optimization, existing efforts focus on standalone kernel generation. We argue tha
• Personalized Turn-Level User Conversation Sati (Micro (self-reported)): MoonshotAI: Kimi K2.6 leads with 4.74 across 7 models. User satisfaction with AI assistants is highly personalized: the same response may satisfy one user but disappoint another depending on what each user expects and what they have asked for before. Existing automatic evaluation methods mostly measure generic response quality, making it difficult to judge whether a response satisfies a user at a specific turn. We study this problem as personalized tu
• ResearchClawBench (Overall (self-reported)): Claude Code (Claude-Opus-4.6) leads with 21.5 across 18 models. ResearchClawBench evaluates model capability on agentic tasks from the linked upstream source with Average Score as the primary reported metric.
• WMW (World Models in Words) (Trace–ans. consistency (self-reported)): Claude Opus 4.7 leads with 91.0 across 7 models. Vision-language models (VLMs) are increasingly used to answer questions about physical scenes, yet most evaluations reduce performance to a final answer. This hides whether the model perceived the right objects, represented the right physical state, predicted a plausible transition, or merely selected the right option for the wrong reasons. We introduce \\wmw, an evaluation framework for auditing
• ATRBench (TSAcc Default (%) (self-reported)): DeepSeek V4 Flash leads with 23.7 across 8 models. A long-lived LLM agent, such as OpenClaw, earns its value by acting on a user's preferences and constraints across sessions, not just the current request. Yet today's agents keep what a user volunteers but rarely ask for what stays unspoken, leaving a proactivity gap in long-lived LLM agents: an agent cannot act on a preference it never obtained. As users delegate more of their affairs to agents,
• Can Large Language Models Handle Discourse Par (Overall Avg (MS) (self-reported)): GPT-5 leads with 72.4 across 10 models. Discourse particles, such as \\textit{well} and \\textit{kind of}, are crucial components that enable LLMs to ``speak'' more like humans. They are used to convey emotions, intentions, and interpersonal meanings. However, existing studies have not yet built a comprehensive understanding of LLMs' capabilities in handling discourse particles. Moreover, the limited number of studies focuses primarily
• DisasterBench (Exact-match Accuracy (self-reported)): Gemini 3.1 Pro Preview leads with 73.39 across 12 models. Disasters cause severe societal impacts, demanding rapid coordination of heterogeneous AI tools, from satellite analysis to flood prediction and damage assessment, into coherent multi-step workflows. As LLMs increasingly serve as orchestrators of such pipelines, effective coordination requires more than selecting semantically plausible tools: LLMs must generate executable workflows with correct pa
• Do Agents Know What They Can't Do? Evaluating (Avg. (self-reported)): GPT-5.5 leads with 61.2 across 9 models. Infeasibility-awareness benchmark — can tool-using agents detect that a task is impossible under a constrained tool environment instead of burning compute; self-reported average across settings.
• From Knowing to Doing (Total ret. (self-reported)): Qwen3.6 Plus leads with 85.29 across 10 models. Evaluating whether large language model (LLM) agents can profit in capital markets is increasingly framed as end-to-end trading: place an agent in a historical market, let it trade, and measure portfolio returns. This setup is vulnerable to two evaluation failures. First, long backtests often overlap with the knowledge cutoffs of frontier LLMs, allowing memorized tickers, dates, prices, and market
• HardMTBench (HardMTBench zh-en GEMBA-DA (self-reported)): GPT-5.5 leads with 91.4 across 20 models. General-purpose machine translation benchmarks such as FLORES-200 have reached a saturation regime on Chinese-English pairs, where modern large language models cluster within a narrow band of high scores. Across 22 systems, FLORES-200 zh-en GEMBA scores fall in a 7.87-point range with a standard deviation of 2.29, which compresses the separation between systems on knowledge-intensive domains such
• IFMTBench (IF$_\text{T}$ (self-reported)): Gemini 3.1 Pro Preview leads with 89.08 across 15 models. Modern translation workflows demand more than semantic equivalence. Users routinely require models to preserve JSON or HTML schemas, honor curated glossaries, disambiguate with provided context, and match prescribed registers, often several at once. Conventional metrics such as BLEU and xCOMET capture semantic fidelity but provide little signal on constraint adherence, while general instruction fo
• MUSE (Final Score (self-reported)): GPT-5.5 leads with 67.14 across 15 models. Large language models (LLMs) have recently advanced text-driven 3D generation, yet Text-to-CAD remains far from supporting industrial product design. Existing benchmarks focus primarily on generating single-part CAD models and evaluate them using geometric similarity metrics that fail to capture functionality, manufacturability, and assemblability. To address this gap, we introduce MUSE, a Text-to
• OR-Space (Build Pass@1 (self-reported)): Gemini 3.1 Pro Preview leads with 72.0 across 19 models. Large language model (LLM) agents are increasingly used to assist with operations research (OR) modeling, yet existing OR-oriented benchmarks often reduce evaluation to one-shot translation from a self-contained problem statement into a mathematical formulation or solver program. Such settings abstract away two characteristics of real industrial OR workflows: persistent multi-artifact workspaces a
• PDP-Bench (Macro-F1 (self-reported)): Gemini 3.1 Pro Preview leads with 78.53 across 7 models. Legal Judgment Prediction (LJP) has become a core benchmark for evaluating AI in the criminal legal domain, but it only sees criminal cases that have already passed prosecutorial review and been formally indicted. As a result, LJP leaves a substantial blind spot in assessing criminal liability, overlooking cases involving insufficient evidence, no criminal liability, or guilt exempted from punishm
• Plant, Persist, Trigger (Avg. (self-reported)): Gemini 3 Flash Preview leads with 51.1 across 7 models. Large Language Model (LLM) agents remain vulnerable to safety threats from the external environment, where attackers inject adversarial content into external observations such as tool-returned data, webpages, or MCP context, causing harmful agentic behaviors such as unsafe actions or incorrect outputs. Existing studies typically focus on single-interaction attacks, where the agent observes adversa
• When Context Flips, Safety Breaks (PacifAIst BSR (self-reported)): Llama 3.1 (8B) leads with 77.6 across 12 models. Safety benchmark scores provide incomplete evidence of deployment readiness: aligned language models often adhere to rigid rules even when a situational update flips which action is safe. We term this failure brittle safety. To diagnose it, we introduce context-flip evaluation, testing 12 models across a safety benchmark (PacifAIst) and two commonsense controls using paired variants where the nomi
• JuICE (F1 Score (self-reported)): Gemini 3.1 Pro Preview leads with 56.67 across 10 models. As large language models (LLMs) are increasingly deployed to users around the world, they are integrated into everyday tasks across diverse cultural contexts, from drafting personal communications to brainstorming creative ideas. These tasks are inherently cultural: they require contextual appropriateness, symbolic resonance, and tacit cultural expectations that native speakers draw on instinctive
• LiveK12Bench (Mathematics Acc (self-reported)): Gemini 3 leads with 88.3 across 12 models. Advanced Large Multimodal Models (LMMs) have demonstrated impressive performance in K-12 reasoning tasks, exhibiting great promise as intelligent tutors. Realizing this potential requires models to navigate real-world examinations effectively, yet most existing benchmarks fail to capture the complexity of authentic testing environments. Specifically, most datasets are static, prone to data contami
• Qiskit QuantumKatas (Best (self-reported)): GPT-5.5 leads with 83.1 across 16 models. We adapt Microsoft's QuantumKatas -- a well-established quantum computing curriculum -- from Q# to Qiskit, the most widely-adopted quantum computing framework, and package it with an evaluation framework for systematic LLM assessment. The resulting benchmark comprises 350 tasks across 26 categories, spanning fundamental gates through advanced algorithms (Grover's, Simon's, Deutsch-Jozsa), error co
• Self-Ensembling Vision-Language Models for Cha (ChartQA (self-reported)): TinyChart + Self-ens. (ours) leads with 95.28 across 14 models. Charts effectively convey quantitative information, but the underlying data are often locked in image form, hindering reuse and analysis. Manually digitizing charts is time-consuming and error-prone, motivating automatic chart-to-table extraction. Recent approaches use specialized vision-language models (VLMs), yet performance still lags on charts with many datapoints or substantial stylistic vari
• Verus-SpecBench (Pass@1 (self-reported)): Gemini 2.5 Pro leads with 77.8 across 6 models. AI coding agents are increasingly used to write real-world software, but ensuring that their outputs are correct remains a fundamental challenge. Formal verification offers a promising path: an agent generates code together with a machine-checked proof, guaranteeing that the code satisfies a formal specification. However, there is no guarantee that the formal spec itself matches the user's intent.
• VitaBench 2.0 (Avg@4 Full Context (self-reported)): Claude Opus 4.6 leads with 50.3 across 20 models. Large language models (LLMs) have evolved into interactive agents that collaborate with users in real-world tasks. Effective collaboration in such settings increasingly depends on understanding the user beyond what is explicitly stated, as user intent is often reflected in fragmented daily interactions and requires both personalized modeling and proactive interaction. However, existing agent bench
• Claw-Anything (Pass@1 (self-reported)): GPT-5.5 leads with 34.5 across 9 models. Large language model agents are increasingly envisioned as always-on personal assistants with access to anything relevant in the user's digital world. Yet current systems operate over only narrow slices of that world, limiting context-sensitive reasoning and effective assistance. Existing benchmarks similarly provide only partial user state and therefore fail to capture performance in such a broad
• DiscoverPhysics (Mean Explanation Score (self-reported)): Claude Opus 4.7 leads with 61.0 across 11 models. Frontier LLMs now perform strongly across a wide range of physics evaluations, but it is hard to disentangle genuine reasoning from recall of established science. We introduce DiscoverPhysics, an interactive benchmark that asks a LLM agent to discover the laws of motion of a simulated world whose physics deliberately deviates from our own. We construct 22 worlds governed by, among others, screened
• QUIET (QUIET Total (self-reported)): Gemini 3.1 Pro Preview leads with 8.69 across 12 models. Large language models (LLMs) face a dual challenge in creative capability evaluation: existing benchmarks (e.g., Story Cloze Test, HellaSwag) measure models' discriminative ability over narrative continuation using multiple-choice recognition paradigms, rather than directly measuring creative generation capability; rubric-based scoring and LLM-as-Judge methods rely on subjective dimension assessme
• RepoMirage (Avg. (self-reported)): Gemini 3.1 Pro Preview leads with 41.4 across 8 models. Code agents are currently having skillful performance on repository-level software engineering benchmarks, but it remains unclear whether success on end-to-end tasks such as issue resolution truly reflects repository context reasoning, the ability to identify the task-relevant information across multiple files and reason over the relations among them. To investigate this question, we introduce Rep
• StakeBench (Agg (self-reported)): GPT-5.5 leads with 20.4 across 15 models. Existing financial NLP benchmarks often rely on labels supplied by outside observers, measuring how language is perceived rather than what speakers have committed to in the market. We introduce StakeBench, an evaluation framework for language understanding grounded in market commitment. StakeBench links 560,876 comments from 2,261 resolved markets to verified position, action, and market-odds reco
• VisualNeedle (w/ Tools Acc. (%) (self-reported)): Gemini 3.1 Pro Preview leads with 56.01 across 9 models. Frontier multimodal large language models (MLLMs) have been reported to achieve over 90% accuracy on fine-grained perception benchmarks. However, such scores do not necessarily imply faithful use of visual evidence. Prior studies have identified three shortcuts that inflate benchmark performance. First, linguistic priors and lexical cues in questions often enable models to infer plausible answers
• FrontierOR (Sol. quality (self-reported)): Gemini 3.1 Pro Preview leads with 52.0 across 7 models. Large language models (LLMs) are increasingly used for optimization modeling and solver-code generation, yet practical operations research and optimization problems often require a harder capability: designing scalable algorithms that exploit problem structure and outperform direct formulation-and-solve baselines. Existing benchmarks are limited to small or simplified examples far below real-world
• GlobalDentBench (Macro-Average Score (self-reported)): Gemini 3.1 Pro Preview leads with 63.27 across 12 models. While large language models (LLMs) hold transformative potential for medicine, their reasoning robustness and safety in real-world clinical scenarios remain critically underexplored, particularly in dentistry. Here we introduce GlobalDentBench, the first multinational dental benchmark, featuring a taxonomy that encompasses 14 dental specialties across 88 countries and regions spanning six continen
• EvoCode-Bench (MT@4 (self-reported)): Opus 4.7 leads with 54.0 across 13 models. Coding agents are increasingly used as iterative development partners, but most benchmarks still evaluate one specification followed by one final assessment. This leaves out a basic question: can an agent keep its own codebase working as requirements change? We introduce EvoCode-Bench, a benchmark of 26 stateful coding tasks and 227 evaluated rounds. Each task preserves the agent's workspace for 5
• GENSTRAT (Alpha (chips/game) (self-reported)): GPT-5.4 high leads with 85.0 across 9 models. Large language models (LLMs) are increasingly deployed as economic agents in marketplaces, auctions, and bidding settings. Anticipating their behavior in any specific deployment is hard. Existing strategic-reasoning benchmarks evaluate models on fixed canonical games. These benchmarks may saturate as the frontier improves, and they do not allow evaluators to generalize with confidence from benchma
• OpenSkillEval (Overall avg. (self-reported)): Claude Opus 4.6 leads with 4.51 across 10 models. Skills, i.e., structured workflow instructions distilled for large language models (LLMs), are becoming an increasingly important mechanism for improving agent performance on real-world downstream tasks. However, as the open-source skill ecosystem rapidly expands, it remains unclear how different models and agent frameworks interact with skills, how to evaluate skill quality, and how users should
• ForecastBench-Sim (FBSim) (ECI (self-reported)): Claude Opus 4.6 leads with 155.0 across 27 models. We document inverse scaling in LLMs on forecasting problems whose underlying time series exhibit superlinear growth and tail risk of regime change, a structure common in finance and epidemiology. On these tasks, more capable models produce worse distributional forecasts. The pattern appears on ForecastBench-Sim (FBSim), a contamination-free, simulated-world benchmark we release, in forecasting syn
• Perception or Prejudice (HR (self-reported)): Gemini 3 Flash Preview leads with 33.5 across 27 models. Multimodal Large Language Models (MLLMs) are increasingly deployed in human-facing roles where personality perception is critical, yet existing benchmarks evaluate this capability solely on numerical Big Five score prediction, leaving open whether models truly perceive personality through behavioral understanding or merely prejudge through superficial pattern matching. We address this gap with thr
• SGR-Bench (Item-F1 (self-reported)): GPT-5.5 leads with 66.18 across 11 models. Recent advances in large language models and tool-using agents have expanded the range of benchmarked web tasks. Yet an important class of specialized retrieval tasks remains undercharacterized. On many specialized data-retrieval websites, answer-bearing evidence becomes accessible only after establishing the correct site-specific retrieval state through filters, views, hierarchies, or scopes. We
• SpaceDG (Avg. (self-reported)): SpaceDG-SFT$_{\textit{Qwen3-VL-8B-Instruct}}$ leads with 66.1 across 29 models. Multimodal Large Language Models (MLLMs) have made rapid progress in spatial intelligence, yet existing spatial reasoning benchmarks largely assume pristine visual inputs and overlook the degradations that commonly occur in real-world deployment, such as motion blur, low light, adverse weather, lens distortion, and compression artifacts. This raises a fundamental question: how robust is the spatia
• ArchSIBench (Avg. (self-reported)): Human Level(w bg in Arch.) leads with 89.2 across 28 models. Architectural spatial intelligence, the ability to recognize and infer architectural space, is fundamental to tasks such as robot navigation, embodied interaction, and 3D scene understanding and generation. Although extensive research has evaluated the basic spatial skills of Vision-Language Models (VLMs) such as relative orientation, distance comparison, and object counting, these tasks cover onl
• AttuneBench (Composite (self-reported)): Opus 4.6 leads with 54.3 across 11 models. Emotional intelligence (EI), the ability to perceive, understand, and respond appropriately to others' emotional states, is central to human communication, and increasingly important to assess as LLMs assume conversational roles in everyday life. Existing EI benchmarks rely on synthetic prompts, single-turn cases, or third-party annotation. These approaches do not directly measure how models infer
• DeepWeb-Bench (Overall score (self-reported)): GPT-5.5 leads with 33.37 across 9 models. Deep research, in which an agent searches the open web, collects evidence, and derives an answer through extended reasoning, is a prominent use case for frontier language models. Frontier deep research products score high on existing benchmarks, making it difficult to distinguish their capabilities from current evaluation data alone. We introduce DeepWeb-Bench, a deep research benchmark that is su
• Hack-Verifiable TextArena (Avg HR (self-reported)): Grok 4.1 Fast leads with 28.5 across 12 models. Aligning autonomous agents with human intent remains a central challenge in modern AI. A key manifestation of this challenge is reward hacking, whereby agents appear successful under the evaluation signal while violating the intended objective. Reward hacking has been observed across a wide range of settings, yet methods for reliably measuring it at scale remain lacking. In this work, we introduce
• HIDBench (DARPA E3 CADETS MCC (self-reported)): Claude Opus 4.6 leads with 60.2 across 9 models. Recent benchmark efforts have advanced the evaluation of large language models (LLMs) in cybersecurity, including tasks such as penetration testing and vulnerability identification. However, a critical cybersecurity task, namely intrusion detection from system logs, remains unexplored. In this work, we present a new benchmark to assess LLMs' capabilities in supporting host-based intrusion detectio
• PlanningBench (All-pass (%) (self-reported)): GPT-5.4 x-high leads with 63.17 across 16 models. Planning is a fundamental capability for large language models (LLMs) because such complex tasks require models to coordinate goals, constraints, resources, and long-term consequences into executable and verifiable solutions. Existing planning benchmarks, however, usually treat planning data as fixed collections of instances rather than controllable generation targets. This limits scenario coverag
• QuestBench (Pass Rate (%) (self-reported)): GPT-5.5 leads with 57.58 across 13 models. As AI becomes part of everyday learning, many courses teach students to use it mainly as a productivity tool: how to prompt, search, summarize, write, code, and use tools more efficiently. We argue that AI education also needs a setting in which students learn to test AI and understand their own role in judging machine-produced knowledge. To this end, we introduce a course-based practice that teac
• RankJudge (Elo (All) (self-reported)): Gemini 3.1 Pro Preview leads with 1959.0 across 21 models. As interactive LLM-based applications are created and refined, model developers need to evaluate the quality of generated text along many possible axes. For simpler systems, human evaluation may be practical, but in complicated systems like conversational chatbots, the amount of generated text can overwhelm human annotation resources. Model developers have begun to rely heavily on auto-evaluation,
• RefusalBench (Strict Refusal Rate (self-reported)): Command A leads with 94.6 across 19 models. Frontier large language models are increasingly deployed as orchestration backbones for biological research workflows, yet no shared evidence base exists for comparing their refusal behaviour on legitimate research prompts. RefusalBench, introduced here, is a matched-triple benchmark of 141 prompts in 47 bundles that holds task framing constant while varying only biological risk tier (benign, bord
• TempGlitch (Acc. (1 FPS) (self-reported)): GPT-5.4 Mini leads with 52.4 across 10 models. Vision-language models (VLMs) are increasingly being explored for video game quality assurance, especially gameplay glitch detection. Most existing evaluations, however, treat glitches as static visual anomalies, asking models to detect failures from a single frame. We argue that this framing misses a key distinction: some glitches are spatial and visible in an isolated frame, whereas others are t
• WikiVQABench (Accuracy (self-reported)): InternVL3-78B leads with 75.6 across 15 models. Visual Question Answering (VQA) benchmarks have largely emphasized perception-based tasks that can be solved from visual content alone. In contrast, many real-world scenarios require external knowledge that is not directly observable in the image to answer correctly. We introduce WikiVQABench, a human-curated knowledge-grounded VQA benchmark constructed by systematically combining Wikipedia images
• HalluWorld (Overall (self-reported)): GPT-4o-mini leads with 28.1 across 13 models. Hallucination remains a central failure mode of large language models, but existing benchmarks operationalize it inconsistently across summarization, question answering, retrieval-augmented generation, and agentic interaction. This fragmentation makes it unclear whether a mitigation that works in one setting reduces hallucinations across contexts. Current benchmarks either require human annotation
• WildRoadBench (AP50 (self-reported)): Gemini 3 leads with 42.1 across 25 models. We introduce WildRoadBench, a wild aerial road-damage grounding benchmark that couples direct visual grounding by vision-language models with autonomous research-and-engineering by LLM-driven agents on a single professionally annotated UAV corpus. The same image set and the same per-class AP_50 metric are evaluated under two protocols. The VLM Track measures whether a fixed VLM can localise domain
• ChildAgentEval (Total (self-reported)): GPT-5.4 leads with 53.0 across 6 models. While agentic AI and its core multimodal large language models (MLLMs) have demonstrated remarkable promise in language and visual reasoning across domains ranging from daily life to advanced scientific research, a profound gap remains between artificial and human intelligence. Despite the integration of powerful tools and advanced MLLMs, state-of-the-art AI agents frequently fail at foundational,
• STT-Arena (Overall (self-reported)): Claude Opus 4.6 leads with 35.39 across 23 models. Large language models (LLMs) deployed in real-world agentic applications must be capable of replanning and adapting when mid-task disruptions invalidate their prior decisions. Existing dynamic benchmarks primarily measure whether LLMs can detect temporal changes in a timely manner, leaving the complementary challenge of adaptive replanning under spatio-temporal dynamics largely unexplored. We intr
• SVFSearch (Overall Acc. (self-reported)): Qwen3.5-27B leads with 95.4 across 23 models. Multimodal large language models are increasingly used as agent backbones that understand multimodal inputs, plan retrieval actions, invoke external tools, and reason over retrieved information. Yet existing benchmarks rarely evaluate this ability in short-video applications, where a paused frame is often visually ambiguous and answering requires vertical, long-tail, and fast-evolving domain knowl
• Time to REFLECT (Overall (Report Quality) (self-reported)): GPT-5.3-Codex leads with 47.5 across 14 models. Deep research agents increasingly automate complex information-seeking tasks, producing evidence-grounded reports via multi-step reasoning, tool use, and synthesis. Their growing role demands scalable, reliable evaluation, positioning LLM-as-judge as a supervision paradigm for assessing factual accuracy, evidence use, and reasoning quality. Yet the reliability of these judges for deep research age
• ASPI (ASR exec_tool (self-reported)): DeepSeek V3.2 leads with 65.4 across 10 models. Clarification-seeking behavior is widely regarded as a desirable property of LLM agents, enabling them to resolve ambiguity before acting on underspecified tasks. However, the security implications of this interaction pattern remain unexplored. We investigate whether the transition from standard execution to a clarification-seeking state increases an agent's susceptibility to prompt injection atta
• CAM-Bench (Pass@32 (self-reported)): DeepSeek V4 Pro leads with 19.67 across 5 models. Formal theorem-proving benchmarks enable mechanically verifiable evaluation of mathematical reasoning in large language models. However, existing benchmarks mainly focus on Olympiad-style problems and algebraic domains, leaving computational and applied mathematics underrepresented. We introduce CAM-Bench, a Lean 4 theorem-proving benchmark of 1,000 Lean proof targets in computational and applied
• ContractBench (SR% (self-reported)): Claude Opus 4.6 leads with 77.8 across 20 models. Tool-augmented LLM agents call APIs whose intermediate outputs, such as presigned URLs, session tokens, and OAuth state parameters, are observation contracts: artifacts whose later use is constrained by the external system that produced them. We show that observation contract compliance (preserving the temporal validity and byte-level integrity) is an emergent, regression-prone capability: it is n
• ConsumerSimBench (Avg (95% CI) (self-reported)): Gemini 3.1 Pro Preview leads with 47.8 across 13 models. LLMs are increasingly used as ``digital consumers'' to simulate public opinion, pre-test marketing decisions, and anticipate audience response. However, existing evaluations rarely ask whether a model can reconstruct the concrete reaction patterns that real consumers surface in public discourse. We introduce ConsumerSimBench, a benchmark built from 1,553 real Chinese social-media topics and 23,122
• TOBench (Avg. (self-reported)): Qwen3.5 Plus 2026-04-20 leads with 41.0 across 18 models. Tool-using agents are increasingly expected to operate across realistic professional workflows, where they must interpret multimodal inputs, coordinate external tools, inspect intermediate artifacts, and revise their actions before producing a final result. Existing benchmarks, however, often evaluate tool use, computer use, and multimodal reasoning in isolation, leaving a gap between benchmark se
• RoadmapBench (Resolved (%) (self-reported)): Claude Opus 4.7 leads with 39.1 across 13 models. Coding agents are increasingly deployed in real software development, where a single version iteration requires months of coordinated work across many files. However, most existing benchmarks focus predominantly on single-issue bug fixes from Python repositories, with coarse pass/fail evaluation outcomes, and thus fail to capture long-horizon, multi-target development at real engineering scale. To
• SaaS-Bench (Overall (self-reported)): Claude Opus 4.7 leads with 43.9 across 15 models. Computer-Using Agents (CUAs) are rapidly extending large language models (LLMs) beyond text-based reasoning toward action execution in more complex environments, such as web browsers and graphical user interfaces (GUIs). However, existing web and GUI agent benchmarks often rely on simplified settings, isolated tasks, or short-horizon interactions, making it difficult to assess capabilities of agen
• Are Agents Ready to Teach? A Multi-Stage Bench (Eq. pass (self-reported)): GLM 5.1 leads with 63.8 across 11 models. Language agents are increasingly deployed in complex professional workflows, with tutoring emerging as a particularly high-stakes capability that remains largely unmeasured in existing benchmarks. Effective tutor agents require more than producing correct answers or executing accurate tool calls: a robust tutor must diagnose learner state, adapt support over time, make pedagogically justified deci
• Do Coding Agents Understand Least-Privilege Au (TSR (self-reported)): Full-Access leads with 94.0 across 11 models. As coding agents gain access to shells, repositories, and user files, least-privilege authorization becomes a prerequisite for safe deployment: an agent should receive enough authority to complete the task, without unnecessary authority that exposes sensitive surfaces. To study whether current models can infer this boundary themselves, we first introduce permission-boundary inference, where a mode
• RxEval (F1 (self-reported)): Gemini 3.1 Pro Preview leads with 77.1 across 17 models. Inpatient medication recommendation requires clinicians to repeatedly select specific medications, doses, and routes as a patient's condition evolves. Existing benchmarks formulate this task as admission-level prediction over coarse drug codes with multi-hot diagnostic and procedure code inputs, failing to capture the per-timepoint, information-rich nature of real prescribing. We propose RxEval, a
• SciPaths (F1 (self-reported)): Gemini 3.1 Pro Preview leads with 18.9 across 10 models. Scientific progress depends on sequences of enabling contributions, yet existing AI4Science benchmarks largely focus on citation prediction, literature retrieval, or idea generation rather than the dependencies that make progress possible. In this paper, we introduce discovery pathway forecasting: given a target scientific contribution and the prior literature available at a specified time, the ta
• CiteVQA (Overall SAA (self-reported)): Gemini 3.1 Pro Preview leads with 66.0 across 22 models. Multimodal Large Language Models (MLLMs) have significantly advanced document understanding, yet current Doc-VQA evaluations score only the final answer and leave the supporting evidence unchecked. This answer-only approach masks a critical failure mode: a model can land on the correct answer while grounding it in the wrong passage -- a critical risk in high-stakes domains like law, finance, and m
• ClawForge (Strict Acc. (self-reported)): Claude Opus 4.6 leads with 45.3 across 7 models. Interactive agent benchmarks face a tension between scalable construction and realistic workflow evaluation. Hand-authored tasks are expensive to extend and revise, while static prompt evaluation misses failures that only appear when agents operate over persistent state. Existing interactive benchmarks have advanced agent evaluation significantly, but most initialize tasks from clean state and do
• Ego2World (Goal Tasks (self-reported)): GLM 5 leads with 183.0 across 6 models. Embodied agents in household environments must plan under partial observation: they need to remember objects, track state changes, and recover when actions fail. Existing benchmarks only partially test this ability. Egocentric video datasets capture realistic human activities but remain passive, while interactive simulators support execution but rely on synthetic scenes and hand-crafted dynamics,
• PerfCodeBench (CGRE (self-reported)): GPT-5 leads with 73.99 across 20 models. Large language models (LLMs) can often generate functionally correct code, but their ability to produce efficient implementations for performance-critical systems tasks remains limited. Existing code benchmarks mainly emphasize correctness or algorithmic problem solving, while realistic systems-level optimization is still underexplored. To address this gap, we introduce PerfCodeBench, an executabl
• AcuityBench (QA Exact (self-reported)): Claude Opus 4.7 leads with 85.3 across 12 models. We introduce AcuityBench, a benchmark for evaluating whether language models identify the appropriate urgency of care from user medical presentations. Existing health benchmarks emphasize medical question answering, broad health interactions, or narrow workflow-specific triage tasks, but they do not offer a unified evaluation of acuity identification across these settings. AcuityBench addresses th
• Do Enterprise Systems Need Learned World Model (CascadeBench IoU w/ BR (self-reported)): Qwen-3.5-27B-LoRA leads with 50.9 across 10 models. World models enable agents to anticipate the effects of their actions by internalizing environment dynamics. In enterprise systems, however, these dynamics are often defined by tenant-specific business logic that varies across deployments and evolves over time, making models trained on historical transitions brittle under deployment shift. We ask a question the world-models literature has not addr
• Human-Grounded Multimodal Benchmark with 900K (Science MC Accuracy (self-reported)): GPT-5 leads with 90.9 across 11 models. Authentic school examinations provide a high-validity test bed for evaluating multimodal large language models (MLLMs), yet benchmarks grounded in Japanese K-12 assessments remain scarce. We present a multimodal dataset constructed from Japan's National Assessment of Academic Ability, comprising officially released middle-school items in Science, Mathematics, and Japanese Language. Unlike existing
• MMCL-Bench (Overall (self-reported)): GPT-5.4 leads with 26.5 across 5 models. We introduce MMCL-Bench, a benchmark for multimodal context learning: learning task-local rules, procedures, and empirical patterns from visual or mixed-modality teaching context and applying them to new visual instances. Unlike text-only context learning or standard multimodal question answering, this setting requires models to recover and localize relevant evidence from images, screenshots, manu
• SpatialBabel (Three.js (self-reported)): Gemini 3 leads with 83.4 across 14 models. Vision-language models (VLMs) exhibit a striking paradox: they can generate executable code that reconstructs a 3D scene from geometric primitives with correct object counts, classes, and approximate positions, yet the same models fail at simpler spatial questions on the same image. We show that 3D geometric primitives (cubes, spheres, cylinders, expressed in executable code) serve as a powerful i
• Visual Aesthetic Benchmark (VAB) (Overall Top-1 ap@1 (self-reported)): Human Expert leads with 77.7 across 22 models. Multimodal large language models (MLLMs) are now routinely deployed for visual understanding, generation, and curation. A substantial fraction of these applications require an explicit aesthetic judgment. Most existing solutions reduce this judgment to predicting a scalar score for a single image. We first ask whether such scores faithfully capture comparative preference: in a controlled study wit
• Agent-ValueBench (Authority (self-reported)): Grok 4.20 leads with 7.8 across 14 models. Agent value-alignment benchmark with executable environments and value-conflict tasks, testing whether autonomous agents express stable values across domains, harnesses, and trajectories.
• CADBench (IoU Aggregate (self-reported)): CADFit leads with 85.9 across 11 models. Recovering editable CAD programs from images or 3D observations is central to AI-assisted design, but progress is difficult to measure because existing evaluations are fragmented across datasets, modalities, and metrics. We introduce CADBench, a unified benchmark for multimodal CAD program generation. CADBench contains 18,000 evaluation samples spanning six benchmark families derived from DeepCAD,
• Greenland Sovereignty Game (implicitly as a st (Escalation Composite (self-reported)): MoonshotAI: Kimi K2.6 leads with 22.0 across 8 models. What happens when the strongest alliance member pressures a weaker member over territory and strategic control? We examine the Greenland sovereignty crisis as a stress test for LLM geopolitics, centered on the 2019-2026 U.S. push to acquire Greenland from the Kingdom of Denmark. The crisis nests two collective-action problems: Arctic strategic control and whether NATO can enforce alliance norms ag
• gwBenchmarks (Waveform (self-reported)): Haiku 4.5 leads with 59.3 across 12 models. Modern gravitational wave astronomy relies on modeling tasks that often require months of graduate-level effort, including building fast waveform surrogates from expensive numerical relativity simulations, modeling orbital dynamics of black holes, fitting merger remnant properties and constructing template banks. These problems demand extreme precision to support detection and parameter inference,
• IndustryBench (Final (SV) (self-reported)): Gemini 3.1 Pro Preview leads with 2.08 across 17 models. In industrial procurement, an LLM answer is useful only if it survives a standards check: recommended material must match operating condition, every parameter must respect a regulated threshold, and no procedure may contradict a safety clause. Partial correctness can mask safety-critical contradictions that aggregate LLM benchmarks rarely capture. We introduce IndustryBench, a 2,049-item benchmark
• KnotBench (Accuracy (%) (self-reported)): Claude Opus 4.7 leads with 54.6 across 2 models. A vision-language model can look at a knot diagram and report what it sees, yet fail to act on that structure. KnotBench pairs an 858,318-image corpus from 1,951 prime-knot prototypes (crossing numbers 3 to 19) with a protocol whose answers are checked against Regina's canonical knot signature. Its 14 tasks span four families, equivalence judgment, move prediction, identification, and cross-modal
• LITMUS (Attack Success Rate (self-reported)): DeepSeek V3.2 leads with 71.51 across 6 models. The rapid proliferation of LLM-based autonomous agents in real operating system environments introduces a new category of safety risk beyond content safety: behavior jailbreak, where an adversary induces an agent to execute dangerous OS-level operations with irreversible consequences. Existing benchmarks either evaluate safety at the semantic layer alone, missing physical-layer harms, or fail to i
• Metacognitive Probe (Mean(T1,T2,T4,T5) (self-reported)): Claude Sonnet 4.6 leads with 82.0 across 8 models. The Metacognitive Probe is an exploratory five-task, 15-slot diagnostic that decomposes an LLM's confidence behaviour into five behaviourally-distinct dimensions: confidence calibration (T1-CC), epistemic vigilance (T2-EV), knowledge boundary (T3-KB), calibration range (T4-CR), and reasoning-chain validation (T5-RCV). It is evaluated on N=8 frontier models and N=69 humans. The instrument is motiva
• Multi-domain Multi-modal Document Classificati (HF1 (self-reported)): Gemini 3.1 Pro Preview leads with 64.98 across 15 models. Document classification forms the backbone of modern enterprise content management, yet existing benchmarks remain trapped in oversimplified paradigms -- single domain settings with flat label structures -- that bear little resemblance to the hierarchical, multi-modal, and cross-domain nature of real-world business documents. This gap not only misrepresents practical complexity but also stifles pr
• PaperFit-Bench (Compile (self-reported)): GPT-5.4 leads with 100.0 across 4 models. A LaTeX manuscript that compiles without error is not necessarily publication-ready. The resulting PDFs frequently suffer from misplaced floats, overflowing equations, inconsistent table scaling, widow and orphan lines, and poor page balance, forcing authors into repetitive compile-inspect-edit cycles. Rule-based tools are blind to rendered visuals, operating only on source code and log files. Tex
• Polaris-Bench (Overall C (self-reported)): Gemini 3.1 Pro Preview leads with 82.6 across 15 models. As current Multimodal Large Language Models rapidly saturate canonical visual reasoning benchmarks, a key question emerges: do these strong scores genuinely reflect robust visual understanding? We identify a pervasive vulnerability, the Cartesian Shortcut: visual reasoning benchmarks prevalently build on orthogonal grid-based layouts that can be readily discretized into explicit textual coordinate
• StereoTales (Emissions (self-reported)): Grok 4 leads with 304.0 across 23 models. Multilingual studies of social bias in open-ended LLM generation remain limited: most existing benchmarks are English-centric, template-based, or restricted to recognizing pre-specified stereotypes. We introduce StereoTales, a multilingual dataset and evaluation pipeline for systematically studying the emergence of social bias in open-ended LLM generation. The dataset covers 10 languages and 79 so
• Ambig-DS (Full (self-reported)): Gemini 3.1 Pro Preview leads with 64.0 across 5 models. As data-science agents shift from co-pilots to auto-pilots, silent misframing becomes a critical failure mode. Agents quietly commit to plausible but unintended task framings, producing clean, executable artifacts that hide their incorrect assessment of the task. Existing benchmarks score whether the pipeline runs, ignoring whether the agent recognized the task was underspecified. We introduce Amb
• CalBench (Excess (self-reported)): GPT-5.4 Mini leads with 149.0 across 7 models. Personal AI assistants are beginning to act as delegates with access to calendars, inboxes, and user preferences. Calendar scheduling makes the trust problem concrete: an assistant must coordinate with other assistants while deciding what to reveal about the person it represents. We introduce CalBench, a controlled benchmark for multi-agent calendar scheduling under private information. In each ta
• CodeClinic (Overall (self-reported)): Claude Sonnet 4.6 leads with 53.1 across 8 models. Clinical reasoning agents based on large language models (LLMs) aim to automate tasks such as intensive care unit (ICU) monitoring and patient state tracking from electronic health records (EHRs). Existing systems typically rely on manually curated clinical tools or skills for concepts such as sepsis detection and organ failure assessment. However, maintaining these tool libraries requires substan
• SeePhys Pro (Cons4 (self-reported)): Human Performance leads with 49.0 across 16 models. We introduce SeePhys Pro, a fine-grained modality transfer benchmark that studies whether models preserve the same reasoning capability when critical information is progressively transferred from text to image. Unlike standard vision-essential benchmarks that evaluate a single input form, SeePhys Pro features four semantically aligned variants for each problem with progressively increasing visual
• TraceEval (Average F1 (self-reported)): Claude Opus 4.6 leads with 72.9 across 10 models. Evaluating whether large language models (LLMs) can recover execution-relevant program structure, rather than only produce code that passes tests, remains an open problem. Existing code benchmarks emphasize test-passing outputs, from standalone programming tasks (HumanEval, MBPP, LiveCodeBench) to repository repair (SWE-Bench); this is useful, but offers limited diagnostic signal about which progr
• Beyond the All-in-One Agent (avg. (self-reported)): DeepSeek V4 Pro leads with 62.0 across 12 models. Large language model (LLM) agents are increasingly expected to operate in enterprise environments, where work is distributed across specialized roles, permission-controlled systems, and cross-departmental procedures. However, existing enterprise benchmarks largely evaluate single agents with broad tool access, while existing multi-agent benchmarks rarely capture realistic enterprise constraints su
• DiagnosticIQ (Macro % D.IQ (self-reported)): Claude Opus 4.6 leads with 73.59 across 33 models. Monitoring complex industrial assets relies on engineer-authored symbolic rules that trigger based on sensor conditions and prompt technicians to perform corrective actions. The bottleneck is not detection but response: translating rules into maintenance steps requires asset-specific knowledge gained through years of practice. We investigate whether LLMs can serve as decision support for this rule
• DocScope (ACC All (self-reported)): Gemini 3.1 Pro Preview leads with 78.9 across 23 models. Evaluating whether Multimodal Large Language Models can produce trustworthy, verifiable reasoning over long, visually rich documents requires evaluation beyond end-to-end answer accuracy. We introduce DocScope, a benchmark that formulates long-document QA as a structured reasoning trajectory prediction problem: given a complete PDF document and a question, the model outputs evidence pages, support
• Done, But Not Sure (B All (self-reported)): Gemini 3.1 Pro Preview leads with 56.4 across 20 models. Standard embodied evaluations do not independently score whether an agent correctly commits to task completion at episode closure, a capacity we call terminal commitment. Behaviorally distinct failures--never completing the task, completing it but failing to stop, and reporting success without sufficient evidence--collapse into the same benchmark failure. We introduce VIGIL, an evaluation framewor
• FORTIS (Task 1 EM (self-reported)): Claude Opus 4.7 leads with 54.8 across 10 models. Large language model agents increasingly operate through an intermediate skill layer that mediates between user intent and concrete task execution. This layer is widely treated as an organizational abstraction, but we argue it is also a privilege boundary that current models routinely exceed. We present \\textbf{FORTIS}, a benchmark that evaluates over-privilege in agent skills across two stages:
• ProactBench (Overall Pass Rate (self-reported)): GPT-5.5 leads with 61.5 across 16 models. Most LLM benchmarks score how well a model responds to explicit requests. They leave unmeasured a different conversational ability: noticing and acting on needs the user has implied but not said. We call this \\emph{conversational proactivity}. ProactBench decomposes it into three phase-tied types: \\textsc{Emergent}, inference from a single disclosed anchor; \\textsc{Critical}, synthesis across m
• EnvSimBench (Cm Overall (self-reported)): Ours (Full-Balance2, 4B) leads with 45.3 across 8 models. Scalable AI agents training relies on interactive environments that faithfully simulate the consequences of agent actions. Manually crafted environments are expensive to build, brittle to extend, and fundamentally limited in diversity. A promising direction is to replace manually crafted environments with LLM-simulated counterparts. However, this paradigm hinges on an unexamined core assumption: L
• From 0-Order Selection to 2-Order Judgment (H-Comb (self-reported)): Human leads with 79.5 across 13 models. Multiple-choice reasoning benchmarks face dual challenges: rapid saturation from advancing models and data contamination that undermines static evaluations. Ad-hoc hardening methods (paraphrasing, perturbation) attempt to increase difficulty but sacrifice logical validity for surface complexity, falling short to challenge advanced reasoning models. We present LogiHard, a formal framework that dete
• InterLV-Search (Level 3 +Tool Avg (self-reported)): Gemini 3.1 Pro Preview leads with 46.46 across 8 models. Existing benchmarks for multimodal agentic search evaluate multimodal search and visual browsing, but visual evidence is either confined to the input or treated as an answer endpoint rather than part of an interleaved search trajectory. We introduce \\textbf{InterLV-Search}, a benchmark for Interleaved Language-Vision Agentic Search, in which textual and visual evidence is repeatedly used to condi
• MathConstraint (Accuracy (self-reported)): GPT-5.5 leads with 66.9 across 12 models. We introduce MathConstraint, a hard, adaptive benchmark for evaluating the combinatorial reasoning capabilities of LLMs. We combine constraint satisfaction problems with rigorous solver-based verification and design an adaptive generator to create instances that remain challenging as the LLMs improve in their reasoning capabilities. Unlike existing benchmarks that quickly saturate on fixed dataset
• NARRA-Gym for Evaluating Interactive Narrative (StoryQ (self-reported)): Claude Sonnet 4.6 leads with 3.9 across 9 models. Interactive narrative tasks require LLMs to sustain a coherent, evolving story while adapting to a user over multiple turns. However, suitable benchmarks for this setting are limited: existing evaluations often focus on static prompts, isolated story generations, or post-hoc ratings, and therefore miss whether models can jointly manage story generation, long-context state and pacing, character sim
• TeamBench (Solo (self-reported)): Claude Opus 4.7 leads with 35.6 across 13 models. Agent systems often decompose a task across multiple roles, but these roles are typically specified by prompts rather than enforced by access controls. Without enforcement, a team pass rate can mask whether agents actually coordinated or whether one role effectively did another role's work. We present TeamBench, a benchmark with 851 task templates and 931 seeded instances for evaluating agent coor
• VeriContest (End-to-End (end2end) (self-reported)): GPT-5.5 leads with 5.29 across 10 models. Large language models can generate useful code from natural language, but their outputs come without correctness guarantees. Verifiable code generation offers a path beyond testing by requiring models to produce not only executable code, but also formal specifications and machine-checkable proofs. Progress in this direction, however, is difficult to measure: existing benchmarks are often small, fo
• An Empirical Study of Proactive Coding Assista (Pass@1 (self-reported)): Claude Sonnet 4.6 leads with 13.57 across 7 models. Large language model (LLM)-based coding assistants have made substantial progress, yet most systems remain reactive, requiring developers to explicitly formulate their needs. Proactive coding assistants aim to infer latent developer intent from integrated development environment (IDE) interactions and repository context, thereby reducing interaction overhead and supporting more seamless assistance
• Artificial Intelligence Quotient (AIQ) Benchmark (Accuracy (self-reported)): Gemini 3.1 Pro Preview leads with 99.46 across 6 models. The pursuit of artificial general intelligence necessitates robust methods for evaluating the cognitive capabilities of models beyond narrow task performance. Here, we introduce a psychometric framework to assess the cognitive profiles of generative AI, comparing them to human norms and tracking their evolution across generations. Initial evaluation of leading multimodal models using tasks adapted
• Cited but Not Verified (Relevant Content (self-reported)): Claude Opus 4.5 leads with 95.7 across 14 models. Large language models (LLMs) power deep research agents that synthesize information from hundreds of web sources into cited reports, yet these citations cannot be reliably verified. Current approaches either trust models to self-cite accurately, risking bias, or employ retrieval-augmented generation (RAG) that does not validate source accessibility, relevance, or factual consistency. We introduce
• CVerifBench (Total (self-reported)): Claude Opus 4.7 leads with 98.3 across 14 models. We introduce an evaluation framework of 500 C verification tasks across five property types (memory safety, overflow, termination, reachability, data races) built on SV-COMP 2025, and evaluate 14 models across six families. We find that high overall accuracy masks a critical weakness: while most models reliably confirm properties hold, violation detection varies widely and degrades sharply with pr
• IntentGrasp (Overall (All Set) Avg (Std) (self-reported)): Gemini 3.1 Pro Preview leads with 59.68 across 20 models. Accurately understanding the intent behind speech, conversation, and writing is crucial to the development of helpful Large Language Model (LLM) assistants. This paper introduces IntentGrasp, a comprehensive benchmark for evaluating the intent understanding capability of LLMs. Derived from 49 high-quality, open-licensed corpora spanning 12 diverse domains, IntentGrasp is constructed through source
• PinTrace (ρ_U(%) (self-reported)): MoonshotAI: Kimi K2.5 leads with 45.78 across 10 models. Large language models (LLMs) are now largely involved in software development workflows, and the code they generate routinely includes third-party library (TPL) imports annotated with specific version identifiers. These version choices can carry security and compatibility risks, yet they have not been systematically studied. We present the first large-scale measurement study of version-level risk
• SmellBench (Weighted Effectiveness (E) (self-reported)): GPT-5.3-Codex leads with 47.8 across 9 models. Architectural code smells erode software maintainability and are costly to repair manually, yet unlike localized bugs, they require cross-module reasoning about design intent that challenges both developers and automated tools. While large language model agents excel at bug fixing and code-level refactoring, their ability to repair architectural code smells remains unexplored. We present the first
• STALE (Overall (self-reported)): CUPMem (Ours) leads with 68.0 across 15 models. Large Language Model (LLM) agents are increasingly expected to maintain coherent, long-term personalized memory, yet current benchmarks primarily measure static fact retrieval, overlooking the ability to revise stored beliefs when new evidence emerges. We identify a critical and underexplored failure mode, Implicit Conflict: a later observation invalidates an earlier memory without explicit negati
• XL-SafetyBench (Overall ASR (self-reported)): Mistral: Mistral Large 3 2512 leads with 98.8 across 10 models. Country-grounded cross-cultural safety benchmark with 5,500 test cases across 10 country-language pairs, separating universal jailbreak robustness from culturally embedded sensitivities.
• AA-LCR (Score (self-reported)): GPT-5.2-Codex leads with 75.7 across 331 models. AA-LCR evaluates model capability on long context tasks from the linked upstream source with Score as the primary reported metric.
• Creative Writing v3 (Elo score (self-reported)): Claude Opus 4.7 leads with 2215.9 across 102 models. Creative Writing v3 evaluates model capability on writing tasks from the linked upstream source with Elo score as the primary reported metric.
• GeneBench (Mean pass rate (self-reported)): GPT-5.5 Pro leads with 33.2 across 16 models. Genetics and quantitative-biology benchmark where models analyze noisy scientific data, detect confounders, and implement statistical workflows with minimal guidance.
• Harvey Legal Agent Benchmark (All-Pass Task Success (self-reported)): Claude Mythos 5 leads with 16.91 across 11 models. Legal-agent benchmark for completing realistic legal workflows with all-pass grading, including held-out Harvey tasks and public legal-agent task sets.
• HMMT 2025 (Score (self-reported)): GPT-5.2 OpenAI leads with 100.0 across 61 models. MathArena evaluation based on Harvard-MIT Mathematics Tournament 2025 problems, emphasizing olympiad-style high-school contest reasoning.
• MathArena Apex (Score (self-reported)): GPT-5.5 OpenAI leads with 80.21 across 47 models. MathArena Apex is a challenging math contest benchmark featuring the most difficult mathematical problems designed to test advanced reasoning and problem-solving abilities of AI models. It focuses on olympiad-level mathematics and complex multi-step mathematical reasoning.
• OmniDocBench 1.5 (Overall (self-reported)): PaddleOCR-VL-1.5 leads with 94.5 across 50 models. Document-understanding benchmark covering OCR, layout parsing, tables, formulas, and information extraction across diverse document types.
• CC-OCR V2 (Average (self-reported)): Qwen3.6 Plus leads with 75.77 across 15 models. Large Multimodal Models (LMMs) have recently shown strong performance on Optical Character Recognition (OCR) tasks, demonstrating their promising capability in document literacy. However, their effectiveness in real-world applications remains underexplored, as existing benchmarks adopt task scopes misaligned with practical applications and assume homogeneous acquisition conditions. To address this
• MCJudgeBench (CJAR (self-reported)): Gemini 3.1 Pro Preview leads with 85.8 across 7 models. Multi-constraint instruction following requires verifying whether a response satisfies multiple individual requirements, yet LLM judges are often assessed only through overall-response judgments. We introduce MCJudgeBench, a benchmark for constraint-level judge evaluation in multi-constraint instruction following. Each instance includes an instruction, a candidate response, an explicit constraint
• AcademiClaw (Pass Rate (self-reported)): Claude Opus 4.6 leads with 55.0 across 7 models. Benchmarks within the OpenClaw ecosystem have thus far evaluated exclusively assistant-level tasks, leaving the academic-level capabilities of OpenClaw largely unexamined. We introduce AcademiClaw, a bilingual benchmark of 80 complex, long-horizon tasks sourced directly from university students' real academic workflows -- homework, research projects, competitions, and personal projects -- that the
• DataClawBench (Overall Acc. (self-reported)): Claude Opus 4.6 leads with 63.4 across 8 models. Autonomous data analysis agents are increasingly expected to conduct exploratory analysis with limited human guidance about data. However, existing benchmarks typically evaluate such agents in prior-guided settings, providing selected data sources, explicit data schemas, or cleaned data, thereby understating the exploratory burden. To evaluate this realistic exploratory data analysis task, we intr
• MolViBench (Pass@1 rate (self-reported)): Claude Opus 4.6 Think (IR) leads with 39.7 across 13 models. Molecular Vibe Coding, a paradigm where chemists interact with LLMs to generate executable programs for molecular tasks, has emerged as a flexible alternative to chemical agents with predefined tools, enabling chemists to express arbitrarily complex, customized workflows. Unlike general coding tasks, molecular coding imposes a distinctive challenge that LLMs should jointly equip programming, molec
• PhysicianBench (Pass@1 (self-reported)): GPT-5.5 OpenAI leads with 46.3 across 12 models. Long-horizon physician workflow benchmark grounded in clinical records, measuring checkpoint and end-to-end task success.
• The Compliance Trap (Δ (degradation) (self-reported)): Qwen3-80B Thinking leads with 13.6 across 10 models. As frontier AI models are deployed in high-stakes decision pipelines, their ability to maintain metacognitive stability (knowing what they do not know, detecting errors, seeking clarification) under adversarial pressure is a critical safety requirement. Current safety evaluations focus on detecting strategic deception (scheming); we investigate a more fundamental failure mode: cognitive collapse.
• TSCG (20 Tools (json-text) (self-reported)): Qwen3 14B leads with 90.2 across 13 models. Production agent frameworks (OpenAI Function Calling, Anthropic Tool Use, MCP) transmit tool schemas as JSON, a format designed for machine parsing, not for interpretation by language models. For small models (4B-14B), this protocol mismatch accounts for the majority of tool-use failure at production catalog sizes. We present TSCG, a deterministic tool-schema compiler that resolves this mismatch a
• HealthBench Professional (Score (self-reported)): Claude Mythos 5 leads with 66.0 across 14 models. HealthBench professional subset for medically challenging, expert-oriented healthcare question answering.
• BioMysteryBench Human-Difficult (Accuracy (self-reported)): Claude Mythos 5 leads with 46.1 across 7 models. Anthropic BioMysteryBench slice covering 23 real-world bioinformatics tasks no human benchmarker solved after QC, evaluated by average accuracy across five trials per problem.
• BioMysteryBench Human-Solvable (Accuracy (self-reported)): Claude Mythos 5 leads with 83.9 across 7 models. Anthropic BioMysteryBench slice covering 76 real-world bioinformatics tasks solved by at least one human benchmarker, evaluated by average accuracy across five trials per problem.
• OccuBench (Completion rate (self-reported)): Gemini 3.1 Pro Preview leads with 45.3 across 15 models. Professional-task benchmark using simulated domain tool environments to evaluate LLM agents across occupation-specific workflows.
• EnterpriseArena (Full Survival % (self-reported)): Human leads with 60.0 across 26 models. EnterpriseArena evaluates LLM agents as CFO-style decision makers in a 132-month FinTech lending simulator. Agents manage liquidity, close books, buy costly signals, and choose equity or debt financing under partial observability, hard resource budgets, delayed consequences, and changing macroeconomic regimes. (Source: benchmarklist.com, self-reported.)
• OrgForge-IT (Verdict F1 (self-reported)): Claude Opus 4.6 leads with 100.0 across 10 models. Synthetic insider-threat detection benchmark built from OrgForge organizational simulation telemetry with triage, verdict, and false-positive scoring.
• EnterpriseOps-Gym (Task Success Rate (self-reported)): Claude Opus 4.6 leads with 44.6 across 22 models. Stateful enterprise operations benchmark for LLM agents performing long-horizon planning, tool use, and policy-governed workflows.
• Vibe Code Bench v1.1 (Score (self-reported)): Claude Fable 5 maxAnthropic leads with 90.35 across 55 models. Vals AI benchmark for vibe-coding agents that build complete applications from product-style prompts and are scored on functional correctness and quality.
• Arena AI Document (Arena ELO (self-reported)): Claude Opus 4.6 leads with 1526.0 across 19 models. Crowdsourced Arena AI pairwise human-preference leaderboard for PDF and document-understanding models.
• LABBench2 Clinical Trials (Score (self-reported)): Claude Mythos 5 leads with 91.2 across 4 models. LABBench2 clinical-trials subset reported in Anthropic's Claude Opus 4.8 system card.
• LABBench2 Patent Questions (Score (self-reported)): Claude Mythos 5 leads with 79.8 across 4 models. LABBench2 patent-question subset reported in Anthropic's Claude Opus 4.8 system card.
• DeepSearchQA (Score (self-reported)): Claude Mythos Preview leads with 94.4 across 5 models. Deep-search question-answering benchmark for agents that must gather, compare, and synthesize evidence across multi-hop web research tasks.
• APEX-Agents (Mean Score (ReAct) (self-reported)): Gemini 3.5 Flash leads with 66.1 across 40 models. The AI Productivity Index for Agents (APEX-Agents) measures whether frontier AI agents can execute long-horizon, cross-application tasks across three jobs in professional services.
• APEX-Agents-AA (Pass@1 (self-reported)): Gemini 3.5 Flash leads with 47.1 across 24 models. Artificial Analysis implementation of APEX-Agents using the Stirrup agent harness for long-horizon, cross-application professional-services tasks.
• EnigmaEval (Score (self-reported)): GPT-5.4 Pro leads with 23.82 across 39 models. EnigmaEval is a benchmark from puzzle hunts, testing AI with complex reasoning, creative problem-solving, and cross-domain knowledge synthesis.
• MCP Atlas (Score (self-reported)): Gemini 3.5 Flash leads with 83.6 across 32 models. Evaluating real-world tool use through the Model Context Protocol (MCP).
• MultiNRC (Score (self-reported)): GPT-5 Pro leads with 65.2 across 39 models. MultiNRC benchmarks LLMs on 1,000+ culturally grounded reasoning questions by native French, Spanish, and Chinese speakers across four reasoning categor...
• PRBench Finance (Score (self-reported)): Claude Opus 4.6 leads with 53.28 across 28 models. Professional Reasoning Bench Finance evaluates frontier LLMs on complex financial reasoning tasks including analysis, modeling, and decision-making.
• Professional Reasoning Bench (Score (self-reported)): Muse Spark leads with 52.29 across 28 models. Professional Reasoning Bench Legal evaluates frontier LLMs on complex legal reasoning tasks drawn from real-world legal practice and case analysis. (Source: benchmarklist.com, self-reported.)
• SWE Atlas (Score (self-reported)): NexAU leads with 45.4 across 16 models. SWE Atlas Codebase QnA evaluates LLMs on deep code comprehension and question answering across real-world software repositories. (Source: benchmarklist.com, self-reported.)
• TutorBench (Score (self-reported)): Muse Spark leads with 68.55 across 23 models. TutorBench evaluates how well LLMs perform common tutoring tasks for high school and AP-level subjects.
• Visual-Language Understanding (Score (self-reported)): Gemini 2.5 Pro Experimental (March 2025) leads with 54.65 across 54 models. Scale's SEAL Leaderboard evaluates top models' visual-language understanding, testing perception, logic, calculation, and common sense.
• VTB (Score (self-reported)): GPT-5.4 high leads with 29.17 across 17 models. Evaluating how LLMs can dynamically interact with and reason about visual information.
• NL2Repo (Score (self-reported)): Claude Opus 4.8 leads with 69.7 across 12 models. NL2Repo evaluates long-horizon coding capabilities including repository-level understanding, where models must generate or modify code across entire repositories from natural language specifications.
• Arena AI Code (Arena ELO (self-reported)): Claude Opus 4.7 leads with 1570.0 across 64 models. Crowdsourced Arena AI pairwise human-preference leaderboard for code generation and coding-assistant models.
• LMArena WebDev Arena (Arena rating (self-reported)): Claude Opus 4.7 leads with 1567.85 across 21 models. LMArena's WebDev Arena leaderboard for model performance on interactive web development tasks judged by human preference.
• IMO-AnswerBench (Score (self-reported)): DeepSeek V4 Flash leads with 91.1 across 14 models. International Mathematical Olympiad answer benchmark evaluating final-answer correctness on high-difficulty olympiad-style mathematical problems.
• Toolathlon (Score (self-reported)): Claude Fable 5 leads with 61.7 across 18 models. Tool-use benchmark spanning many tool categories, testing whether agents can select, sequence, and combine tools to complete realistic tasks.
• CritPt (Accuracy (self-reported)): GPT-5.5 Pro leads with 30.6 across 328 models. Research-level physics reasoning benchmark with composite challenges designed by active physics researchers.
• GDPval (Wins/Ties vs Human (self-reported)): GPT-5.5 leads with 84.9 across 18 models. Real-world, economically valuable knowledge work tasks across 44 occupations and 9 U.S. GDP sectors.
• OSWorld-Verified (Score (self-reported)): Claude Mythos Preview max leads with 85.4 across 20 models. OSWorld-Verified evaluates model capability on agentic tasks from the linked upstream source with Score as the primary reported metric.
• CyberGym (Score (self-reported)): Claude Mythos 5 leads with 83.8 across 8 models. Cybersecurity agent benchmark for discovering, exploiting, and reasoning about vulnerabilities in controlled challenge environments.
• LiveSQLBench (Success Rate (self-reported)): Gemini 3.1 Pro Preview leads with 43.1 across 33 models. Dynamic contamination-free text-to-SQL benchmark for real-world database tasks, including business-intelligence queries, CRUD/management SQL, hierarchical knowledge bases, and large industrial-scale database variants.
• HealthBench Hard (Overall score (self-reported)): gpt-oss-120b leads with 60.0 across 55 models. Hard subset of HealthBench, evaluating difficult clinical and biomedical advice with physician-written rubrics and stricter scoring.
• ChartQAPro (With tools score (self-reported)): Claude Mythos Preview leads with 73.6 across 4 models. Harder chart-understanding evaluation for professional and technical visual-question-answering tasks.
• ScreenSpot-Pro (Grounding score (self-reported)): Claude Mythos Preview leads with 93.0 across 34 models. Professional GUI grounding benchmark requiring agents to identify precise screen locations in high-resolution development, creative, and scientific software.
• ITBench-AA (Average Precision at Full Recall (self-reported)): Claude Opus 4.7 max leads with 46.7 across 23 models. Artificial Analysis implementation of IBM\
• MedXpertQA (Score (self-reported)): Gemini 3.1 Pro Preview leads with 80.7 across 19 models. MedXpertQA evaluates model capability on healthcare & medical tasks from the linked upstream source with Score as the primary reported metric.
• MultiChallenge (Score (self-reported)): Muse Spark leads with 75.52 across 34 models. MultiChallenge evaluates frontier LLMs on realistic multi-turn conversations, assessing instruction retention, inference memory, and self-coherence.
• OCRBench v2 (Average (self-reported)): KDL Frontier🥇 leads with 68.1 across 29 models. OCRBench v2 evaluates large multimodal models on bilingual visual text localization and reasoning tasks.
• OCRBench-V2 (en) (Score (self-reported)): Qwen3.7 Plus leads with 70.7 across 26 models. OCRBench v2 English subset: Enhanced benchmark for evaluating Large Multimodal Models on visual text localization and reasoning with English text content.
• OCRBench-V2 (zh) (Score (self-reported)): Qwen3.7 Plus leads with 67.1 across 26 models. OCRBench v2 Chinese subset: Enhanced benchmark for evaluating Large Multimodal Models on visual text localization and reasoning with Chinese text content.
• MMMU Pro (Score (self-reported)): Claude Fable 5 maxAnthropic leads with 89.31 across 68 models. MMMU Pro evaluates model capability on intelligence & reasoning tasks from the linked upstream source with Score as the primary reported metric.
• FigQA (Score (self-reported)): Claude Mythos 5 leads with 90.7 across 4 models. FigQA evaluates model capability on multimodal tasks from the linked upstream source with Score as the primary reported metric.
• LiveBench (LiveBench average (self-reported)): GPT-5.5 x-high leads with 81.28 across 43 models. Continuously updated benchmark measuring many capabilities. Sort models by weighted score or sub-task scores. Tests math, coding, reasoning, language, instruction following, and data analysis.
• CharXiv-R (Score (self-reported)): Claude Mythos 5 leads with 93.5 across 12 models. CharXiv-R evaluates model capability on multimodal tasks from the linked upstream source with Score as the primary reported metric.
• LVBench (Score (self-reported)): GPT-5.4 leads with 77.4 across 27 models. LVBench evaluates model capability on multimodal tasks from the linked upstream source with Score as the primary reported metric.
• VideoMME w sub. (Score (self-reported)): GPT-5.4 leads with 89.5 across 56 models. VideoMME w sub. evaluates model capability on multimodal tasks from the linked upstream source with Score as the primary reported metric.
• LegalBench (Score (self-reported)): Claude Fable 5 maxAnthropic leads with 88.56 across 107 models. Evaluating language models on a wide range of open source legal reasoning tasks.
• LMArena Text Arena (Arena rating (self-reported)): Claude Opus 4.6 leads with 1500.24 across 19 models. Crowdsourced pairwise human-preference leaderboard for text chat models in LMArena, formerly LMSYS Chatbot Arena.
• 100Q-Hard Net Score (Net score (self-reported)): Claude Mythos 5 leads with 42.0 across 7 models. Closed-book factuality benchmark reported by Anthropic as net score: correct responses minus incorrect responses, with abstentions scoring zero.
• AA-Omniscience Net Score (Net score (self-reported)): Claude Mythos 5 leads with 53.0 across 7 models. AA-Omniscience factuality results reported by Anthropic as net score: correct responses minus incorrect responses, with abstentions scoring zero.
• AIIQ Composite IQ (Composite IQ (self-reported)): GPT-5.5 leads with 136.0 across 47 models. AIIQ composite estimate that combines abstract, mathematical, programmatic, and academic reasoning benchmark evidence into IQ-like model scores.
• ArxivMath (Score (self-reported)): Claude Fable 5 leads with 78.6 across 10 models. MathArena ArxivMath final-answer research-math benchmark slice from the March and April 2026 releases, as reported in Anthropic's Claude Opus 4.8 system card.
• AutoLab (Overall Score (self-reported)): Claude Opus 4.6 leads with 68.0 across 11 models. AutoLab evaluates AI agents on iterative performance-engineering tasks across model development, puzzle/challenge tasks, and system optimization.
• AutoMedBench (Average Overall Score (self-reported)): Claude Opus 4.6 leads with 69.69 across 7 models. AutoMedBench evaluates model capability on healthcare & medical tasks from the linked upstream source with Average Overall Score as the primary reported metric.
• BioPipelineBench Verified (Accuracy (self-reported)): Claude Mythos Preview leads with 88.1 across 4 models. Verified BioPipelineBench slice for bioinformatics pipeline tasks, reported in Anthropic's Claude Opus 4.8 system card.
• BLXBench (Score (self-reported)): Grok 4.3 leads with 85.5 across 25 models. Community benchmark runner and public leaderboard for AI model performance across coding, debugging, reasoning, hallucination, refactoring, security, and speed slices.
• CAIS Risk Index (Risk Index (self-reported)): Claude Opus 4.7 leads with 32.9 across 37 models. Composite CAIS AI Dashboard risk index averaging VCT refusal risk, HLE miscalibration, MASK risk, Machiavelli, and TextQuests Harm for models with all component scores. Lower is better.
• CAIS Text Capabilities Index (Text Capabilities Index (self-reported)): GPT-5.5 leads with 54.1 across 39 models. Composite CAIS AI Dashboard text index averaging Humanity's Last Exam, ARC-AGI-2, TextQuests, and SWE-bench Pro for models with all component scores.
• CAIS Vision Capabilities Index (Vision Capabilities Index (self-reported)): Gemini 3.5 Flash leads with 65.7 across 28 models. Composite CAIS AI Dashboard vision index averaging EnigmaEval, IntPhys2, ERQA, MindCube, ART, and SpatialViz for models with all component scores.
• CaseLaw v2 (Score (self-reported)): Grok 4.3 xAI leads with 79.31 across 53 models. Private question-answer benchmark over Canadian court-cases.
• Claw-Eval-Live (Pass Rate (self-reported)): Claude Opus 4.6 leads with 66.7 across 13 models. Quarterly refreshed enterprise-workflow benchmark grounded in live ClawHub marketplace signals and scored with deterministic checks plus structured judging.
• ClawProBench (Final Score (self-reported)): gpt-5.5-xhigh x-highopenai leads with 67.9 across 57 models. OpenClaw agent benchmark measuring model performance on reasoning, planning, tool use, reliability, efficiency, and safety across repeated runs.
• CorpFin v2 (Score (self-reported)): Claude Fable 5 maxAnthropic leads with 71.83 across 101 models. A private benchmark evaluating understanding of long-context credit agreements.
• CTFBench (Vulnerability Detection Rate (self-reported)): SavantChat Dec 2025 leads with 100.0 across 27 models. CTFBench: Measures model robustness, truthfulness, calibration, bias, harmfulness, jailbreak resistance, or alignment-relevant behavior.
• DuelLab Overall (Avg score (self-reported)): Claude Opus 4.7 Anthropic leads with 74.4 across 51 models. DuelLab evaluates model-generated game-playing programs by compiling submitted code and running head-to-head tournaments on hidden abstract strategy games.
• EQ-Bench (Normalized Elo (self-reported)): Claude Fable 5 leads with 2069.4 across 77 models. Emotional intelligence benchmark testing how well models understand and process complex emotional scenarios and nuanced human interactions.
• Finance Agent v1.1 (Score (self-reported)): Claude Opus 4.7 leads with 64.4 across 56 models. Finance Agent v1.1 evaluates model capability on finance tasks from the linked upstream source with Score as the primary reported metric.
• Finance Agent v2 (Score (self-reported)): Gemini 3.5 Flash leads with 57.9 across 33 models. Evaluating agents on core financial analyst tasks using the FAB v2 harness.
• Graphwalks BFS 1M F1 (F1 (self-reported)): Claude Mythos 5 leads with 79.4 across 7 models. Graphwalks breadth-first-search long-context reasoning task reported at 1M context with F1 scoring.
• Graphwalks BFS 256k F1 (F1 (self-reported)): Claude Mythos 5 leads with 91.1 across 7 models. Graphwalks breadth-first-search long-context reasoning task reported at 256k context with F1 scoring.
• Graphwalks Parents 1M F1 (F1 (self-reported)): Claude Mythos 5 leads with 97.5 across 7 models. Graphwalks parent-node long-context reasoning task reported at 1M context with F1 scoring.
• Graphwalks Parents 256k F1 (F1 (self-reported)): Claude Mythos 5 leads with 99.96 across 7 models. Graphwalks parent-node long-context reasoning task reported at 256k context with F1 scoring.
• HMMT February 2026 (Score (self-reported)): Qwen3.7 Max max leads with 97.1 across 24 models. Official Hugging Face benchmark for model performance on the February 2026 Harvard-MIT Mathematics Tournament problem set.
• INCLUDE (Score (self-reported)): Gemini 3.1 Pro Preview leads with 90.7 across 15 models. INCLUDE multilingual evaluation reported in Anthropic's Claude Opus 4.8 system card.
• Lech Mazur Writing (Comparison Score (self-reported)): GPT-5.5 x-highx-high leads with 3.4 across 32 models. Lech Mazur Writing evaluates model capability on writing tasks from the linked upstream source with Comparison Score as the primary reported metric.
• MedCode (Score (self-reported)): Gemini 3.1 Pro Preview highGoogle leads with 59.06 across 54 models. MedCode evaluates model capability on healthcare & medical tasks from the linked upstream source with Score as the primary reported metric.
• MedScribe (Score (self-reported)): Claude Fable 5 maxAnthropic leads with 88.52 across 53 models. Can models support doctors with their administrative work?.
• MILU (Score (self-reported)): Gemini 3.1 Pro Preview leads with 93.6 across 8 models. Multilingual knowledge-and-reasoning evaluation reported in Anthropic's Claude Opus 4.8 system card.
• MortgageTax (Score (self-reported)): Claude Opus 4.7 maxAnthropic leads with 70.27 across 72 models. Evaluating reading and understanding tax certificates as images.
• Multilingual Factual Questions Net Score (Net score (self-reported)): Claude Mythos Preview leads with 48.0 across 7 models. Closed-book multilingual factuality benchmark reported by Anthropic as net score: correct responses minus incorrect responses, with abstentions scoring zero.
• OpenClaw Arena Model Leaderboard (Avg Score (self-reported)): claude-opus-4.5 leads with 67.4 across 13 models. Personal AI agent benchmark evaluating frontier models across real-world OpenClaw-style tasks.
• PlaceboBench (Non-Hallucination Rate (self-reported)): Gemini 3 leads with 73.91 across 7 models. Medical-domain hallucination benchmark with labeled model answers to pharmaceutical questions grounded in EMA product information.
• ProofBench (Score (self-reported)): Claude Fable 5 maxAnthropic leads with 77.0 across 37 models. ProofBench evaluates model capability on math tasks from the linked upstream source with Score as the primary reported metric.
• ProteinGym Hard (Score (self-reported)): Claude Mythos 5 leads with 45.0 across 6 models. Hard ProteinGym subset reported in Anthropic's Claude Opus 4.8 system card.
• RealWorldQA (RealWorldQA (self-reported)): Qwen3.7 Plus leads with 86.9 across 12 models. RealWorldQA: Evaluates multimodal understanding across image, text, chart, diagram, or cross-modal reasoning tasks.
• Rogo Big Finance Bench (Rubric Score (self-reported)): Claude Opus 4.7 leads with 59.0 across 10 models. Vendor-reported 928-question finance-agent benchmark spanning vertical-specific skills, metrics, financial-statement analysis, and forecasting workflows.
• scBench (Accuracy (self-reported)): Claude Mythos 5 leads with 59.3 across 21 models. Bioinformatics agent benchmark with verifiable single-cell RNA-seq workflow tasks and deterministic graders.
• TaxBench (Mean pass^5 (computed) (self-reported)): GPT-5.5 Pro leads with 29.27 across 16 models. TaxBench evaluates AI models on real-world tax tasks from Rivet's active tax workflows, spanning tax knowledge and judgment, tax calculations, and agentic data-retrieval question answering.
• TaxEval v2 (Score (self-reported)): Muse Spark Meta leads with 77.68 across 109 models. A Vals-created set of questions and responses to tax questions.
• Vals Index (Score (self-reported)): Claude Fable 5 maxAnthropic leads with 75.14 across 25 models. Benchmark consisting of a weighted performance across finance and coding tasks. Showing the potential impact that LLM's can have on the economy.
• Vals Multimodal Index (Score (self-reported)): Claude Fable 5 maxAnthropic leads with 74.15 across 20 models. Benchmark consisting of a weighted performance across finance, coding, and education tasks. Showing the potential impact that LLM's can have on the economy.
• WildClawBench (Overall Score (self-reported)): Claude Opus 4.7 leads with 62.2 across 20 models. WildClawBench evaluates model capability on agentic tasks from the linked upstream source with Overall Score as the primary reported metric.
• KernelBench Hub - Mega (Best Speedup vs Reference (x)): Claude Opus 4.8 leads with 19.4 across 8 models. kernelbench.com Mega suite (independent of Stanford KernelBench) — agentic GPU megakernel building; best speedup over a reference megakernel across RTX PRO 6000, H100, and B200 runs.
• KernelBench Hub - Hard (Best % of Hardware Roofline): GLM-5.2 leads with 26.0 across 8 models. kernelbench.com Hard suite (independent of Stanford KernelBench) — agentic CUDA/Triton kernel optimization scored as percent of hardware roofline achieved, best across RTX PRO 6000, H100, and B200 runs.
• Benchmarks.bio - BioSecBench-Refusal (Pass Rate (%)): Gemini 3.5 Flash leads with 51.5 across 10 models. BioSecBench-Refusal (benchmarks.bio) — biosecurity refusal and over-refusal benchmark; red-team prompts that should be refused plus routine biology work that should be answered, scored as combined pass rate.
• Senior SWE-Bench (Tasteful Solve Rate (pass@1, %)): Claude Opus 4.8 leads with 24.0 across 10 models. Senior SWE-Bench (Snorkel) — 100 senior-engineer tasks from real production PRs (design-and-build, investigate-and-fix) with under-specified instructions; tasteful solve rate combines verifier passes, expert rubrics, and a code-quality taste judge.
• TerminalWorld (Pass Rate (%, 200 verified tasks)): Claude Opus 4.7 leads with 62.5 across 8 models. We introduce TerminalWorld, a scalable data engine that automatically reverse-engineers high-fidelity evaluation tasks from \
• Business Utility Eval (Business Utility (0-1)): Claude Opus 4.8 leads with 0.42 across 6 models. deepsense.ai's benchmark for how well LLMs perform realistic analytical business workflows — multi-step tasks over spreadsheets, reports and data where the model must reason to a business-useful answer rather than a single fact. Scored as a Business Utility rate; frontier models still score low (top model ~0.42), making it a hard, discriminating agentic-reasoning benchmark.
• Vals AI Excel Modeling (Accuracy (%)): claude-opus-4-8 leads with 69.37 across 17 models. Vals AI Excel Modeling Benchmark — financial spreadsheet modeling (LBO, DCF, M&A, comps) built from templates and from scratch; accuracy of produced Excel models.
• Vals AI CyberBench (Accuracy (%)): gpt-5.5 leads with 80.51 across 13 models. Vals AI CyberBench — offensive/defensive cybersecurity tasks run agentically; overall accuracy on private cyber task set.
• Vals AI Harvey Legal Agent Bench (Accuracy (%)): claude-fable-5 leads with 11.25 across 15 models. Harvey's Legal Agent Benchmark (Vals AI) — agentic legal work across documents, spreadsheets, presentations, and file-system tools spanning practice areas (M&A, antitrust, capital markets, tax); final score percentage.
• Vals AI Legal Research Bench (Accuracy (%)): claude-opus-4-8 leads with 43.75 across 14 models. Vals AI Legal Research Bench — legal research questions requiring case-law and statute lookup with citations; accuracy on private legal research task set.
• Vals AI ProgramBench (Raw Pass Rate (%)): claude-fable-5 leads with 76.8 across 25 models. ProgramBench (Vals AI run) — rebuild complete programs from binaries and documentation; raw pass rate is the average percent of hidden behavioral tests passed per task.
• Vals AI SkillsBench (Accuracy (%)): gpt-5.5-codex leads with 62.55 across 12 models. Vals AI SkillsBench — evaluates how well models use skill files (procedural instructions plus scripts) to complete office/document tasks; accuracy per task set.
• Vals AI Code Migration (Accuracy (%)): claude-fable-5 leads with 55.06 across 24 models. Vals AI Code Migration — large-scale code migration tasks (framework/language version upgrades) run agentically; accuracy on private migration task set.
• Foresight Bench (Forecast Skill (100 - Brier x 100)): claude-opus-4.8 leads with 91.61 across 13 models. Foresight Bench (Aleatoric AI) — LLM forecasting on real-world open questions; models keep updating predictions as events unfold, scored by mean Brier on resolved questions (reported as 100 − Brier×100).

New Models (1)

• Claude Haiku 4.5 (20251001) — ELO 1956, #161/1462, above DeepSeek V3.2 Exp, below Step3 VL 10B
  • ForecastBench: 64.5 (#64/223)

New Scores From Top-10 Models (11)

• Claude Fable 5 on HalluHard: 0.4327 Turn-1 Hallucination Rate (#3/32)
• Claude Fable 5 on TrackingAI IQ Test: 93.75 IQ Test Score (%) (#2/31)
• Claude Fable 5 on TrackingAI IQ Test (Offline): 81.25 Offline IQ Score (%) (#2/31)
• Claude Mythos Preview on ExploitBench v8-bench: 78.0 Mean Capability (%) (#1/9)
• Claude Opus 4.8 on SQL Capability - Dialect Conversion: 74.0 Ability Score (#12/36)
• Claude Opus 4.8 on SQL Capability - SQL Optimization: 67.1 Ability Score (#4/36)
• Claude Opus 4.8 on SQL Capability - SQL Understanding: 77.1 Ability Score (#20/35)
• Claude Opus 4.8 on SQL Capability Leaderboard: 72.73 Average Ability Score (#12/37)
• Claude Opus 4.8 on Vals AI SWE-bench Verified: 88.6 Resolved (%) (#2/65)
• Claude Opus 4.8 on Vals AI SWE-bench Verified: 88.6 Resolved (%) (#2/65)
• Qwen 3.7 Max on Vals AI SWE-bench Verified: 68.8 Resolved (%) (#48/65)

New #1 Leaders (4)

• VoxelBench (Rating): Claude Fable 5 (Max) (2233.0) beat GPT-5.5 Pro (2035.0) by 198.0.
• Vals AI SWE-bench Verified (Resolved (%)): Claude Fable 5 (95.0) beat GPT-5.5 (82.6) by 12.4.
• TrackingAI IQ Test (Vision) (IQ Test Score (%)): Claude Opus 4 (Thinking) (87.5) beat GPT-5 Pro (82.35) by 5.15.
• ForecastBench (Overall Score (higher is better)): Cassi-2026-05-10 (69.1) beat Gemini (68.4) by 0.7.