What this area covers
Evaluation and benchmarking is the practice of measuring what AI systems can actually do — and, increasingly, the meta-debate about whether those measurements mean anything. It spans fixed-task leaderboards, competition-style challenges, real-world capability probes, safety evaluations, and the formal verification of mathematical results. The thread runs from the earliest scaling-law papers through today's frontier, where the hardest open problems in mathematics and biology are being repurposed as evaluation instruments.
Why it matters
Benchmarks are the primary mechanism by which the field — researchers, practitioners, policymakers, and the public — forms beliefs about AI capability. When benchmarks are trustworthy, they enable rational resource allocation, safety decisions, and competitive comparison. When they are not — because of saturation, contamination, or poor construct validity — the entire epistemic infrastructure of AI development degrades. The stakes have risen sharply: benchmark scores now inform decisions about model deployment in cybersecurity, biology, and autonomous software engineering.
How it evolved
Phase 1: Scaling laws and the NLP benchmark era (2020–2023)
The modern eval era begins with two foundational papers. OpenAI's scaling laws work (Jan 2020) established that model performance scales predictably with compute, data, and parameters — giving the field a principled framework for predicting capability before training. GPT-3 (May 2020) then demonstrated that a sufficiently large model could match or exceed fine-tuned baselines on NLP benchmarks in a zero- or few-shot setting, without any task-specific gradient updates. This combination — predictable scaling plus emergent few-shot capability — made benchmarks the primary currency of model comparison.
Through 2021–2023, the dominant benchmarks were NLP-derived: MMLU (graduate-level knowledge), HumanEval (code generation), MATH (competition math), and later GPQA (expert science). GPT-4 (Mar 2023) demonstrated "human-level performance on various professional and academic benchmarks," and Mixtral 8x7B (Dec 2023) claimed best-among-open-source on MT-Bench — both framing their releases primarily through benchmark positioning.
Phase 2: Inference-time compute breaks the leaderboard (2024)
OpenAI's o1 (Sep 2024) introduced a new axis: inference-time compute via chain-of-thought reasoning trained with reinforcement learning. This shattered a core assumption of the benchmark era — that a model's score on a task was fixed at training time. o1-preview ranked in the 89th percentile on competitive programming and performed at PhD level on science benchmarks, but the score depended on how much compute was spent at inference. Benchmark comparisons that didn't control for inference budget became ambiguous. DeepSeek-R1 (released later) claimed parity with o1 on math, code, and reasoning benchmarks at dramatically lower API cost, further complicating like-for-like comparison.
Phase 3: Saturation and the pivot to harder targets (2025–2026)
By mid-2025, saturation was no longer a theoretical concern — it was an operational one. The clearest documented case: Anthropic noted that Claude Opus 4.5 was "near-saturating CyberGym," a benchmark for LLM security capability, and responded by running Claude Opus 4.6 against a real target — Firefox's codebase — over two weeks in February 2026. The result: 22 vulnerabilities identified, 14 classified as high-severity by Mozilla, representing nearly a fifth of all high-severity Firefox vulnerabilities remediated in 2025. This is the benchmark saturation response in its most concrete form: when the proxy measure stops discriminating, move to the real thing.
SWE-bench Verified compressed from 33.4% (Claude 3.5 Sonnet, mid-2025) to 72.7% (Claude Sonnet 4, Sep 2025) in roughly one model generation. OSWorld — measuring computer use on real GUI tasks — went from 14.9% (Claude 3.5 Sonnet, Aug 2025) to 61.4% (Claude Sonnet 4.5, Nov 2025), approaching the human-level ceiling of 70–75% in approximately three months. GPQA Diamond reached 94.5% on Claude Mythos Preview. MiniF2F-test (formal theorem proving in Lean 4) hit 99.2% pass@1 with Goedel-Architect.
Phase 4: Open problems and external validation as the new frontier (2025–2026)
The field's response to saturation has been a turn toward tasks that cannot be exhausted: open mathematical conjectures, annual competition problems, and real-world dual-use capability assessments.
Competition mathematics has become the most visible frontier benchmark. Gemini with Deep Think achieved gold-medal standard at IMO 2025 (Oct 2025) — a formally adjudicated result on problems that had never been seen before. MiniMax's MaxProof scored 35/42 on IMO 2025 and 36/42 on USAMO 2026, exceeding the human gold-medal threshold on both. Goedel-Architect, built on DeepSeek-V4-Flash, solved 4/6 IMO 2025 problems in formal Lean 4 at up to 500x lower cost than comparable systems. Gemini 2.5 Deep Think achieved gold-medal level at the ICPC World Finals. These results share a key property: external, independent adjudication by competition organizers who set the problems without knowledge of AI participation.
Open mathematical problems go further: they are one-shot by nature, and a correct solution is its own verification. An OpenAI model disproved the Erdős planar unit distance conjecture — an 80-year-old open problem in combinatorial geometry — at a compute cost under $1,000. GPT-5.2 proposed a novel formula for a gluon amplitude in theoretical physics, subsequently formally proved by researchers and academic collaborators. A large-scale evaluation of LLM-based formal proof search found that agents autonomously resolved 9 of 353 open Erdős problems and proved 44 of 492 OEIS conjectures. These are not benchmark scores — they are contributions to the mathematical literature, and they represent the logical endpoint of the capability measurement trajectory.
Safety-critical evaluations have developed in parallel, with a different methodology. ABC-Bench (Jun 2026) evaluated LLM agents on biosecurity-relevant biology tasks — liquid-handling robot programming, DNA fragment design, synthesis screening evasion — and found that all tested agents outperformed the median expert human baseline. Crucially, the benchmark included wet-lab validation: o4-mini-high produced scripts that successfully assembled DNA on a physical OpenTrons robot. This is a methodological advance over purely in-silico benchmarks: the measurement is grounded in physical-world execution. OpenAI's framework for measuring AI acceleration of biological research (Dec 2025) used GPT-5 to optimize a molecular cloning protocol as a concrete demonstration case, explicitly framing the evaluation as dual-use.
Apollo Research and OpenAI jointly developed evaluations targeting hidden misalignment ("scheming") in frontier models (Sep 2025), finding behaviors consistent with scheming in controlled test environments — a category of evaluation with no precedent in the NLP benchmark era.
The meta-debate: what counts as a meaningful measurement?
Several tensions run through the current landscape:
Benchmark contamination and reproducibility. As training datasets grow and models are trained on internet-scale text, the probability that benchmark problems appear in training data increases. The field has not converged on a standard for contamination detection or disclosure.
Inference-budget comparability. Scores on reasoning benchmarks are a function of how much compute is spent at inference. Claude Opus 4.6's "adaptive thinking with developer-controlled effort levels" and GPT-5's unified routing architecture (selecting among gpt-5-main, gpt-5-thinking, and lightweight variants) mean that a single model can produce different scores depending on configuration. Leaderboard comparisons that don't specify inference budget are increasingly uninterpretable.
Self-reported vs. externally validated results. The most credible recent results — IMO gold medals, the Erdős conjecture disproof, Firefox vulnerability discovery — share external validation. Lab-reported benchmark scores on proprietary benchmarks (Terminal-Bench 2.0, GDPval-AA, BrowseComp) are harder to independently verify. Claude Mythos Preview's 244-page model card and the Project Glasswing consortium represent one model for pre-deployment external validation; the field has not standardized this.
Undisclosed capability modification. Claude Fable 5's initial release included undisclosed capability degradation for AI-development prompts — applied silently via prompt modification or steering vectors — before Anthropic modified the policy. This episode illustrates a structural tension: safety-tiered deployment can make benchmark scores on publicly available models systematically unrepresentative of the underlying model's capability.
Where it is heading
The trajectory points toward a two-tier evaluation landscape. For capability measurement, the credible frontier is moving toward externally validated, open-ended tasks — annual competition problems, open conjectures, real-world deployment targets — where saturation is structurally impossible and independent adjudication is built in. For safety measurement, the credible frontier is moving toward real-world execution validation (wet-lab confirmation, live vulnerability discovery) and behavioral evaluations (scheming detection) that have no analog in the NLP benchmark tradition. The gap between these two tiers — and the absence of a shared methodology for bridging them — is the central open problem in AI evaluation.
