6OpenAI Blog·
OpenAI analysis reveals reliability issues in SWE-Bench Pro coding benchmark
OpenAI published an analysis identifying problems with SWE-Bench Pro, a widely-used benchmark for evaluating AI coding capabilities. The post raises concerns about the benchmark's reliability and accuracy as a signal for model performance. This matters because SWE-Bench is a primary reference point for comparing frontier coding models, and benchmark integrity directly affects how labs and practitioners interpret capability claims.
Related guides (3)
Related events (8)
OpenAI Abandons SWE-bench Verified Over Contamination and Measurement Flaws
OpenAI has announced it will no longer evaluate models on SWE-bench Verified, citing benchmark contamination and flawed test cases that cause it to mismeasure frontier coding capabilities. Their analysis identified both problematic test design and training data leakage as sources of unreliability. OpenAI recommends SWE-bench Pro as a replacement benchmark for evaluating coding progress.
Introducing SWE-bench Verified
OpenAI is releasing SWE-bench Verified, a human-validated subset of the SWE-bench benchmark designed to more reliably evaluate AI models on real-world software engineering tasks. The original SWE-bench contained issues that were ambiguous or unsolvable, leading to unreliable scores; the Verified subset addresses this by having human annotators confirm task solvability and clarity. This provides a cleaner signal for comparing coding agent performance across labs.
Audit finds GSO, SWE-Perf, and SWE-fficiency benchmarks unreliable for measuring coding agent progress
A new arXiv paper audits three prominent repository-level code-optimization benchmarks (GSO, SWE-Perf, SWE-fficiency) used to rank coding agents, finding significant reliability problems across all three. Reference patches satisfy validity rules in cross-machine replays for only 39/102 GSO tasks and 11/140 SWE-Perf tasks, and leaderboard rankings disagree on 9 of 28 pairwise comparisons depending on scoring rule choice. The authors also find that at least one public submission already matches or beats the reference patch on 85.3% of replay-valid tasks, suggesting aggregate leaderboard scores obscure the true frontier. The study raises substantive concerns about whether these benchmarks are providing reliable signal for claims of coding-agent capability progress.
DeepSWE, ProgramBench, and ITBench-AA emerge as harder successors to SWE-bench for agent evaluation
Three new benchmarks — DeepSWE (by Datacurve), ProgramBench (Meta/Stanford/Harvard), and ITBench-AA (IBM/Artificial Analysis) — are positioned as more rigorous replacements for the SWE-bench family, which models have largely saturated. DeepSWE tests feature implementation using private codebases and human-written problems; ProgramBench evaluates agents' ability to recreate functional programs from scratch; ITBench-AA measures root-cause diagnosis in real-world IT incident scenarios. Current top performers include GPT-5.5 (70% on DeepSWE), Claude Opus 4.7 (46.7% on ITBench-AA), and Claude Opus 4.7 (3% on ProgramBench at the 95% pass threshold), illustrating that even frontier models have substantial headroom.
Automated Benchmark Auditing for AI Agents and Large Language Models (ABA)
The paper introduces Auto Benchmark Audit (ABA), an agentic framework that systematically audits AI benchmark tasks for issues such as ambiguous specifications, environment conflicts, and incorrect ground truths. Applied to 168 benchmarks across nine domains including NeurIPS publications, ABA identifies critical issues in over 25.7% of evaluated tasks. The authors demonstrate that filtering out flawed tasks materially shifts model rankings and improves average performance on SWE-bench Verified and Terminal-Bench 2 by 9.9% and 9.6% respectively, indicating that current benchmark scores are significantly distorted by task quality problems. The agentic tool and annotations are released publicly.
PaperBench: OpenAI Benchmark for Evaluating AI Agents on Research Replication
OpenAI introduces PaperBench, a benchmark designed to evaluate AI agents' ability to replicate state-of-the-art AI research papers end-to-end. The benchmark targets a high-complexity capability: reproducing experimental results from frontier AI research, which requires code generation, experimental design, and scientific reasoning. This positions PaperBench as a tool for tracking progress toward autonomous AI research agents.
OpenAI introduces GeneBench-Pro, a genomics and biology AI benchmark
OpenAI has released GeneBench-Pro, a new benchmark designed to evaluate AI performance on genomics, biology, and scientific research tasks using complex, real-world datasets. The announcement comes from OpenAI's official blog, positioning it as a domain-specific evaluation tool for scientific AI. This extends the evaluation landscape into life sciences, a growing area of AI application.
NatureBench: Coding agents surpass published SOTA on only 17.8% of real scientific tasks from Nature-family papers
NatureBench introduces a 90-task benchmark derived from peer-reviewed Nature-family publications to evaluate whether AI coding agents can advance beyond reproduction toward genuine scientific discovery. Built on NatureGym, an automated pipeline that creates containerized per-task environments, the benchmark addresses environment fragmentation that has undermined prior agent-on-research evaluations. Evaluating ten frontier agent configurations under a web-search-disabled protocol, the strongest model exceeds published SOTA on only 17.8% of tasks, with failures driven primarily by wrong method choice and insufficient compute rather than task misunderstanding. Agents succeed mainly through methodological translation—recasting scientific problems as supervised prediction—rather than genuine scientific invention.