6arXiv cs.CL (Computation and Language)·
ORCA-bench evaluates LLM agents on production oncall root cause analysis tasks
ORCA-bench is a new benchmark that tests general-purpose coding agents on oncall root cause analysis (RCA) in a production-fidelity environment, using a live OpenTelemetry-instrumented microservice system with 1,079 tasks spanning metrics, logs, traces, and source code. Across five frontier agents, the best RCA accuracy is 25.3% on medium-difficulty tasks and 10.0% on hard tasks, with even Claude Fable 5 failing to close the gap significantly. The benchmark is notable for its expert-curated ground truth (Cohen's κ_w=0.90 human-LLM agreement) and its argument that reported performance is a lower bound on real-world difficulty, since production systems are far larger and more dynamic than the testbed.
Related guides (2)
Related events (8)
RealClawBench: Live benchmark framework built from real developer-agent sessions
RealClawBench is a new benchmark framework that converts real OpenClaw developer-agent sessions into reproducible, automatically scored evaluation tasks. It addresses realism gaps in existing agent benchmarks through reconstructed execution environments and deterministic verifiable scorers, releasing 281 executable tasks sampled to preserve the source session distribution. Evaluation of 14 contemporary models shows the best system solves only 65.8% of tasks, indicating substantial headroom on realistic developer-agent workloads.
DBA-Bench: Production-fidelity benchmark for LLM-based database operations agents
Researchers introduce DBA-Bench, a benchmark designed to evaluate LLM-based database agents under production-realistic conditions using live PostgreSQL environments with active workloads, persistent state, and multi-source observations. The benchmark covers 106 scenarios across seven task domains and evaluates nine baseline groups including two GPT-5.5-backed database agents and a Human DBA reference. Results reveal a stark capability gap: the best automated system achieves only 17.9% Safe Pass rate versus 93.4% for human DBAs, with performance dropping from 19.6% on Easy to 7.6% on Hard scenarios. The work highlights that current LLM agents fall far short of production-grade database operations, particularly for safe end-to-end fault remediation.
AARRI-Bench evaluates frontier LLMs and agents on granular research-intern-level tasks
Researchers introduce AARR (Act As a Real Researcher), a new benchmark series targeting whether AI agents can emulate the professionalism, thoroughness, and nuanced judgment of human researchers in granular research scenarios—not just macro-level task execution. The first benchmark, AARRI-Bench, tests frontier models and agentic harnesses, finding that even the best configuration (Mini-SWE-Agent with Claude Opus 4.7) achieves only 68.3% success, frequently missing subtle but critical details obvious to human researchers. The work argues that closing the gap requires deeper modeling of research behavior rather than more complex scaffolding.
EnterpriseClawBench: A benchmark for enterprise agents derived from real workplace sessions
Researchers introduce EnterpriseClawBench, an enterprise agent benchmark constructed from proprietary real-world workplace sessions, yielding 852 reproducible tasks with fixtures, prompts, role classes, skill subclasses, and semantic rubrics. Because the sessions contain internal enterprise content, the benchmark data is not publicly released, but the construction and evaluation protocol is the reusable contribution. The best evaluated configuration (Codex with GPT-5.5) achieves only 0.663, indicating substantial headroom. The paper argues enterprise agent evaluation must report harness-model combinations, artifact delivery, visual quality, cost, runtime, and skill-transfer behavior rather than collapsing to a single score.
RuBench: Repository-level agentic coding benchmark with native Russian task specifications
RuBench 1.0 is a new benchmark of 25 repository-level agentic coding tasks drawn from real fix commits in five live open-source projects, where task specifications are written natively in Russian in the style of customer requests rather than translated from English. The benchmark evaluates deployed product configurations including Claude Code with Opus 4.8, Sonnet 5, and Haiku 4.5, and Codex CLI with GPT-5.5, with the best configuration resolving 78.7% of tasks. A notable finding is that auditing trajectories of a fifth configuration (Claude Code + Fable 5) revealed that on 20% of tasks an official safeguard fallback silently re-routed the model to Opus 4.8, providing direct evidence that the deployed product rather than the underlying model is the actual unit of measurement in agentic evaluations.
OmniaBench: A broad diagnostic benchmark for general AI agents across 354 domains
OmniaBench is a new benchmark for evaluating general AI agents across diverse scenarios, spanning 90 level-1 and 354 level-2 domains derived from app stores, product documents, and web retrieval. The benchmark contains 1,431 tasks with single-turn and multi-turn formats, a ten-dimensional capability taxonomy, and eight atomic difficulty factors for fine-grained analysis. Even frontier models like Claude Sonnet 5 and GPT-5.6-Sol achieve only ~58% Overall Pass@1, revealing persistent weaknesses in planning, constraint maintenance, and adaptive correction. The work addresses a gap in existing agent benchmarks that tend to cover narrow tool ecosystems or interaction formats.
Community benchmark of Claude Opus 5 on SlopCodeBench
A GitHub-hosted document from the humanlayer/advanced-context-engineering-for-coding-agents project benchmarks Claude Opus 5 on a dataset called SlopCodeBench, apparently evaluating coding agent performance. The post gained moderate traction on Hacker News with 177 points and 41 comments. Details of methodology and results are not available from the snippet, but the existence of a named benchmark and a new model version (Opus 5) are notable signals.
ToolBench-X benchmarks LLM agents under tool-environment unreliability
A new arXiv preprint introduces ToolBench-X, a benchmark for evaluating LLM agents under five structured hazard types including Specification Drift, Invocation Error, Execution Failure, Output Drift, and Cross-source Conflict. Each injected hazard remains solvable via recovery paths such as retrying, fallback, or cross-checking, enabling measurement of agent resilience rather than just function-call accuracy. Experiments reveal a substantial reliability gap: agents that perform well in clean environments frequently fail under recoverable hazards, with failures driven by poor hazard diagnosis rather than insufficient tool-use volume or inference budget. The findings argue for shifting tool-use evaluation toward task completion under realistic, unreliable conditions.