6arXiv cs.CL (Computation and Language)·
DBA-Bench: Production-fidelity benchmark for LLM-based database operations agents
Researchers introduce DBA-Bench, a benchmark designed to evaluate LLM-based database agents under production-realistic conditions using live PostgreSQL environments with active workloads, persistent state, and multi-source observations. The benchmark covers 106 scenarios across seven task domains and evaluates nine baseline groups including two GPT-5.5-backed database agents and a Human DBA reference. Results reveal a stark capability gap: the best automated system achieves only 17.9% Safe Pass rate versus 93.4% for human DBAs, with performance dropping from 19.6% on Easy to 7.6% on Hard scenarios. The work highlights that current LLM agents fall far short of production-grade database operations, particularly for safe end-to-end fault remediation.
Related guides (4)
Related events (8)
PlanBench-XL: Benchmark for LLM Agent Planning in Large-Scale Tool Ecosystems
Researchers introduce PlanBench-XL, an interactive benchmark of 327 retail tasks spanning 1,665 tools designed to evaluate LLM agents on long-horizon planning under retrieval-limited tool visibility. The benchmark includes a blocking mechanism simulating real-world disruptions such as missing or failing tools, forcing agents to detect and recover from broken execution paths. Experiments on ten leading LLMs reveal severe performance degradation: GPT-5.4 drops from 51.90% accuracy in unblocked settings to 11.36% under the most severe blocking condition, highlighting fragility in adaptive planning for large, imperfect tool environments.
ToolBench-X benchmarks LLM agents under tool-environment unreliability
A new arXiv preprint introduces ToolBench-X, a benchmark for evaluating LLM agents under five structured hazard types including Specification Drift, Invocation Error, Execution Failure, Output Drift, and Cross-source Conflict. Each injected hazard remains solvable via recovery paths such as retrying, fallback, or cross-checking, enabling measurement of agent resilience rather than just function-call accuracy. Experiments reveal a substantial reliability gap: agents that perform well in clean environments frequently fail under recoverable hazards, with failures driven by poor hazard diagnosis rather than insufficient tool-use volume or inference budget. The findings argue for shifting tool-use evaluation toward task completion under realistic, unreliable conditions.
ORCA-bench evaluates LLM agents on production oncall root cause analysis tasks
ORCA-bench is a new benchmark that tests general-purpose coding agents on oncall root cause analysis (RCA) in a production-fidelity environment, using a live OpenTelemetry-instrumented microservice system with 1,079 tasks spanning metrics, logs, traces, and source code. Across five frontier agents, the best RCA accuracy is 25.3% on medium-difficulty tasks and 10.0% on hard tasks, with even Claude Fable 5 failing to close the gap significantly. The benchmark is notable for its expert-curated ground truth (Cohen's κ_w=0.90 human-LLM agreement) and its argument that reported performance is a lower bound on real-world difficulty, since production systems are far larger and more dynamic than the testbed.
EnterpriseClawBench: A benchmark for enterprise agents derived from real workplace sessions
Researchers introduce EnterpriseClawBench, an enterprise agent benchmark constructed from proprietary real-world workplace sessions, yielding 852 reproducible tasks with fixtures, prompts, role classes, skill subclasses, and semantic rubrics. Because the sessions contain internal enterprise content, the benchmark data is not publicly released, but the construction and evaluation protocol is the reusable contribution. The best evaluated configuration (Codex with GPT-5.5) achieves only 0.663, indicating substantial headroom. The paper argues enterprise agent evaluation must report harness-model combinations, artifact delivery, visual quality, cost, runtime, and skill-transfer behavior rather than collapsing to a single score.
T1-Bench: Multi-scenario agent benchmark across 25 real-world domains
T1-Bench is a new benchmark for evaluating agentic LLM systems in realistic customer-facing, multi-domain environments, covering 25 domains of varying difficulty with interleaved multi-turn scenarios. The authors evaluate 12 proprietary and open-weight models and combine automatic evaluation with human judgments. The benchmark targets gaps in existing agent evals around task complexity, domain diversity, and compositional reasoning across multi-step interactions.
Benchmark Agent: Autonomous system for end-to-end benchmark construction
Researchers introduce Benchmark Agent, a fully autonomous agentic system that orchestrates the complete benchmark construction pipeline — from query analysis and subtask design to data annotation and quality control. The system was used to produce 15 benchmarks spanning text understanding, multimodal understanding, and domain-specific reasoning, with evaluation via human judges, LLM-as-a-judge, and consistency checks. The work addresses two persistent problems in the field: the labor intensity of benchmark creation and rapid performance saturation after release. Code and a demo will be publicly released.
AARRI-Bench evaluates frontier LLMs and agents on granular research-intern-level tasks
Researchers introduce AARR (Act As a Real Researcher), a new benchmark series targeting whether AI agents can emulate the professionalism, thoroughness, and nuanced judgment of human researchers in granular research scenarios—not just macro-level task execution. The first benchmark, AARRI-Bench, tests frontier models and agentic harnesses, finding that even the best configuration (Mini-SWE-Agent with Claude Opus 4.7) achieves only 68.3% success, frequently missing subtle but critical details obvious to human researchers. The work argues that closing the gap requires deeper modeling of research behavior rather than more complex scaffolding.
DataGovBench: Benchmarking LLMs on Real-World Governmental Data Analysis Tasks
Researchers introduce DataGovBench, a benchmark derived from governmental open data designed to evaluate LLMs on practical data analysis tasks beyond simple fact retrieval from small tables. The benchmark covers two tasks: Table QA (complex decomposable questions with textual or visualization outputs) and Table Insight (exploratory data analysis for expert-level findings). Experiments with state-of-the-art LLMs, with and without agentic frameworks, reveal significant performance gaps, suggesting current systems fall well short of real-world data analytics demands.