5arXiv cs.CL (Computation and Language)·
WorkSurface-Bench: New benchmark for enterprise agent knowledge surface routing across documents, tables, and graphs
WorkSurface-Bench is a new benchmark of 1,151 tasks evaluating whether enterprise agents correctly select among heterogeneous knowledge sources (documents, tables, dependency graphs) before answering—a capability the authors term 'surface routing.' Evaluating four model backbones across six agent settings, the benchmark reveals a critical gap: agents achieve near-perfect route selection (98.7–99.8% Route F1) under gold-constrained tool access but only 56.1–75.3% answer accuracy, demonstrating that correct surface selection is necessary but insufficient for task completion. The dataset, pipeline, scoring code, and agent harness are released publicly.
Related guides (3)
Related events (8)
EnterpriseClawBench: A benchmark for enterprise agents derived from real workplace sessions
Researchers introduce EnterpriseClawBench, an enterprise agent benchmark constructed from proprietary real-world workplace sessions, yielding 852 reproducible tasks with fixtures, prompts, role classes, skill subclasses, and semantic rubrics. Because the sessions contain internal enterprise content, the benchmark data is not publicly released, but the construction and evaluation protocol is the reusable contribution. The best evaluated configuration (Codex with GPT-5.5) achieves only 0.663, indicating substantial headroom. The paper argues enterprise agent evaluation must report harness-model combinations, artifact delivery, visual quality, cost, runtime, and skill-transfer behavior rather than collapsing to a single score.
Benchmark Agent: Autonomous system for end-to-end benchmark construction
Researchers introduce Benchmark Agent, a fully autonomous agentic system that orchestrates the complete benchmark construction pipeline — from query analysis and subtask design to data annotation and quality control. The system was used to produce 15 benchmarks spanning text understanding, multimodal understanding, and domain-specific reasoning, with evaluation via human judges, LLM-as-a-judge, and consistency checks. The work addresses two persistent problems in the field: the labor intensity of benchmark creation and rapid performance saturation after release. Code and a demo will be publicly released.
OmniaBench: A broad diagnostic benchmark for general AI agents across 354 domains
OmniaBench is a new benchmark for evaluating general AI agents across diverse scenarios, spanning 90 level-1 and 354 level-2 domains derived from app stores, product documents, and web retrieval. The benchmark contains 1,431 tasks with single-turn and multi-turn formats, a ten-dimensional capability taxonomy, and eight atomic difficulty factors for fine-grained analysis. Even frontier models like Claude Sonnet 5 and GPT-5.6-Sol achieve only ~58% Overall Pass@1, revealing persistent weaknesses in planning, constraint maintenance, and adaptive correction. The work addresses a gap in existing agent benchmarks that tend to cover narrow tool ecosystems or interaction formats.
T1-Bench: Multi-scenario agent benchmark across 25 real-world domains
T1-Bench is a new benchmark for evaluating agentic LLM systems in realistic customer-facing, multi-domain environments, covering 25 domains of varying difficulty with interleaved multi-turn scenarios. The authors evaluate 12 proprietary and open-weight models and combine automatic evaluation with human judgments. The benchmark targets gaps in existing agent evals around task complexity, domain diversity, and compositional reasoning across multi-step interactions.
MacAgentBench: New benchmark for AI agents on real-world macOS desktop tasks
MacAgentBench introduces a 676-task benchmark across 25 macOS applications designed to evaluate computer use agents (CUAs) with framework augmentation and fine-grained multi-checkpoint scoring, addressing gaps in existing binary-evaluation benchmarks. Nearly 60% of tasks involve both GUI and CLI interaction, and the benchmark tests 16 models across three agent frameworks. The best result — Claude Opus 4.6 on the OpenClaw framework — achieves 73.7% Pass@1, with performance gains attributed primarily to the skill library rather than framework design. Fine-grained metrics reveal that models with similar Pass@1 scores can differ substantially in sub-goal completion, highlighting limitations of coarse evaluation.
BusinessCaseBench: a new benchmark measuring frontier AI on business school analytical reasoning
Researchers introduce BusinessCaseBench, a benchmark of hundreds of questions drawn from business school case studies across eighteen disciplines, each graded against expert instructor rubrics. The benchmark targets analytical knowledge work — synthesis, judgment under uncertainty, strategic reasoning, trade-off analysis — that existing benchmarks largely miss. Frontier AI models already score highly on the benchmark, and capability within one model family improved substantially over two years. The authors argue this has direct implications for business education and entry-level professional roles historically anchored by these skills.
NatureBench: Coding agents surpass published SOTA on only 17.8% of real scientific tasks from Nature-family papers
NatureBench introduces a 90-task benchmark derived from peer-reviewed Nature-family publications to evaluate whether AI coding agents can advance beyond reproduction toward genuine scientific discovery. Built on NatureGym, an automated pipeline that creates containerized per-task environments, the benchmark addresses environment fragmentation that has undermined prior agent-on-research evaluations. Evaluating ten frontier agent configurations under a web-search-disabled protocol, the strongest model exceeds published SOTA on only 17.8% of tasks, with failures driven primarily by wrong method choice and insufficient compute rather than task misunderstanding. Agents succeed mainly through methodological translation—recasting scientific problems as supervised prediction—rather than genuine scientific invention.
DeepSWE, ProgramBench, and ITBench-AA emerge as harder successors to SWE-bench for agent evaluation
Three new benchmarks — DeepSWE (by Datacurve), ProgramBench (Meta/Stanford/Harvard), and ITBench-AA (IBM/Artificial Analysis) — are positioned as more rigorous replacements for the SWE-bench family, which models have largely saturated. DeepSWE tests feature implementation using private codebases and human-written problems; ProgramBench evaluates agents' ability to recreate functional programs from scratch; ITBench-AA measures root-cause diagnosis in real-world IT incident scenarios. Current top performers include GPT-5.5 (70% on DeepSWE), Claude Opus 4.7 (46.7% on ITBench-AA), and Claude Opus 4.7 (3% on ProgramBench at the 95% pass threshold), illustrating that even frontier models have substantial headroom.