5arXiv cs.AI (Artificial Intelligence)·
Desktop-Delta Bench: New benchmark for evaluating GUI state-transition understanding in computer-use agents
Researchers introduce Desktop-Delta Bench (DDB), an offline step-level benchmark with 2,013 human-verified instances designed to test whether computer-use agents can correctly interpret causal GUI transitions produced by actions — a capability distinct from end-task success or single-frame grounding. The benchmark covers ~15 Linux applications across 50 task domains and targets three failure dimensions: state verification, source tracking, and context-aware control. Evaluation of 8 closed and open-source model families reveals consistent gaps, with best temporal-ordering exact-match rates at ~65% and action-family inference notably harder than action localization. DDB fills a diagnostic gap between GUI grounding benchmarks and end-to-end task success metrics.
Related guides (2)
Related events (8)
MacAgentBench: New benchmark for AI agents on real-world macOS desktop tasks
MacAgentBench introduces a 676-task benchmark across 25 macOS applications designed to evaluate computer use agents (CUAs) with framework augmentation and fine-grained multi-checkpoint scoring, addressing gaps in existing binary-evaluation benchmarks. Nearly 60% of tasks involve both GUI and CLI interaction, and the benchmark tests 16 models across three agent frameworks. The best result — Claude Opus 4.6 on the OpenClaw framework — achieves 73.7% Pass@1, with performance gains attributed primarily to the skill library rather than framework design. Fine-grained metrics reveal that models with similar Pass@1 scores can differ substantially in sub-goal completion, highlighting limitations of coarse evaluation.
HiViG: History-aware visually grounded critic improves computer use agents across GUI benchmarks
Researchers introduce HiViG, a test-time framework for Computer Use Agents that addresses two weaknesses in existing critic models: short-sighted decision loops and lack of visual grounding. The system trains a multimodal critic on real GUI trajectories to maintain a compact macro-action history and verify execution coordinates against live screenshots before action execution. Evaluated on web, mobile, and desktop benchmarks, HiViG improves average success rates by 5.8% over the strongest baseline with Qwen3-VL-32B and 9.0% with Gemini-3-Flash, with both history and grounding components shown to be independently necessary.
Argus benchmark evaluates uncertainty quantification methods for computer-use GUI agents across VLMs and datasets
Researchers introduce Argus, a cross-regime benchmark for post-hoc uncertainty quantification (UQ) in single-step GUI grounding agents, covering 27 methods across 4 open-weight VLMs and 4 datasets, plus an 8-method closed-source matrix across 3 frontier vendors. The central finding is 'selective transfer': UQ rankings are stable across datasets for a fixed model but degrade across model classes and observable interfaces, with cross-tier transfer to closed-source vendors averaging only +0.08 Spearman correlation. Hidden-state and density methods prove most stable for open-weight models, while conformal click regions reveal that score-level discrimination alone is insufficient for deployment safety. The benchmark releases per-item records and analysis scripts to support regime-aware UQ selection in GUI agents.
DevicesWorld: New benchmark for cross-device LLM agents across mobile, desktop, and IoT environments
Researchers introduce DevicesWorld, a large-scale benchmark of 6,140 tasks designed to evaluate LLM-based agents operating across heterogeneous device environments including mobile, desktop, and IoT systems. The benchmark addresses a gap in existing evaluations that focus on single-device settings, instead requiring agents to acquire and integrate information across devices to complete end-to-end tasks. Five frontier LLM-agent systems were evaluated, with the best achieving only 12.5% success rate, revealing systematic failure modes including device confusion and premature termination. The work provides an executable, automatically verifiable evaluation framework for cross-device agent research.
DBA-Bench: Production-fidelity benchmark for LLM-based database operations agents
Researchers introduce DBA-Bench, a benchmark designed to evaluate LLM-based database agents under production-realistic conditions using live PostgreSQL environments with active workloads, persistent state, and multi-source observations. The benchmark covers 106 scenarios across seven task domains and evaluates nine baseline groups including two GPT-5.5-backed database agents and a Human DBA reference. Results reveal a stark capability gap: the best automated system achieves only 17.9% Safe Pass rate versus 93.4% for human DBAs, with performance dropping from 19.6% on Easy to 7.6% on Hard scenarios. The work highlights that current LLM agents fall far short of production-grade database operations, particularly for safe end-to-end fault remediation.
OmniaBench: A broad diagnostic benchmark for general AI agents across 354 domains
OmniaBench is a new benchmark for evaluating general AI agents across diverse scenarios, spanning 90 level-1 and 354 level-2 domains derived from app stores, product documents, and web retrieval. The benchmark contains 1,431 tasks with single-turn and multi-turn formats, a ten-dimensional capability taxonomy, and eight atomic difficulty factors for fine-grained analysis. Even frontier models like Claude Sonnet 5 and GPT-5.6-Sol achieve only ~58% Overall Pass@1, revealing persistent weaknesses in planning, constraint maintenance, and adaptive correction. The work addresses a gap in existing agent benchmarks that tend to cover narrow tool ecosystems or interaction formats.
DABStep: Data Agent Benchmark for Multi-step Reasoning
Hugging Face introduces DABStep, a benchmark designed to evaluate data agents on multi-step reasoning tasks. The benchmark targets agentic systems that must perform complex, sequential data operations rather than single-step queries. It aims to fill a gap in evaluation tooling for realistic data analysis workflows involving tool use and chained reasoning.
MM-ToolSandBox: Apple benchmark reveals visual precision as primary bottleneck for tool-calling agents
Apple researchers introduce MM-ToolSandBox, a benchmark and evaluation framework for visually grounded tool-calling agents, covering 500+ tools across 16 domains with 258 human-verified scenarios. Evaluating 12 models from 4B open-weight to frontier proprietary systems, even the best model achieves below 50% success rate. A key finding is a planning-to-precision crossover: smaller models fail at task planning while larger models fail at visual information extraction, with 53% of failures in capable models stemming from incorrect image interpretation despite correct workflows. The framework and benchmark are publicly released.