5arXiv cs.CL (Computation and Language)·
MissionBench: Zero-shot mission-level evaluation of MLLMs as aerial embodied agents
Researchers introduce MissionBench, a benchmark comprising 120 missions across five simulated 3D aerial environments and four task families, designed to evaluate multimodal LLMs as zero-shot embodied agents. Across 22 open- and closed-source MLLMs, the best model achieves under 35% success versus 84.4% human performance, revealing a large capability gap in long-horizon embodied tasks. The study finds scaling yields gains but that mission-level competence requires coordinating spatial perception, multi-step planning, and adaptive reasoning beyond what current models reliably provide.
Related guides (2)
Related events (8)
ActiveVision benchmark reveals frontier MLLMs score near zero on active visual observation tasks
Researchers introduce ActiveVision, a 17-task benchmark designed to measure whether multimodal LLMs can perform active visual observation — iteratively redirecting attention based on intermediate hypotheses rather than processing a single static image. Frontier models collapse catastrophically: GPT-5.5 at maximum reasoning effort scores only 10.6% and zeros out on 11 of 17 tasks, while Claude Fable 5 scores 3.5%, versus 96.1% average for human participants. The gap persists even when models write and execute their own vision code, because detecting code failures itself requires the active perception the models lack. The results suggest a fundamental architectural gap between current MLLMs and human closed-loop visual cognition.
PhysTool-Bench reveals severe gaps in MLLM physical tool use and embodied planning
Researchers introduce PhysTool-Bench, the first benchmark evaluating multimodal LLMs on physical tool use across 2,510 queries and 2,678 real-world tools spanning manufacturing, electrical work, agriculture, and healthcare. Evaluation of 13 leading MLLMs shows even the best model (Gemini-3.1-Pro) identifies only 58.7% of tools in a scene and completes just 21.0% of queries end-to-end. The results expose a two-level deficit: poor tool perception in realistic scenes and a much larger drop at the planning stage, indicating a lack of functional commonsense for mapping tools to task semantics. This pinpoints a critical bottleneck for embodied AI development.
MLE-bench: Evaluating Machine Learning Agents on Machine Learning Engineering
OpenAI introduces MLE-bench, a benchmark designed to measure AI agent performance on machine learning engineering tasks. The benchmark draws from Kaggle competitions to evaluate agents on realistic ML engineering workflows. Initial results show that current agents, including those powered by o1-preview, achieve competitive performance on a subset of tasks but fall well short of top human competitors. The benchmark is intended to track progress in agentic ML capabilities over time.
MECoBench: Benchmark for Multimodal Agent Collaboration in Embodied Environments
Researchers introduce MECoBench, a benchmark and evaluation platform for assessing multimodal LLM collaboration in visually grounded embodied environments. The benchmark spans diverse real-world tasks, two cooperation structures, and three collaboration modes. Key findings include that collaboration generally improves task completion but depends on balancing gains against coordination complexity, that communication is essential to collaboration benefits, and that collaboration improves robustness under noisy conditions.
Benchmark and study of LLM agents in deliberative multi-agent collaboration under partial observability
Researchers introduce a scalable benchmark for evaluating LLM agents on cooperative joint decision-making tasks where agents must exchange information under partial and asymmetric observations to reach a shared decision. A systematic evaluation of representative LLMs finds that state-of-the-art models still struggle with complex deliberative collaboration, failing in either information alignment or downstream reasoning even with external mathematical tools. Diagnostic analysis also reveals that deliberation can enable reflection and error correction, sometimes outperforming centralized baselines, offering a nuanced picture of multi-agent LLM capabilities.
AARRI-Bench evaluates frontier LLMs and agents on granular research-intern-level tasks
Researchers introduce AARR (Act As a Real Researcher), a new benchmark series targeting whether AI agents can emulate the professionalism, thoroughness, and nuanced judgment of human researchers in granular research scenarios—not just macro-level task execution. The first benchmark, AARRI-Bench, tests frontier models and agentic harnesses, finding that even the best configuration (Mini-SWE-Agent with Claude Opus 4.7) achieves only 68.3% success, frequently missing subtle but critical details obvious to human researchers. The work argues that closing the gap requires deeper modeling of research behavior rather than more complex scaffolding.
TriViewBench: Controlled benchmark reveals fundamental multi-view spatial reasoning failures in MLLMs
Researchers introduce TriViewBench, a synthetic 3D benchmark of 1,923 scenes and 14K+ QA pairs designed to probe multi-view structural reasoning in MLLMs under controlled complexity scaling. Evaluating 18 open- and closed-source models, the study finds a universal capability hierarchy (Local Decision > Object Counting > Global Recovery) with severe performance collapse on Global Recovery tasks (80% relative drop at highest complexity). Chain-of-Thought prompting provides near-zero benefit, suggesting the bottleneck is cross-view spatial representation rather than reasoning strategy. The work identifies two mechanistically distinct failure modes in object counting: occlusion blindness causing undercounting in single-view tasks and cross-view identity confusion causing overcounting in multi-view tasks.
DevicesWorld: New benchmark for cross-device LLM agents across mobile, desktop, and IoT environments
Researchers introduce DevicesWorld, a large-scale benchmark of 6,140 tasks designed to evaluate LLM-based agents operating across heterogeneous device environments including mobile, desktop, and IoT systems. The benchmark addresses a gap in existing evaluations that focus on single-device settings, instead requiring agents to acquire and integrate information across devices to complete end-to-end tasks. Five frontier LLM-agent systems were evaluated, with the best achieving only 12.5% success rate, revealing systematic failure modes including device confusion and premature termination. The work provides an executable, automatically verifiable evaluation framework for cross-device agent research.