5arXiv cs.CL (Computation and Language)·
Benchmark reveals uneven scientific visualization literacy across six MLLMs
A new arXiv preprint benchmarks six multimodal LLMs (three closed-source, three open-source) on a standardized 49-item scientific visualization literacy test spanning 18 visualizations, 8 techniques, and 11 task types, comparing results against 485 human participants. Gemini emerges as the strongest model, exceeding the human mean, while open-source models fall below the human baseline. Performance is highly uneven: models handle scientific illustration and spatial tasks well but struggle with texture-based visualizations, flow-direction interpretation, and quantitative estimation. The authors argue SciVis literacy should be treated as a necessary evaluation dimension for multimodal AI systems.
Related guides (3)
Related events (8)
ActiveVision benchmark reveals frontier MLLMs score near zero on active visual observation tasks
Researchers introduce ActiveVision, a 17-task benchmark designed to measure whether multimodal LLMs can perform active visual observation — iteratively redirecting attention based on intermediate hypotheses rather than processing a single static image. Frontier models collapse catastrophically: GPT-5.5 at maximum reasoning effort scores only 10.6% and zeros out on 11 of 17 tasks, while Claude Fable 5 scores 3.5%, versus 96.1% average for human participants. The gap persists even when models write and execute their own vision code, because detecting code failures itself requires the active perception the models lack. The results suggest a fundamental architectural gap between current MLLMs and human closed-loop visual cognition.
PhysTool-Bench reveals severe gaps in MLLM physical tool use and embodied planning
Researchers introduce PhysTool-Bench, the first benchmark evaluating multimodal LLMs on physical tool use across 2,510 queries and 2,678 real-world tools spanning manufacturing, electrical work, agriculture, and healthcare. Evaluation of 13 leading MLLMs shows even the best model (Gemini-3.1-Pro) identifies only 58.7% of tools in a scene and completes just 21.0% of queries end-to-end. The results expose a two-level deficit: poor tool perception in realistic scenes and a much larger drop at the planning stage, indicating a lack of functional commonsense for mapping tools to task semantics. This pinpoints a critical bottleneck for embodied AI development.
TriViewBench: Controlled benchmark reveals fundamental multi-view spatial reasoning failures in MLLMs
Researchers introduce TriViewBench, a synthetic 3D benchmark of 1,923 scenes and 14K+ QA pairs designed to probe multi-view structural reasoning in MLLMs under controlled complexity scaling. Evaluating 18 open- and closed-source models, the study finds a universal capability hierarchy (Local Decision > Object Counting > Global Recovery) with severe performance collapse on Global Recovery tasks (80% relative drop at highest complexity). Chain-of-Thought prompting provides near-zero benefit, suggesting the bottleneck is cross-view spatial representation rather than reasoning strategy. The work identifies two mechanistically distinct failure modes in object counting: occlusion blindness causing undercounting in single-view tasks and cross-view identity confusion causing overcounting in multi-view tasks.
MM-ToolSandBox: Apple benchmark reveals visual precision as primary bottleneck for tool-calling agents
Apple researchers introduce MM-ToolSandBox, a benchmark and evaluation framework for visually grounded tool-calling agents, covering 500+ tools across 16 domains with 258 human-verified scenarios. Evaluating 12 models from 4B open-weight to frontier proprietary systems, even the best model achieves below 50% success rate. A key finding is a planning-to-precision crossover: smaller models fail at task planning while larger models fail at visual information extraction, with 53% of failures in capable models stemming from incorrect image interpretation despite correct workflows. The framework and benchmark are publicly released.
Benchmarking study finds LLMs fail at counterintuitive probability problems despite strong standard performance
A new arXiv paper evaluates 8 state-of-the-art LLMs on discrete probability problems using two datasets: standard exercises (average accuracy 0.96) and counterintuitive exercises designed to trigger heuristic reasoning (average accuracy 0.59). The authors document token bias causing 20%+ performance drops when canonical problem formulations are disguised, and up to 34% degradation when misleading suggestions are embedded in prompts. The findings argue that current LLMs are not genuine probabilistic reasoners despite their success on advanced math benchmarks.
Paper challenges LLM expert-level claims by measuring variance and error magnitude in code-based data analysis tasks
A new arXiv paper argues that standard LLM benchmarks overstate model capabilities by focusing on average performance on training-data-adjacent tasks while ignoring response variance and error magnitude. The authors introduce a novel benchmark requiring frontier LLMs to write code for data analysis tasks, comparing results against human expert submissions. Human experts outperformed the frontier LLM on average across multiple metrics and showed lower performance variability. The findings challenge the prevailing narrative that LLMs perform at human-expert level on knowledge economy tasks.
MM-IssueLoc: Controlled benchmark for multimodal repository-level issue localization with visual evidence
Researchers introduce MM-IssueLoc, a benchmark of 652 issue-PR instances across 23 programming languages designed to evaluate whether AI systems actually use visual evidence (screenshots, error dialogs, UI states) when localizing bugs in code repositories. The benchmark provides file-level and function-level gold labels, paired text-only and with-image evaluation modes, and VCE-based diagnostics that convert images to structured text. Evaluation of current LLM-based and retrieval-based systems shows poor performance (best agent reaches 38.96 file Acc@5), and high scores on text-dominant SWE benchmarks do not transfer to multimodal localization. The work isolates visual evidence as an explicit evaluation variable, exposing a gap in existing software engineering agent benchmarks.
LLMs evaluated for automated grading of Linux/bash exams using four-level cognitive taxonomy
A new arXiv paper evaluates GPT, Claude Opus, Gemini, and GLM on automated grading of 1,200 real student Linux/bash command responses, benchmarked against three expert instructors. Using a four-level cognitive taxonomy, Gemini 3.0 Pro with rubric-guided prompting achieved the highest human-AI agreement (ICC=0.888, MAE=0.10). Key findings: rubric quality mattered more than model choice, and grading accuracy declined consistently at higher cognitive complexity levels. The study proposes a taxonomy-based framework for deciding which exam questions are suitable for AI-assisted grading.