5arXiv cs.AI (Artificial Intelligence)·
MM-IssueLoc: Controlled benchmark for multimodal repository-level issue localization with visual evidence
Researchers introduce MM-IssueLoc, a benchmark of 652 issue-PR instances across 23 programming languages designed to evaluate whether AI systems actually use visual evidence (screenshots, error dialogs, UI states) when localizing bugs in code repositories. The benchmark provides file-level and function-level gold labels, paired text-only and with-image evaluation modes, and VCE-based diagnostics that convert images to structured text. Evaluation of current LLM-based and retrieval-based systems shows poor performance (best agent reaches 38.96 file Acc@5), and high scores on text-dominant SWE benchmarks do not transfer to multimodal localization. The work isolates visual evidence as an explicit evaluation variable, exposing a gap in existing software engineering agent benchmarks.
Related guides (3)
Related events (8)
Benchmark reveals uneven scientific visualization literacy across six MLLMs
A new arXiv preprint benchmarks six multimodal LLMs (three closed-source, three open-source) on a standardized 49-item scientific visualization literacy test spanning 18 visualizations, 8 techniques, and 11 task types, comparing results against 485 human participants. Gemini emerges as the strongest model, exceeding the human mean, while open-source models fall below the human baseline. Performance is highly uneven: models handle scientific illustration and spatial tasks well but struggle with texture-based visualizations, flow-direction interpretation, and quantitative estimation. The authors argue SciVis literacy should be treated as a necessary evaluation dimension for multimodal AI systems.
M³Exam: Benchmark for Multimodal Memory in Realistic User-Agent Interactions
Researchers introduce M³Exam, a query-centric multimodal conversational memory benchmark designed to evaluate language agents on realistic user-agent interactions, including cross-modal grounding and implicit information inference. Existing benchmarks are critiqued for assuming sparse visuals and human-human interaction formats. The paper also proposes M³Proctor, a companion memory method that detects query modality bias and retrieves raw visual sources on demand, achieving 13% accuracy improvement while reducing index-construction time and retrieved tokens by over 70%.
Moment-Video: Benchmark Diagnosing Temporal Fidelity of Video MLLMs on Momentary Visual Events
Moment-Video is a new benchmark of 1,000 human-verified video-QA pairs designed to evaluate how well video multimodal large language models (MLLMs) handle brief, localized visual events that may span only a few frames. The benchmark covers 7 domains and 25 subcategories across four task types: Temporal Occurrence, Temporal Counting, Action Description, and Temporal Reasoning. Evaluation of 33 proprietary and open-source models reveals severe deficiencies: the best model (Seed-2.0-Pro) achieves only 39.6% accuracy, while most open-source models score below 25%. Diagnostic analyses show that denser frame sampling helps but does not resolve the bottleneck, pointing to fundamental limitations in how current video MLLMs represent and preserve transient visual evidence.
Benchmark for view-level visual evidence identification in multi-view MLLMs for autonomous driving
A new arXiv preprint introduces a multi-view visual question answering benchmark targeting evidence-source identification in autonomous driving scenarios. Given six synchronized NuScenes camera views and a question, models must identify which camera view supports the answer — not just produce a correct answer. The 122-pair benchmark spans causality, counterfactual reasoning, and intent prediction, and exposes grounding failures that answer-only evaluation misses. The work addresses a meaningful gap between answer accuracy and correct visual grounding in safety-critical multimodal systems.
ReproRepo: Scalable LLM agent framework for reproducibility auditing using GitHub issues
ReproRepo is a new framework for evaluating LLM agents on reproducibility auditing of ML research, using naturally occurring GitHub issues as supervision signals rather than costly manual curation. The framework is instantiated on 1,149 recent ML papers from major conferences and benchmarks four frontier model-agent configurations. The best-performing agent (Codex with GPT-5.5) surfaces at least one semantically related human-reported reproduction blocker for ~90% of papers, though exact localization of issues remains a weakness. The work provides a reusable, scalable evaluation harness for this underexplored agentic task.
MM-ToolSandBox: Apple benchmark reveals visual precision as primary bottleneck for tool-calling agents
Apple researchers introduce MM-ToolSandBox, a benchmark and evaluation framework for visually grounded tool-calling agents, covering 500+ tools across 16 domains with 258 human-verified scenarios. Evaluating 12 models from 4B open-weight to frontier proprietary systems, even the best model achieves below 50% success rate. A key finding is a planning-to-precision crossover: smaller models fail at task planning while larger models fail at visual information extraction, with 53% of failures in capable models stemming from incorrect image interpretation despite correct workflows. The framework and benchmark are publicly released.
Pluralis v0.1: Multicultural, multimodal, multilingual benchmark for AI safety across Asia-Pacific
Researchers introduce Pluralis v0.1, a 6,448-prompt multimodal benchmark spanning six Asia-Pacific countries and eight languages, designed to evaluate Vision-Language Models on culturally localized safety hazards rather than Western-adapted defaults. The benchmark introduces a novel evaluation paradigm where text and image are individually innocuous but synergistically trigger cultural or legal violations. The authors also present Judge-Pluralis, an LLM-as-a-Judge ensemble for scoring, and document recurring locale-specific failure modes in current VLMs including image misidentification and missed item-context-locale interactions.
BabelJudge: Benchmark for measuring LLM-as-a-judge reliability across languages and agent trajectories
BabelJudge is a new open-source benchmark and audit framework that systematically measures four failure modes in LLM-as-a-judge systems: position bias, verbosity bias, order inconsistency, and cross-lingual degradation. The framework uses a 'gold-labelling by degradation' technique to generate labeled evaluation pairs without human annotation. Evaluation of Qwen2.5-7B-Instruct-4bit across English, Hindi, Arabic, and Swahili reveals severe cross-lingual reliability drops, with Swahili order consistency collapsing to near-random (0.480). The framework is extended to agentic evaluation with nine trajectory-level perturbations and three new metrics, released as a Python package supporting 11 judge backends.