5arXiv cs.CL (Computation and Language)·
MeetingToM benchmark evaluates multimodal LLMs on Theory-of-Mind reasoning in multi-party meetings
Researchers introduce MeetingToM, a benchmark for evaluating multimodal LLMs on Theory-of-Mind (ToM) reasoning in naturalistic multi-party meeting settings. The benchmark is hierarchically structured across subject-level mental state prediction, dyadic addressee understanding, and group-level consensus reasoning, with particular attention to phenomena like pseudo-consensus where apparent agreement masks private dissent. Systematic evaluation of representative MLLMs reveals persistent limitations in integrating non-verbal cues, inferring hidden attitudes, and distinguishing genuine from surface-level consensus. The work addresses gaps in existing multimodal ToM benchmarks that focus primarily on overt, externally verifiable signals.
Related guides (2)
Related events (8)
NCP-ExploreToM benchmark evaluates LLMs' ability to induce belief states through action rather than conversation
Researchers introduce Non-Conversational Planning Theory of Mind (NCP-ToM) and a novel evaluation framework, NCP-ExploreToM, which tests whether LLMs can manipulate belief states in other agents by moving objects or directing characters rather than through dialogue. Six frontier models (including GPT-5, Gemini 2.5 Pro, and Claude 4 series) and human participants were evaluated across 600 task instances; GPT-5 achieved ~80% success and was the only model to outperform humans, though it remained less robust across contexts. All models, like humans, performed better at inducing true belief states than false ones, which the authors flag as a positive alignment signal. The work highlights both emerging agentic social-reasoning capabilities and new manipulation/misinformation risks that passive ToM benchmarks fail to capture.
Benchmark and study of LLM agents in deliberative multi-agent collaboration under partial observability
Researchers introduce a scalable benchmark for evaluating LLM agents on cooperative joint decision-making tasks where agents must exchange information under partial and asymmetric observations to reach a shared decision. A systematic evaluation of representative LLMs finds that state-of-the-art models still struggle with complex deliberative collaboration, failing in either information alignment or downstream reasoning even with external mathematical tools. Diagnostic analysis also reveals that deliberation can enable reflection and error correction, sometimes outperforming centralized baselines, offering a nuanced picture of multi-agent LLM capabilities.
MECoBench: Benchmark for Multimodal Agent Collaboration in Embodied Environments
Researchers introduce MECoBench, a benchmark and evaluation platform for assessing multimodal LLM collaboration in visually grounded embodied environments. The benchmark spans diverse real-world tasks, two cooperation structures, and three collaboration modes. Key findings include that collaboration generally improves task completion but depends on balancing gains against coordination complexity, that communication is essential to collaboration benefits, and that collaboration improves robustness under noisy conditions.
ClinMM-Bench: Multi-turn multimodal benchmark for clinical diagnostic reasoning across 1,089 real-world cases
Researchers introduce ClinMM-Bench, a benchmark of 1,089 real-world clinical cases and 3,760 medical images spanning eight specialties, designed to evaluate multimodal LLMs on multi-turn diagnostic reasoning with progressive information disclosure. Fifteen MLLMs were evaluated using a two-level framework assessing both diagnostic accuracy and reasoning quality. Proprietary models led on accuracy but all models showed limited rates of fully correct diagnoses, and error analysis identified five failure modes including visual hallucination and premature closure. The work addresses a gap in clinical AI evaluation by moving beyond single-turn isolated tasks toward more realistic diagnostic workflows.
TriViewBench: Controlled benchmark reveals fundamental multi-view spatial reasoning failures in MLLMs
Researchers introduce TriViewBench, a synthetic 3D benchmark of 1,923 scenes and 14K+ QA pairs designed to probe multi-view structural reasoning in MLLMs under controlled complexity scaling. Evaluating 18 open- and closed-source models, the study finds a universal capability hierarchy (Local Decision > Object Counting > Global Recovery) with severe performance collapse on Global Recovery tasks (80% relative drop at highest complexity). Chain-of-Thought prompting provides near-zero benefit, suggesting the bottleneck is cross-view spatial representation rather than reasoning strategy. The work identifies two mechanistically distinct failure modes in object counting: occlusion blindness causing undercounting in single-view tasks and cross-view identity confusion causing overcounting in multi-view tasks.
Survey: Human-View Video Understanding with MLLMs — Watch, Remember, Reason Framework
A new arXiv survey paper proposes a unified 'human-view' framework for analyzing multimodal LLM-based video understanding, organized around three functional abilities: watching (perception), remembering (memory), and reasoning. The authors introduce a formulation characterizing video understanding systems by perceptual representations, memory states, reasoning traces, and predictions, then survey methods, datasets, and benchmarks across these dimensions. The work covers challenges including spatio-temporal perception, long-video processing, streaming understanding, and faithful reasoning, with application domains spanning egocentric, sports, medical, and narrative video.
GIM: A Grounded Integration Measure Benchmark for Evaluating Multi-Domain Cognitive Coordination in LLMs
The Grounded Integration Measure (GIM) is a new LLM benchmark of 820 original problems designed to resist benchmark saturation by requiring integration of multiple cognitive operations—constraint satisfaction, state tracking, epistemic vigilance, audience calibration—over broadly accessible knowledge. Unlike knowledge-escalation benchmarks (GPQA, HLE) or pure abstraction benchmarks (ARC-AGI), GIM grounds reasoning in realistic tasks without gating on specialized expertise. The authors calibrate a 2-parameter logistic IRT model over 200k+ prompt-response pairs across 28 models and 47 test configurations, producing the most extensive published study of test-time compute vs. model capability tradeoffs on a fixed benchmark. A key finding is that within-family configuration choices (thinking budget, quantization) matter as much as model selection.
M³Exam: Benchmark for Multimodal Memory in Realistic User-Agent Interactions
Researchers introduce M³Exam, a query-centric multimodal conversational memory benchmark designed to evaluate language agents on realistic user-agent interactions, including cross-modal grounding and implicit information inference. Existing benchmarks are critiqued for assuming sparse visuals and human-human interaction formats. The paper also proposes M³Proctor, a companion memory method that detects query modality bias and retrieves raw visual sources on demand, achieving 13% accuracy improvement while reducing index-construction time and retrieved tokens by over 70%.