6arXiv cs.CL (Computation and Language)·
ClinMM-Bench: Multi-turn multimodal benchmark for clinical diagnostic reasoning across 1,089 real-world cases
Researchers introduce ClinMM-Bench, a benchmark of 1,089 real-world clinical cases and 3,760 medical images spanning eight specialties, designed to evaluate multimodal LLMs on multi-turn diagnostic reasoning with progressive information disclosure. Fifteen MLLMs were evaluated using a two-level framework assessing both diagnostic accuracy and reasoning quality. Proprietary models led on accuracy but all models showed limited rates of fully correct diagnoses, and error analysis identified five failure modes including visual hallucination and premature closure. The work addresses a gap in clinical AI evaluation by moving beyond single-turn isolated tasks toward more realistic diagnostic workflows.
Related guides (2)
Related events (8)
MedRealMM: Real-world multimodal benchmark for Chinese online medical consultation
Researchers introduce MedRealMM, a benchmark of 5,620 real-world multimodal patient-doctor interaction cases drawn from a nationwide Chinese internet hospital, spanning 64 clinical departments. The benchmark uses a Multimodal Clinical Challenge Point (MCCP) framework to identify clinically demanding moments and evaluates 19 general-purpose and medical-specialized LLMs. Key findings show that frontier models still fall below online physician performance, particularly on safety-sensitive error avoidance, and that image information is critical for reliable clinical performance. The dataset will be publicly released on Hugging Face.
ClinHallu benchmark diagnoses stage-wise hallucinations in medical multimodal LLM reasoning
Researchers from Alibaba DAMO Academy introduce ClinHallu, a benchmark of 7,031 validated instances designed to identify where hallucinations originate within medical MLLM reasoning pipelines. Each instance is annotated with a structured reasoning trace decomposed into Visual Recognition, Knowledge Recall, and Reasoning Integration stages, with stage-replacement interventions to measure the causal impact of correcting each stage. The paper also demonstrates that trace-supervised fine-tuning reduces stage-wise hallucinations, offering both diagnostic and mitigation value for clinical AI systems.
MedCase-Structured: A Text-to-FHIR Dataset for Benchmarking Diagnostic Reasoning in Clinically Realistic EHR Settings
The paper introduces a pipeline for converting unstructured clinical text into HL7 FHIR R4 bundles, enabling evaluation of LLMs in realistic electronic health record settings. Applied to the MedCaseReasoning dataset, it produces MedCase-Structured, a synthetic benchmark achieving valid FHIR generation for 82.5% of cases. Key finding: LLMs show consistently lower diagnostic accuracy on structured FHIR inputs compared to plain text, underscoring the gap between standard benchmarks and real-world clinical deployment conditions.
ClinEnv: Interactive Multi-Stage Long-Horizon EHR Benchmark for Clinical Agent Evaluation
ClinEnv is a new interactive benchmark that evaluates LLMs as attending physicians over real inpatient admissions using a Longitudinal Inpatient Simulation paradigm. Each case is decomposed into sequential decision stages where models must query four specialized agents before committing to medications, procedures, and diagnoses. Across seven evaluated models, the best achieves only 0.31 decision F1, with a sharp gap between diagnosis recovery (0.51 F1) and management actions (0.17 F1). The benchmark uniquely measures information-acquisition process quality alongside outcome quality, exposing a gap invisible to static or outcome-only evaluations.
OpenMedReason: Large-scale multimodal medical reasoning corpus with 450K instances for clinical VLM training
Researchers introduce OpenMedReason, a 450K-instance open multimodal medical reasoning corpus with reasoning traces derived from human-authored biomedical literature rather than synthetic chains of thought. The dataset covers diverse medical imaging modalities and is paired with OpenMedReason-Bench, a held-out benchmark evaluating LVLMs on perception, medical knowledge, and rationale axes. Training with OpenMedReason yields a 20% average VQA accuracy improvement over base models and achieves performance within 4.2% of leading comparable-scale medical VLMs. Both the dataset and code are publicly released.
ClinPRISM: Cost-effective multimodal LLM framework for clinical time series question answering
Researchers introduce ClinPRISM, a multimodal LLM reasoning framework designed for question answering over irregular clinical time series (ICTS). The system combines an irregularity-aware multi-scale encoder, a temporal evidence distiller, and a progressive alignment strategy to handle the sparsity and asynchrony typical of clinical observations. Built on a 4B-parameter LLM backbone, it achieves state-of-the-art results on a held-out clinical benchmark using only 16 time-series tokens with 0.15-second inference latency. The authors also release a training corpus of 30,000 clinical time series with multi-scale descriptions and 41,000 instruction-tuning instances across 11 tasks.
ClinFusion: Vision-centric multimodal LLM for holistic medical image understanding
Researchers introduce ClinFusion, a multimodal LLM system designed for clinical medical understanding, featuring a cascaded vision encoder architecture (Cascade Spatial-Aware Locality Fusion) that handles both 2D and native 3D medical images within a unified encoder. The system is evaluated on a new benchmark suite including MedIF-Bench and a region-of-interest-grounded report generation metric, claiming state-of-the-art results on 20 of 24 benchmarks against open-source medical MLLMs and outperforming GPT-5.2 and Gemini-3-Flash on 13 of 16 multimodal benchmarks. Blinded evaluation by board-certified radiologists confirms ClinFusion produces the highest-ranked radiology reports, and the proposed RoI-grounded metric shows the strongest correlation with expert judgment among automatic metrics tested.
TriViewBench: Controlled benchmark reveals fundamental multi-view spatial reasoning failures in MLLMs
Researchers introduce TriViewBench, a synthetic 3D benchmark of 1,923 scenes and 14K+ QA pairs designed to probe multi-view structural reasoning in MLLMs under controlled complexity scaling. Evaluating 18 open- and closed-source models, the study finds a universal capability hierarchy (Local Decision > Object Counting > Global Recovery) with severe performance collapse on Global Recovery tasks (80% relative drop at highest complexity). Chain-of-Thought prompting provides near-zero benefit, suggesting the bottleneck is cross-view spatial representation rather than reasoning strategy. The work identifies two mechanistically distinct failure modes in object counting: occlusion blindness causing undercounting in single-view tasks and cross-view identity confusion causing overcounting in multi-view tasks.