6arXiv cs.AI (Artificial Intelligence)·
ClinFusion: Vision-centric multimodal LLM for holistic medical image understanding
Researchers introduce ClinFusion, a multimodal LLM system designed for clinical medical understanding, featuring a cascaded vision encoder architecture (Cascade Spatial-Aware Locality Fusion) that handles both 2D and native 3D medical images within a unified encoder. The system is evaluated on a new benchmark suite including MedIF-Bench and a region-of-interest-grounded report generation metric, claiming state-of-the-art results on 20 of 24 benchmarks against open-source medical MLLMs and outperforming GPT-5.2 and Gemini-3-Flash on 13 of 16 multimodal benchmarks. Blinded evaluation by board-certified radiologists confirms ClinFusion produces the highest-ranked radiology reports, and the proposed RoI-grounded metric shows the strongest correlation with expert judgment among automatic metrics tested.
Related guides (3)
Related events (8)
LoRA-based cascaded multimodal fusion framework for action recognition in medical training environments
A new arXiv preprint proposes a cascaded Low-Rank Adaptation (LoRA) framework for multimodal fusion applied to action and activity recognition in healthcare training settings. The architecture integrates modalities sequentially without retraining prior components, enabling scalable adaptation across datasets with different modality sets. Evaluation on two healthcare datasets (NurViD and Nurse Training) shows competitive performance against dataset-specific baselines. The work is a narrow application of parameter-efficient fine-tuning techniques to a specialized domain.
ClinMM-Bench: Multi-turn multimodal benchmark for clinical diagnostic reasoning across 1,089 real-world cases
Researchers introduce ClinMM-Bench, a benchmark of 1,089 real-world clinical cases and 3,760 medical images spanning eight specialties, designed to evaluate multimodal LLMs on multi-turn diagnostic reasoning with progressive information disclosure. Fifteen MLLMs were evaluated using a two-level framework assessing both diagnostic accuracy and reasoning quality. Proprietary models led on accuracy but all models showed limited rates of fully correct diagnoses, and error analysis identified five failure modes including visual hallucination and premature closure. The work addresses a gap in clinical AI evaluation by moving beyond single-turn isolated tasks toward more realistic diagnostic workflows.
ClinPRISM: Cost-effective multimodal LLM framework for clinical time series question answering
Researchers introduce ClinPRISM, a multimodal LLM reasoning framework designed for question answering over irregular clinical time series (ICTS). The system combines an irregularity-aware multi-scale encoder, a temporal evidence distiller, and a progressive alignment strategy to handle the sparsity and asynchrony typical of clinical observations. Built on a 4B-parameter LLM backbone, it achieves state-of-the-art results on a held-out clinical benchmark using only 16 time-series tokens with 0.15-second inference latency. The authors also release a training corpus of 30,000 clinical time series with multi-scale descriptions and 41,000 instruction-tuning instances across 11 tasks.
MedRealMM: Real-world multimodal benchmark for Chinese online medical consultation
Researchers introduce MedRealMM, a benchmark of 5,620 real-world multimodal patient-doctor interaction cases drawn from a nationwide Chinese internet hospital, spanning 64 clinical departments. The benchmark uses a Multimodal Clinical Challenge Point (MCCP) framework to identify clinically demanding moments and evaluates 19 general-purpose and medical-specialized LLMs. Key findings show that frontier models still fall below online physician performance, particularly on safety-sensitive error avoidance, and that image information is critical for reliable clinical performance. The dataset will be publicly released on Hugging Face.
FusionRS: Large-scale RGB-infrared-text dataset for dual-modal remote sensing vision-language models
Researchers introduce FusionRS, the first large-scale dataset pairing RGB and infrared remote sensing images with both conventional and IR-aware text captions, designed to support dual-modal vision-language learning. The dataset is constructed by translating public RGB remote sensing images into infrared-style counterparts using image translation. Using FusionRS, the authors train CLIP-style alignment models and fine-tune generative VLMs, demonstrating improvements in RGB-IR alignment, infrared-to-text retrieval, and dual-modal captioning over RGB-only baselines. The work addresses a gap in multimodal remote sensing foundation models by providing modality-specific textual supervision for infrared imagery.
LoMo: Local Modality Substitution for Deeper Vision-Language Fusion
This paper identifies a 'carrier sensitivity' problem in Vision-Language Models (VLMs), where replacing textual queries with rendered-image equivalents causes significant performance degradation due to asymmetric roles of text and images in training data. The authors propose Local Modality Substitution (LoMo), a data curation paradigm that reformulates single-modality prompts into interleaved multimodal sequences by dynamically rendering text spans as images, enforcing cross-modal representational invariance. Evaluated across 13 multimodal benchmarks, LoMo improves over standard supervised fine-tuning by 2.67 points on LLaVA-OneVision-1.5-8B and 2.82 points on Qwen3.5-9B. The approach is architecture-agnostic and lightweight, requiring no changes to model architecture.
MedRLM: Recursive multimodal agent framework for long-context clinical decision support
MedRLM is a proposed framework for clinical decision support that uses recursive multi-agent reasoning over heterogeneous patient data including EHRs, medical images, physiological sensor streams, and clinical guidelines. Rather than single-step prompting, it decomposes patient cases into an inspectable external environment coordinated by specialized agents, with a Clinical Evidence Graph Memory and sensor-triggered deeper reasoning. The paper outlines an evaluation design using public and credentialed clinical datasets spanning radiology, ECG, ICU time series, and referral outcomes. The work targets a gap between static medical QA benchmarks and real-world longitudinal clinical workflows.
SeRIn: Decomposed multimodal fusion architecture achieves state-of-the-art on sentiment analysis benchmarks
Researchers propose SeRIn (Segregate, Refine, Integrate), a multimodal language model fusion scheme that separates modality-specific refinement from cross-modal interaction via distinct architectural pathways. The design defers full cross-modal interaction to a final prediction step, with ablations showing structured interaction rather than added capacity drives performance gains. SeRIn achieves state-of-the-art results on CH-SIMS and CMU-MOSEI benchmarks, and exhibits emergent modality reweighting under visual corruption without explicit supervision.