5arXiv cs.CL (Computation and Language)·
MedDDC-Eval: Diagnosis-Decoupled Evaluation Framework for Multi-Turn Medical Consultation Agents
Researchers introduce MedDDC-Eval, a benchmark framework that decouples history-elicitation quality from diagnosis generation in multi-turn medical consultation agents, using a shared frozen reader to hold the history-to-diagnosis mapping constant. The paper demonstrates that varying only the diagnostic reader shifts diagnosis F1 by 2.2–19.0 points and reverses 18–36% of pairwise policy orderings, exposing confounds in coupled evaluation. The authors also apply Group Relative Policy Optimization (GRPO) to post-train Qwen3-32B using diagnosis-result and trajectory feedback, achieving 9.7 and 4.6 total-score point improvements on two evaluation splits. The work addresses a methodological gap in evaluating medical dialogue agents by enabling controlled attribution of policy quality.
Related guides (3)
Related events (8)
ClinEnv: Interactive Multi-Stage Long-Horizon EHR Benchmark for Clinical Agent Evaluation
ClinEnv is a new interactive benchmark that evaluates LLMs as attending physicians over real inpatient admissions using a Longitudinal Inpatient Simulation paradigm. Each case is decomposed into sequential decision stages where models must query four specialized agents before committing to medications, procedures, and diagnoses. Across seven evaluated models, the best achieves only 0.31 decision F1, with a sharp gap between diagnosis recovery (0.51 F1) and management actions (0.17 F1). The benchmark uniquely measures information-acquisition process quality alongside outcome quality, exposing a gap invisible to static or outcome-only evaluations.
ClinMM-Bench: Multi-turn multimodal benchmark for clinical diagnostic reasoning across 1,089 real-world cases
Researchers introduce ClinMM-Bench, a benchmark of 1,089 real-world clinical cases and 3,760 medical images spanning eight specialties, designed to evaluate multimodal LLMs on multi-turn diagnostic reasoning with progressive information disclosure. Fifteen MLLMs were evaluated using a two-level framework assessing both diagnostic accuracy and reasoning quality. Proprietary models led on accuracy but all models showed limited rates of fully correct diagnoses, and error analysis identified five failure modes including visual hallucination and premature closure. The work addresses a gap in clinical AI evaluation by moving beyond single-turn isolated tasks toward more realistic diagnostic workflows.
MedRLM: Recursive multimodal agent framework for long-context clinical decision support
MedRLM is a proposed framework for clinical decision support that uses recursive multi-agent reasoning over heterogeneous patient data including EHRs, medical images, physiological sensor streams, and clinical guidelines. Rather than single-step prompting, it decomposes patient cases into an inspectable external environment coordinated by specialized agents, with a Clinical Evidence Graph Memory and sensor-triggered deeper reasoning. The paper outlines an evaluation design using public and credentialed clinical datasets spanning radiology, ECG, ICU time series, and referral outcomes. The work targets a gap between static medical QA benchmarks and real-world longitudinal clinical workflows.
ThReadMed-QA benchmark reveals frontier LLMs degrade sharply on medical misconception correction across multi-turn conversations
Researchers introduce ThReadMed-QA, a multi-turn medical dialogue dataset of 2,437 conversation threads (8,204 QA pairs) derived from real AskDocs patient interactions, designed to evaluate LLM ability to detect and correct embedded misconceptions over multiple turns. Evaluation of five LLMs using an LLM-as-a-Judge rubric finds that even frontier models like GPT-5 and Claude Haiku drop from ~85% misconception correction on initial questions to ~50% within two follow-up turns. Oracle analysis attributes much of this degradation to error propagation from prior model outputs rather than context length alone. The findings highlight a significant safety gap in current evaluation frameworks, which do not capture multi-turn conversational dynamics in medical settings.
MedRealMM: Real-world multimodal benchmark for Chinese online medical consultation
Researchers introduce MedRealMM, a benchmark of 5,620 real-world multimodal patient-doctor interaction cases drawn from a nationwide Chinese internet hospital, spanning 64 clinical departments. The benchmark uses a Multimodal Clinical Challenge Point (MCCP) framework to identify clinically demanding moments and evaluates 19 general-purpose and medical-specialized LLMs. Key findings show that frontier models still fall below online physician performance, particularly on safety-sensitive error avoidance, and that image information is critical for reliable clinical performance. The dataset will be publicly released on Hugging Face.
LLM-guided MAP-Elites evolution improves medical decision pipelines at inference time
Researchers propose using LLM-guided MAP-Elites evolutionary search as an inference-time alternative to fine-tuning for adapting LLMs to clinical workflows, formulating triage, consultation, and image classification as evolutionary searches over executable artifacts. Across three medical settings, evolved programs substantially outperform manually designed baselines: triage accuracy improves from 77.3% to 87.1% and emergency recall from 0.60 to 0.97, with gains also shown on MIMIC-ESI, iCRAFTMD, and PneumoniaMNIST. The approach works across Llama-3, Qwen-3.5, and Gemma-4 backbones and produces interpretable program-level mechanisms rather than superficial prompt changes.
Retrospective analysis of multimodal VQA systems for clinical GI endoscopy reveals evaluation gaps
A new arXiv paper analyzes nine documented systems from the MediaEval Medico 2025 challenge for GI endoscopy visual question answering, examining design choices around parameter-efficient adaptation, structured reasoning, and explanation quality. The authors find that leaderboard performance gains do not reliably translate to faithful clinical reasoning, while methods enforcing structured reasoning and explicit grounding show more consistent behavior. The paper argues for evaluation frameworks that go beyond lexical overlap, incorporating evidence-linked explanations, leakage-aware data governance, and calibration checks for trustworthy healthcare AI.
C-DIC: Context-Driven Incremental Compression for efficient long-horizon multi-turn dialogue
A new arXiv preprint introduces Context-Driven Incremental Compression (C-DIC), a method for managing growing dialogue history in conversational agents by treating conversations as interleaved contextual threads with revisable per-thread compression states stored in a compact dialogue memory. A retrieve-revise-write-back loop shares information across turns and updates stale memories, while truncated backpropagation-through-time (TBPTT) is adapted to learn cross-turn dependencies. Experiments on long-form dialogue benchmarks show stable inference latency and perplexity over hundreds of turns, addressing compounding errors seen in existing context compressors.