4arXiv cs.CL (Computation and Language)·
MedGame: LLM-powered storytelling framework and benchmark for clinical medical education
MedGame is a new framework that converts static clinical cases into interactive, decision-centered storytelling games using a dual-engine LLM architecture: a Medical Narrative Designer and a Story Director. The authors also release MedGame Bench, a 5,000-case benchmark for evaluating medical narrative generation and story direction. Task-specific fine-tuning of open-source LLMs substantially closes the gap with commercial models on the benchmark, and a pilot student study reports improved engagement over text-only alternatives.
Related guides (1)
Related events (8)
MedRLM: Recursive multimodal agent framework for long-context clinical decision support
MedRLM is a proposed framework for clinical decision support that uses recursive multi-agent reasoning over heterogeneous patient data including EHRs, medical images, physiological sensor streams, and clinical guidelines. Rather than single-step prompting, it decomposes patient cases into an inspectable external environment coordinated by specialized agents, with a Clinical Evidence Graph Memory and sensor-triggered deeper reasoning. The paper outlines an evaluation design using public and credentialed clinical datasets spanning radiology, ECG, ICU time series, and referral outcomes. The work targets a gap between static medical QA benchmarks and real-world longitudinal clinical workflows.
MedRealMM: Real-world multimodal benchmark for Chinese online medical consultation
Researchers introduce MedRealMM, a benchmark of 5,620 real-world multimodal patient-doctor interaction cases drawn from a nationwide Chinese internet hospital, spanning 64 clinical departments. The benchmark uses a Multimodal Clinical Challenge Point (MCCP) framework to identify clinically demanding moments and evaluates 19 general-purpose and medical-specialized LLMs. Key findings show that frontier models still fall below online physician performance, particularly on safety-sensitive error avoidance, and that image information is critical for reliable clinical performance. The dataset will be publicly released on Hugging Face.
MedCase-Structured: A Text-to-FHIR Dataset for Benchmarking Diagnostic Reasoning in Clinically Realistic EHR Settings
The paper introduces a pipeline for converting unstructured clinical text into HL7 FHIR R4 bundles, enabling evaluation of LLMs in realistic electronic health record settings. Applied to the MedCaseReasoning dataset, it produces MedCase-Structured, a synthetic benchmark achieving valid FHIR generation for 82.5% of cases. Key finding: LLMs show consistently lower diagnostic accuracy on structured FHIR inputs compared to plain text, underscoring the gap between standard benchmarks and real-world clinical deployment conditions.
MedQADE benchmark reveals LLM evaluators match physician agreement scores but lack clinical caution and show lineage bias
Researchers introduce MedQADE, a standardized open-response clinical benchmark for German comprising 3,800 items annotated by ten physicians and nine LLM evaluators. The top LLM evaluator (Gemini 3 Flash) reached statistical alignment near the physician inter-rater ceiling (κ=0.694 vs. κ=0.709), but automated evaluators showed near-zero clinical metacognition: unlike physicians, they never abstained regardless of item difficulty. The study also documents systematic lineage-dependent scoring bias, where models preferentially rate architectural siblings more favorably, independent of language.
MedMisBench: LLMs show fragile epistemic resilience under misleading medical context
Researchers introduce MedMisBench, a benchmark of 10,932 medical questions paired with 48,889 misleading context injections, to measure whether LLMs maintain correct medical judgment under adversarial pressure. Across 11 model configurations, mean accuracy drops from 71.1% to 38.0% when misleading context is injected, with authority-framed falsehoods achieving 69.5% attack success. A 14-member international clinical panel flagged serious potential harm in 38.2% of reviewed cases. The work argues that existing medical benchmarks measure knowledge but not robustness to manipulation, exposing a structural gap in LLM safety evaluation for healthcare.
LLM-guided MAP-Elites evolution improves medical decision pipelines at inference time
Researchers propose using LLM-guided MAP-Elites evolutionary search as an inference-time alternative to fine-tuning for adapting LLMs to clinical workflows, formulating triage, consultation, and image classification as evolutionary searches over executable artifacts. Across three medical settings, evolved programs substantially outperform manually designed baselines: triage accuracy improves from 77.3% to 87.1% and emergency recall from 0.60 to 0.97, with gains also shown on MIMIC-ESI, iCRAFTMD, and PneumoniaMNIST. The approach works across Llama-3, Qwen-3.5, and Gemma-4 backbones and produces interpretable program-level mechanisms rather than superficial prompt changes.
ThReadMed-QA benchmark reveals frontier LLMs degrade sharply on medical misconception correction across multi-turn conversations
Researchers introduce ThReadMed-QA, a multi-turn medical dialogue dataset of 2,437 conversation threads (8,204 QA pairs) derived from real AskDocs patient interactions, designed to evaluate LLM ability to detect and correct embedded misconceptions over multiple turns. Evaluation of five LLMs using an LLM-as-a-Judge rubric finds that even frontier models like GPT-5 and Claude Haiku drop from ~85% misconception correction on initial questions to ~50% within two follow-up turns. Oracle analysis attributes much of this degradation to error propagation from prior model outputs rather than context length alone. The findings highlight a significant safety gap in current evaluation frameworks, which do not capture multi-turn conversational dynamics in medical settings.
The Open Medical-LLM Leaderboard: Benchmarking Large Language Models in Healthcare
Hugging Face has launched the Open Medical-LLM Leaderboard, a public benchmark for evaluating large language models on healthcare and medical tasks. The leaderboard aggregates performance across multiple medical question-answering datasets to enable standardized comparison of open-weight models in clinical and biomedical domains. This initiative aims to accelerate progress in medical AI by providing transparent, reproducible evaluation infrastructure.