4arXiv cs.CL (Computation and Language)·
MIRA-Ev benchmark introduces granular evidence and argumentation evaluation for clinical NLP
MIRA-Ev is a new clinical argument mining benchmark built on Spanish medical licensing exam cases, annotated by expert clinicians with span-level premises, claims, and support/attack relations. It addresses a gap in clinical NLP evaluation where MCQA benchmarks only score final-answer accuracy, missing cases where models reach correct answers via flawed reasoning. The benchmark is released in Spanish, English, and Basque (the first clinical argumentation resource in Basque) and organizes evaluation into three tiers: evidence retrieval, argumentative component extraction, and relation classification.
Related guides (1)
Related events (8)
MedFailBench: Clinician-built benchmark for medical AI safety boundary failures
MedFailBench is a new open-source benchmark designed to identify which safety boundary a medical AI system failed, rather than measuring correctness. The v0.2.1 release contains 44 clinician-reviewed synthetic cases annotated by severity (1–5) and six safety gate types including missed urgent escalation, unsafe dosing, and evidence fabrication. The benchmark includes a HuggingFace leaderboard preview, a safety gate taxonomy, and an automated pipeline for archiving model-response screening runs, released under Apache-2.0 and CC-BY-4.0.
ClinEnv: Interactive Multi-Stage Long-Horizon EHR Benchmark for Clinical Agent Evaluation
ClinEnv is a new interactive benchmark that evaluates LLMs as attending physicians over real inpatient admissions using a Longitudinal Inpatient Simulation paradigm. Each case is decomposed into sequential decision stages where models must query four specialized agents before committing to medications, procedures, and diagnoses. Across seven evaluated models, the best achieves only 0.31 decision F1, with a sharp gap between diagnosis recovery (0.51 F1) and management actions (0.17 F1). The benchmark uniquely measures information-acquisition process quality alongside outcome quality, exposing a gap invisible to static or outcome-only evaluations.
MedRLM: Recursive multimodal agent framework for long-context clinical decision support
MedRLM is a proposed framework for clinical decision support that uses recursive multi-agent reasoning over heterogeneous patient data including EHRs, medical images, physiological sensor streams, and clinical guidelines. Rather than single-step prompting, it decomposes patient cases into an inspectable external environment coordinated by specialized agents, with a Clinical Evidence Graph Memory and sensor-triggered deeper reasoning. The paper outlines an evaluation design using public and credentialed clinical datasets spanning radiology, ECG, ICU time series, and referral outcomes. The work targets a gap between static medical QA benchmarks and real-world longitudinal clinical workflows.
EMPATH: Multilingual multi-turn safety benchmark for emotional-support chatbots reveals score inflation and run-to-run reliability failures
EMPATH is a new arXiv benchmark for evaluating the safety of emotional-support chatbots, using an auditor model to generate multi-turn crisis conversations and a calibrated judge model to score transcripts across 19 metrics in five dimensions. Built for Mexican Spanish and US English, the benchmark surfaces score inflation on 10 of 19 metrics under uncalibrated rubrics and finds that run-to-run reliability is a per-model safety property: one model swings 2–10 points on a crisis metric across identical reruns, and DeepSeek V4 Pro produces different conversations at temperature 0. Evaluation of three frontier models shows aggregate scores within 0.74 points but per-metric divergences up to six points, with rankings stable across a cross-family judge at 93% within ±1.
MedQADE benchmark reveals LLM evaluators match physician agreement scores but lack clinical caution and show lineage bias
Researchers introduce MedQADE, a standardized open-response clinical benchmark for German comprising 3,800 items annotated by ten physicians and nine LLM evaluators. The top LLM evaluator (Gemini 3 Flash) reached statistical alignment near the physician inter-rater ceiling (κ=0.694 vs. κ=0.709), but automated evaluators showed near-zero clinical metacognition: unlike physicians, they never abstained regardless of item difficulty. The study also documents systematic lineage-dependent scoring bias, where models preferentially rate architectural siblings more favorably, independent of language.
MedRealMM: Real-world multimodal benchmark for Chinese online medical consultation
Researchers introduce MedRealMM, a benchmark of 5,620 real-world multimodal patient-doctor interaction cases drawn from a nationwide Chinese internet hospital, spanning 64 clinical departments. The benchmark uses a Multimodal Clinical Challenge Point (MCCP) framework to identify clinically demanding moments and evaluates 19 general-purpose and medical-specialized LLMs. Key findings show that frontier models still fall below online physician performance, particularly on safety-sensitive error avoidance, and that image information is critical for reliable clinical performance. The dataset will be publicly released on Hugging Face.
MedMisBench: LLMs show fragile epistemic resilience under misleading medical context
Researchers introduce MedMisBench, a benchmark of 10,932 medical questions paired with 48,889 misleading context injections, to measure whether LLMs maintain correct medical judgment under adversarial pressure. Across 11 model configurations, mean accuracy drops from 71.1% to 38.0% when misleading context is injected, with authority-framed falsehoods achieving 69.5% attack success. A 14-member international clinical panel flagged serious potential harm in 38.2% of reviewed cases. The work argues that existing medical benchmarks measure knowledge but not robustness to manipulation, exposing a structural gap in LLM safety evaluation for healthcare.
Rubrics on Trial: Self-evolving evaluation rubrics from a single query via synthetic pairwise evidence
A new arXiv preprint introduces 'Rubrics on Trial,' a query-only framework that automatically constructs and validates evaluation rubrics for LLMs without external annotations or model training. The system evolves rubric sets from scratch using synthetic rubric-conditioned response pairs, screening out non-discriminative, over-specific, or style-only rubrics. Evaluated across five preference benchmark suites, it achieves best average accuracy and leads on six of seven evaluation sets, addressing a key bottleneck in scalable LLM evaluation.