AskDocs

ThReadMed-QA benchmark reveals frontier LLMs degrade sharply on medical misconception correction across multi-turn conversations

Researchers introduce ThReadMed-QA, a multi-turn medical dialogue dataset of 2,437 conversation threads (8,204 QA pairs) derived from real AskDocs patient interactions, designed to evaluate LLM ability to detect and correct embedded misconceptions over multiple turns. Evaluation of five LLMs using an LLM-as-a-Judge rubric finds that even frontier models like GPT-5 and Claude Haiku drop from ~85% misconception correction on initial questions to ~50% within two follow-up turns. Oracle analysis attributes much of this degradation to error propagation from prior model outputs rather than context length alone. The findings highlight a significant safety gap in current evaluation frameworks, which do not capture multi-turn conversational dynamics in medical settings.