A Multi-Agent System for Autonomous, Fine-Tuning-Free Clinical Symptom Detection: Development and Validation Study

Pythia: Multi-agent system for fine-tuning-free clinical symptom extraction from notes

Researchers present Pythia, a multi-agent system that autonomously writes and optimizes extraction prompts for clinical concepts in medical notes without manual prompt engineering or model fine-tuning. Running on locally hosted open-weights models, Pythia achieved mean sensitivity of 0.76 and specificity of 0.95 across 72 symptoms from 400 clinical notes, outperforming a curated lexicon on specificity and a per-concept BERT classifier on both metrics. The system's key advantage is recovering high specificity (0.97) for concepts where lexicon-based approaches over-trigger, while remaining deployable on local infrastructure for data privacy. Sensitivity degrades below 5% prevalence, a noted limitation for rare findings.