5arXiv cs.AI (Artificial Intelligence)·
Pythia: Multi-agent system for fine-tuning-free clinical symptom extraction from notes
Researchers present Pythia, a multi-agent system that autonomously writes and optimizes extraction prompts for clinical concepts in medical notes without manual prompt engineering or model fine-tuning. Running on locally hosted open-weights models, Pythia achieved mean sensitivity of 0.76 and specificity of 0.95 across 72 symptoms from 400 clinical notes, outperforming a curated lexicon on specificity and a per-concept BERT classifier on both metrics. The system's key advantage is recovering high specificity (0.97) for concepts where lexicon-based approaches over-trigger, while remaining deployable on local infrastructure for data privacy. Sensitivity degrades below 5% prevalence, a noted limitation for rare findings.
Related guides (2)
Related events (8)
Sentence-Level Clinical Provenance Categorization for Multidisciplinary Hospital Summarization Using Fine-Tuned Llama-3
This pilot study presents a pipeline for categorizing sentence-level clinical provenance across multi-source hospital notes, targeting structured summarization in high-complexity settings like the NICU. The authors fine-tune Llama-3 8B and 70B models on MedSecId (MIMIC-III annotations), achieving Macro F1 above 92% in-domain. Cross-domain evaluation reveals a scale-dependent transfer effect: SFT substantially improves the 70B model (+7% Macro F1) but yields only marginal gains for the 8B model. A quantized fine-tuned 70B model outperforms its full-precision baseline while reducing compute, suggesting quantized adaptation is viable for structured clinical NLP tasks.
HealthClaw: Self-evolving agent architecture for longitudinal personal health management
Researchers introduce HealthClaw, an open-source agent architecture that maintains and updates a structured longitudinal memory across repeated health interactions, separating shared medical knowledge from private per-user profiles, procedures, and episodic traces. Evaluated on a synthetic year-long benchmark and nine biomedical tasks, HealthClaw improves answer accuracy from 0.2% to 45.7% over current-query prompting while reducing prompt-side context exposure by 71.7% versus full-history prompting. Privacy probes show fewer unsafe disclosures than baselines. The authors note that offline benchmark results require prospective clinical validation before deployment claims.
Fine-tuning LLMs to passively estimate depression severity from AI mental health conversations
Researchers fine-tune a Qwen3.5-27B model with a regression head to predict PHQ-9 depression severity scores directly from AI mental health app conversation transcripts, eliminating the need for explicit self-report completion. The training set of 6,283 users combines 3,111 ground-truth labels with pseudolabels generated by Claude Opus and iterative intermediate models. On a held-out test of 842 users, the best model achieves MAE=2.6, Pearson r=0.80, and AUC=0.91 at the clinical PHQ-9≥10 threshold, with AUC>0.87 across all severity thresholds. The work demonstrates a passive, continuous symptom-monitoring approach that could reduce response bias in mental health platforms.
TxBench-PP: New benchmark reveals AI agents struggle with preclinical pharmacology decisions
Researchers introduce TxBench-PP (TherapeuticsBench Preclinical Pharmacology), a 100-evaluation benchmark testing AI agents on realistic small-molecule drug discovery tasks including mechanism-of-action reasoning, compound-target engagement, and translational efficacy. Agents receive real workflow snapshots and are graded deterministically on structured answers. Across 16 model-harness configurations and 4,800 trajectories, no system reliably succeeded; the best performer, Claude Opus 4.8 with the Pi harness, passed only 59.3% of endpoint attempts. The results suggest current frontier models are not yet deployment-ready for autonomous preclinical pharmacology decision-making.
ArogyaSutra: Multi-agent framework for multimodal medical reasoning in Indic languages
Researchers introduce ArogyaSutra, an actor-critic-based multi-agent framework for multilingual multimodal medical reasoning targeting Indic languages, alongside ArogyaBodha, a large-scale dataset spanning 31 body systems, six imaging modalities, and 21 clinical domains across English and seven Indian languages. The framework integrates tool grounding with dual-memory mechanisms and uses actor-critic simulation trajectories for distillation. The work addresses a critical gap in AI healthcare access for low-resource, multilingual settings like rural India where English-centric MLLMs fall short.
Risk-Aware Hybrid Selective Classification for HIV Suspicion Identification in Spanish Clinical Notes
This paper proposes a hybrid selective classification framework for clinical NLP that explicitly handles both aleatoric and epistemic uncertainty to avoid overconfident predictions in medical triage settings. The system combines Mondrian conformal prediction with a Multi-Centroid Mahalanobis Distance veto, evaluated on HIV suspicion identification in Spanish clinical notes. The authors demonstrate that standard uncertainty metrics and baseline classifiers suffer coverage collapse under strict reliability constraints, while their dual-verification approach isolates a trustworthy operational domain. The work critiques inflated benchmark metrics that arise from forcing deterministic classification on inherently ambiguous clinical instances.
RubricsTree: Scalable hierarchical rubric framework for evaluating personal health AI agents
RubricsTree is a new evaluation framework for LLM-powered personal health agents, built around a hierarchical taxonomy of over 100 clinically-verifiable Boolean rubrics derived from 4,000 real user queries and curated with physician oversight. A context-aware router activates only relevant rubrics per query, enabling scalable yet expert-aligned evaluation. The framework outperforms strong LLM-as-a-judge baselines on expert alignment and, when used as training signal, yields up to ~66% relative gains on HealthBench across Gemini, GPT, and Qwen model families. The work addresses a concrete bottleneck in clinical deployment of health AI: the cost-quality tradeoff in evaluation.
EMPATH: Multilingual multi-turn safety benchmark for emotional-support chatbots reveals score inflation and run-to-run reliability failures
EMPATH is a new arXiv benchmark for evaluating the safety of emotional-support chatbots, using an auditor model to generate multi-turn crisis conversations and a calibrated judge model to score transcripts across 19 metrics in five dimensions. Built for Mexican Spanish and US English, the benchmark surfaces score inflation on 10 of 19 metrics under uncalibrated rubrics and finds that run-to-run reliability is a per-model safety property: one model swings 2–10 points on a crisis metric across identical reruns, and DeepSeek V4 Pro produces different conversations at temperature 0. Evaluation of three frontier models shows aggregate scores within 0.74 points but per-metric divergences up to six points, with rankings stable across a cross-family judge at 93% within ±1.