6arXiv cs.CL (Computation and Language)·
Study finds LLM health chatbot triage outcomes vary significantly with patient communication style
Researchers analyzed 2,053 real patient-chatbot conversations and found wide variation in communication patterns and emotional expression, revealing that idealized patient simulations used in chatbot development are inadequate. They built a patient simulator modeling clinical content, emotional state, conversational strategy, and communication style, achieving near-indistinguishability from real conversations (human graders 55% accurate). Evaluating four LLMs across 1,164 clinician-graded cases with five patient personae, the study found that communication style significantly alters triage urgency assessments. The authors warn that systems optimized for cooperative, articulate users risk underperforming and amplifying health disparities in real-world deployment.
Related guides (3)
Related events (8)
Large-scale study of user-reported breakdowns in AI healthcare chatbots across 59 apps
Researchers analyzed over 15,000 user reviews from 59 AI healthcare chatbot apps to identify recurring failure modes in real-world deployment. Topic modeling and interpretive analysis surfaced three breakdown categories: access and reliability issues, interaction quality problems, and billing/support failures, with privacy and security concerns correlating with the most negative user experiences. The study frames healthcare chatbots as information infrastructure and draws implications for designers, policymakers, and health information professionals.
Systematic evaluation of LLM prompt sensitivity in healthcare settings reveals safety risks
Researchers conduct a sensitivity analysis of both general-purpose and medical-specific LLMs using the MedMCQA benchmark, testing robustness to lexical and syntactic prompt perturbations. The study finds that even minor phrasing changes can alter clinical advice, and adversarial prompts can produce dangerous outputs such as incorrect dosages or omitted critical findings. Both general-purpose models (GPT-3.5, Llama 3) and domain-specific models (ClinicalBERT, BioLlama3, BioBERT) exhibit this fragility, with syntactic reordering and misleading contextual cues proving more destabilizing than simple paraphrasing.
Study of security and privacy prompts in the wild reveals LLM response quality gaps and inconsistency
Researchers analyzed 14,727 security and privacy (S&P) prompts drawn from WildChat's 3.2M real user-LLM conversations, categorizing them into nine topic areas and evaluating response quality across 270 advice-seeking prompts. Commercial models substantially outperformed open-weight models (GPT achieving 98% 'good enough' responses vs. Llama 4 at 47%), but even high-performing commercial models showed inconsistent responses across repeated runs of the same prompt. The study is the first to analyze real user S&P queries to LLMs rather than expert-authored test sets, surfacing both a capability gap and a reliability concern.
Systematic Evaluation of LLM Safety Failures on Eating Disorder Queries with Clinician Feedback
This paper investigates how LLMs respond to queries from users with eating disorders, finding that specific linguistic cues in prompts increase the likelihood of unsafe model responses. Working with clinical ED experts, the authors systematically vary risk levels in user prompts to measure the extent to which LLMs uncritically adapt to potentially dangerous inputs. The study highlights a gap between perceived model safety and actual harm facilitation in sensitive health contexts.
RAG-based multimodal university chatbot reduces hallucination from 31.7% to 6.6%
Researchers present a multimodal university chatbot system combining LLMs with retrieval-augmented generation to answer text and image queries from institutional documents like university handbooks. The system uses a vision-language model, quantized inference for constrained hardware, and a FastAPI/Next.js stack. Evaluation shows hallucination reduction from 31.7% to 6.6% compared to a non-RAG baseline, with strong satisfaction scores across modalities.
EMPATH: Multilingual multi-turn safety benchmark for emotional-support chatbots reveals score inflation and run-to-run reliability failures
EMPATH is a new arXiv benchmark for evaluating the safety of emotional-support chatbots, using an auditor model to generate multi-turn crisis conversations and a calibrated judge model to score transcripts across 19 metrics in five dimensions. Built for Mexican Spanish and US English, the benchmark surfaces score inflation on 10 of 19 metrics under uncalibrated rubrics and finds that run-to-run reliability is a per-model safety property: one model swings 2–10 points on a crisis metric across identical reruns, and DeepSeek V4 Pro produces different conversations at temperature 0. Evaluation of three frontier models shows aggregate scores within 0.74 points but per-metric divergences up to six points, with rankings stable across a cross-family judge at 93% within ±1.
MedRealMM: Real-world multimodal benchmark for Chinese online medical consultation
Researchers introduce MedRealMM, a benchmark of 5,620 real-world multimodal patient-doctor interaction cases drawn from a nationwide Chinese internet hospital, spanning 64 clinical departments. The benchmark uses a Multimodal Clinical Challenge Point (MCCP) framework to identify clinically demanding moments and evaluates 19 general-purpose and medical-specialized LLMs. Key findings show that frontier models still fall below online physician performance, particularly on safety-sensitive error avoidance, and that image information is critical for reliable clinical performance. The dataset will be publicly released on Hugging Face.
SIMAX framework generates annotated synthetic clinician-patient dialogues for AI communication coding evaluation
Researchers introduce SIMAX, a framework for generating controlled, annotated synthetic clinician-patient dialogues to support development and evaluation of AI-driven clinical communication coding systems. The framework produces dialogues with reference behavioral annotations using two codebooks (Global and WISER), generating 3,388 simulated dialogues across three medical specialties with varied personas and accent conditions. Evaluation shows reasonable speech naturalness and high transcription fidelity, with downstream testing revealing the framework can expose sensitivity gaps in communication coding systems. The work addresses a data scarcity bottleneck in deploying ambient AI scribes in clinical settings.