paper

Accuracy and Satisfaction in Multi-Turn LLM Dialogues for NFR Assessment

paperactiveprovisionalaccuracy-and-satisfaction-in-multi-turn-llm-dialogues-for-nfr-assessment-cc3052f9·1 events·first seen 30h ago

Aliases: Accuracy and Satisfaction in Multi-Turn LLM Dialogues for NFR Assessment

Co-occurring entities

Microsoft iTrust GitHub Copilot

More like this (12)

Measuring Semantic Progress in Multi-turn Dialogue via Information Gain Multi-Turn Evaluation of Deep Research Agents Under Process-Level Feedback Janus: A Benchmark for Goal-Conditioned Information Distortion in LLMs Context-Driven Incremental Compression for Multi-Turn Dialogue Generation Context-Driven Incremental Compression for Multi-Turn Dialogue Generation When the Chain of Thought Knows Better: Failure Modes in Multi-Turn Reasoning Models RAS: Measuring LLM Safety Through Refusal Alignment Revising Context, Shifting Simulated Stance: Auditing LLM-Based Stance Simulation in Online Discussions On The Effectiveness-Fluency Trade-Off In LLM Conditioning: A Systematic Study ExpRL: Exploratory RL for LLM Mid-Training Can LLMs Reliably Self-Report Adversarial Prefills, and How?Adaptive Turn-Taking for Real-time Multi-Party Voice Agents

Recent events (1)

4arXiv · cs.AI·30h ago·source ↗

Study finds GitHub Copilot dialogue accuracy low for HIPAA compliance NFR assessment despite high developer agreement

A controlled study with 49 programmers using GitHub Copilot to assess 148 HIPAA-derived non-functional requirements (NFRs) against a real codebase finds that developers tend to agree with LLM assessments, but accuracy against expert ground truth is low. The paper evaluates multi-turn dialogue quality across requirement satisfaction, reasoning, and code localization dimensions. User satisfaction modeling reveals that longer responses and more information-providing turns hurt satisfaction, while proactive interactions help. The work highlights a gap in current LLM evaluation benchmarks, which focus on functional correctness and single-turn accuracy rather than multi-turn NFR assessment.

Evaluation and Benchmarking Enterprise Deployment Patterns Microsoft iTrust Accuracy and Satisfaction in Multi-Turn LLM Dialogues for NFR Assessment +1 more