Towards Detecting Inconsistencies in End-to-end Generated TODs

CSP-based pipeline for detecting hallucinations and inconsistencies in LLM-generated task-oriented dialogues

A new arXiv preprint proposes framing task-oriented dialogue (TOD) consistency checking as a Constraint Satisfaction Problem (CSP), where dialogue segments are variables and domain knowledge adherence forms the constraints. The pipeline identifies variables in a generated dialogue, applies a CSP solver to find valid assignments, and flags inconsistencies such as hallucinated entities (e.g., non-existent restaurants). The authors report high accuracy on inconsistency detection and provide analysis of failure modes in end-to-end LLM-generated TODs.