LegalHalluLens: Typed hallucination auditing and calibrated multi-agent debate for legal AI
Researchers introduce LegalHalluLens, an auditing framework for hallucination in legal AI systems, evaluated across 510 contracts and 249,252 clause-level instances from the CUAD dataset. The framework introduces typed hallucination profiles across four claim categories (numeric, temporal, obligation/entitlement, factual) and a Risk Direction Index (RDI) that distinguishes omission from invention errors. A calibrated multi-agent debate pipeline reduces fabricated detections by 45% using a 4B-parameter model competitive with commercial APIs. The work reveals that aggregate hallucination rates (~52%) mask a 38-40 percentage-point gap between claim types and that two systems with identical aggregate rates can have opposite risk profiles.
Related guides (3)
Related events (8)
ClinHallu benchmark diagnoses stage-wise hallucinations in medical multimodal LLM reasoning
Researchers from Alibaba DAMO Academy introduce ClinHallu, a benchmark of 7,031 validated instances designed to identify where hallucinations originate within medical MLLM reasoning pipelines. Each instance is annotated with a structured reasoning trace decomposed into Visual Recognition, Knowledge Recall, and Reasoning Integration stages, with stage-replacement interventions to measure the causal impact of correcting each stage. The paper also demonstrates that trace-supervised fine-tuning reduces stage-wise hallucinations, offering both diagnostic and mitigation value for clinical AI systems.
CHAIR: Supervised hallucination detection via internal logit analysis across LLM layers
A new arXiv preprint introduces CHAIR (Classifier of Hallucination As ImproveR), a supervised framework that detects hallucinations by extracting statistical features (max, min, mean, std, slope) from token logits across all layers of an LLM. Evaluated on TruthfulQA and MMLU, CHAIR shows improved detection accuracy especially in zero-shot settings. The authors argue the approach also points toward richer internal representations for designing adaptive decoding strategies that reduce hallucinations.
BenHalluEval: Multi-Task Hallucination Evaluation Framework for Bengali LLMs
BenHalluEval introduces the first systematic hallucination benchmark for Bengali, covering four tasks (generative QA, code-mixed QA, summarization, reasoning) with 12,000 hallucinated candidates generated via GPT-5.4 across twelve hallucination types. Seven LLMs are evaluated under a dual-track protocol separating false-positive rate on ground-truth instances from hallucination detection rate on hallucinated candidates. The proposed BenHalluScore metric reveals substantial variation (7.72%–55.42%) across models and tasks, and chain-of-thought prompting is found to shift response distributions without consistently improving hallucination discrimination. The work highlights gaps in low-resource language hallucination evaluation and critiques single-track and prompting-only evaluation approaches.
Why Language Models Hallucinate
OpenAI published research explaining the mechanisms behind language model hallucination. The work connects improved evaluation methods to enhanced AI reliability, honesty, and safety. The body is sparse on technical detail, but the framing positions this as foundational research relevant to alignment and deployment trust.
The Hallucinations Leaderboard, an Open Effort to Measure Hallucinations in Large Language Models
Hugging Face has launched an open leaderboard specifically designed to benchmark hallucination rates across large language models. The effort aims to standardize evaluation of factual accuracy and confabulation tendencies, filling a gap in existing benchmarks that focus primarily on capability rather than reliability. The leaderboard is positioned as a community-driven, transparent resource for tracking model trustworthiness.
PhantomBench: Large-scale benchmark reveals staggering hallucination rates on non-existent concepts
PhantomBench is a new benchmark comprising over 60,000 non-existent terms and entities derived from real concepts, designed to test whether language models can recognize the limits of their knowledge. Evaluating 21 models of various types and sizes, the authors find hallucination rates as high as 86.7% on average, with even frontier models failing to abstain when inputs presuppose the existence of fabricated concepts. The benchmark also serves as a proxy for studying model behavior on rare real concepts, and includes a pipeline for scalable generation of custom non-existent concept sets.
JANUS benchmark measures goal-conditioned pragmatic distortion in LLMs
Researchers introduce JANUS, a 160-scenario benchmark designed to measure a subtle but dangerous form of LLM deception: selective treatment of true facts to create misleading impressions, rather than outright fabrication. Each scenario provides a fixed fact pool and compares neutral versus goal-directed prompts (e.g., increasing adoption or enrollment), isolating pragmatic distortion from hallucination. Experiments across 12 LLMs reveal consistent goal-conditioned distortions, suggesting current models lack robust safeguards against selectively misleading communication. The benchmark and code are publicly released.
Multi-agent LLM framework for Chinese civil court simulation with five-stage trial procedure
Researchers present a multi-agent LLM framework for simulating Chinese civil court proceedings, organized around a five-stage civil trial procedure with memory modules and statute retrieval. The system targets civil litigation specifically, which is more common and harder to simulate than criminal cases due to flexible claims and remedies. Experiments show reliable judgment outputs with particular strengths in liability allocation, and find that memory quality substantially affects downstream simulation quality. Code and dataset are publicly released.