6arXiv cs.CL (Computation and Language)·
Activation dispersion signals entity familiarity in Bielik models, but doesn't prevent hallucination
Researchers probe four Polish Bielik LLMs (1.5B–11B parameters) to test whether internal activations reveal entity familiarity before any answer token is generated. Using unsupervised dispersion measures over post-SwiGLU MLP activations, they achieve AUROC 0.95–1.00 in separating known from fabricated entities—a signal that is already at ceiling at 1.5B parameters. Despite this internal awareness, the models almost never abstain: only 2 refusals and 1 hedge were found across 2,520 answers. The paper distinguishes entity familiarity (a representational phenomenon stable across scale) from factual reliability (which scales sharply with model size), with implications for hallucination detection and calibration research.
Related guides (3)
Related events (8)
MMBench2 paper: hallucination in world models is predictable and preventable via coverage signals
Researchers introduce MMBench2, a 427-hour, 210-task dataset for visual world modeling, and train a 350M-parameter world model to study hallucination in generative world models. The paper identifies three distinct hallucination modes (perceptual, action-marginalized, scene-diverging) and develops lightweight signals that predict where models will fail. A coverage-aware sampling technique and curiosity-reward-based data collection enable efficient finetuning to unseen environments with as few as 50 real trajectories. The central finding is that world model hallucination is fundamentally a data coverage problem, with the same signals serving both detection and mitigation.
CHAIR: Supervised hallucination detection via internal logit analysis across LLM layers
A new arXiv preprint introduces CHAIR (Classifier of Hallucination As ImproveR), a supervised framework that detects hallucinations by extracting statistical features (max, min, mean, std, slope) from token logits across all layers of an LLM. Evaluated on TruthfulQA and MMLU, CHAIR shows improved detection accuracy especially in zero-shot settings. The authors argue the approach also points toward richer internal representations for designing adaptive decoding strategies that reduce hallucinations.
BenHalluEval: Multi-Task Hallucination Evaluation Framework for Bengali LLMs
BenHalluEval introduces the first systematic hallucination benchmark for Bengali, covering four tasks (generative QA, code-mixed QA, summarization, reasoning) with 12,000 hallucinated candidates generated via GPT-5.4 across twelve hallucination types. Seven LLMs are evaluated under a dual-track protocol separating false-positive rate on ground-truth instances from hallucination detection rate on hallucinated candidates. The proposed BenHalluScore metric reveals substantial variation (7.72%–55.42%) across models and tasks, and chain-of-thought prompting is found to shift response distributions without consistently improving hallucination discrimination. The work highlights gaps in low-resource language hallucination evaluation and critiques single-track and prompting-only evaluation approaches.
PhantomBench: Large-scale benchmark reveals staggering hallucination rates on non-existent concepts
PhantomBench is a new benchmark comprising over 60,000 non-existent terms and entities derived from real concepts, designed to test whether language models can recognize the limits of their knowledge. Evaluating 21 models of various types and sizes, the authors find hallucination rates as high as 86.7% on average, with even frontier models failing to abstain when inputs presuppose the existence of fabricated concepts. The benchmark also serves as a proxy for studying model behavior on rare real concepts, and includes a pipeline for scalable generation of custom non-existent concept sets.
Why Language Models Hallucinate
OpenAI published research explaining the mechanisms behind language model hallucination. The work connects improved evaluation methods to enhanced AI reliability, honesty, and safety. The body is sparse on technical detail, but the framing positions this as foundational research relevant to alignment and deployment trust.
Evaluation awareness in LLMs is multidimensional, not a single capability — evidence from 37 open models
A new arXiv paper characterizes 'evaluation awareness' — the ability of models to detect they are being tested and adapt behavior accordingly — across 37 open-weight models and 7 families using 8 experiments. Key findings: 24/37 models exceed chance at detecting evaluation conditions, hard refusal drops 5.8 percentage points under hypothetical framing, and compliance can rise up to +30 percentage points on HarmBench under framing shifts. Critically, the three axes of awareness (detection, behavioral manifestation, controllability) are nearly uncorrelated, leading the authors to coin the 'benchmark illusion': no single awareness score reliably predicts deployment safety.
Confidence and Calibration of Activation Oracles for Reliable Interpretation of Language Model Internals
This paper investigates uncertainty quantification (UQ) for activation oracles—systems that make LLM internal activations human-legible—by evaluating 6 confidence estimation methods across 6,000 samples per oracle. The authors find that bootstrap mode frequency achieves the best calibration (ECE 5.7% vs. 25.5% for log-probability baseline on Qwen3-8B), while the log-prob baseline remains useful as a cheap triage signal. Experiments vary verbalizer and context prompts across two Qwen3 model sizes. Code and a patched trainer are released publicly.
LegalHalluLens: Typed hallucination auditing and calibrated multi-agent debate for legal AI
Researchers introduce LegalHalluLens, an auditing framework for hallucination in legal AI systems, evaluated across 510 contracts and 249,252 clause-level instances from the CUAD dataset. The framework introduces typed hallucination profiles across four claim categories (numeric, temporal, obligation/entitlement, factual) and a Risk Direction Index (RDI) that distinguishes omission from invention errors. A calibrated multi-agent debate pipeline reduces fabricated detections by 45% using a 4B-parameter model competitive with commercial APIs. The work reveals that aggregate hallucination rates (~52%) mask a 38-40 percentage-point gap between claim types and that two systems with identical aggregate rates can have opposite risk profiles.