spectral entropy

Activation dispersion signals entity familiarity in Bielik models, but doesn't prevent hallucination

Researchers probe four Polish Bielik LLMs (1.5B–11B parameters) to test whether internal activations reveal entity familiarity before any answer token is generated. Using unsupervised dispersion measures over post-SwiGLU MLP activations, they achieve AUROC 0.95–1.00 in separating known from fabricated entities—a signal that is already at ceiling at 1.5B parameters. Despite this internal awareness, the models almost never abstain: only 2 refusals and 1 hedge were found across 2,520 answers. The paper distinguishes entity familiarity (a representational phenomenon stable across scale) from factual reliability (which scales sharply with model size), with implications for hallucination detection and calibration research.