Entity · paper

Neural Collapse Is Forbidden: Information Floors in Language Models

paperactiveprovisionalneural-collapse-is-forbidden-information-floors-in-language-models-833204ab·1 events·first seen 6d ago

Aliases: Neural Collapse Is Forbidden: Information Floors in Language Models

More like this (12)

Language Modeling Loss Linear representations of grammaticality in neural language models Knowledgeless Language Models: Suppressing Parametric Recall for Evidence-Grounded Language Modeling Scaling Laws for Neural Language Models Knowledge-Less Language Models Scaling limit of the Random Language Model On the Limits of Prompt-Conditioned Language Models as General-Purpose Learners When Does Combining Language Models Help? A Co-Failure Ceiling on Routing, Voting, and Mixture-of-Agents Across 67 Frontier Models A Learning-Rate-Gated Failure of GRPO in a Small Language and Vision-Language Model Web Agent Transformer Language Models Tapered Language Models Reasoning Language Models

Recent events (1)

6arXiv · cs.CL·6d ago·source ↗

Neural Collapse Is Forbidden: Information Floors in Language Model Representations

A new arXiv preprint argues that within-class variance in language model representations is not incomplete neural collapse but rather allocated information storage obeying a measurable law. Across 14 models spanning a 100x parameter range, macro-category structure accounts for only 4-12% of representational variance while within-token context carries 79-91%. The authors prove a theoretical floor on within-category dispersion proportional to the conditional mutual information I(token; context | category), and show this law holds across models, partitions, and over pretraining dynamics, including voiding a family of simplex ETF claims.

Evaluation and Benchmarking Neural Collapse Is Forbidden: Information Floors in Language Models