6arXiv cs.CL (Computation and Language)·
Neural Collapse Is Forbidden: Information Floors in Language Model Representations
A new arXiv preprint argues that within-class variance in language model representations is not incomplete neural collapse but rather allocated information storage obeying a measurable law. Across 14 models spanning a 100x parameter range, macro-category structure accounts for only 4-12% of representational variance while within-token context carries 79-91%. The authors prove a theoretical floor on within-category dispersion proportional to the conditional mutual information I(token; context | category), and show this law holds across models, partitions, and over pretraining dynamics, including voiding a family of simplex ETF claims.
Related guides (1)
Related events (8)
When Does Model Collapse Occur in Structured Interactive Learning?
This paper formalizes model collapse in multi-model interactive learning environments where models are trained on each other's synthetic outputs. The authors represent model interactions as directed graphs and derive necessary and sufficient conditions for collapse based on interaction graph topology. Finite-sample results for linear regression and asymptotic guarantees for general M-estimators are established, supported by numerical experiments.
Aggregate accuracy masks per-example prediction instability from task-irrelevant context in LLMs
A new arXiv paper demonstrates that state-of-the-art LLMs appear robust to task-irrelevant context at the aggregate level, but this stability conceals significant per-example prediction flips. Even semantically meaningless pseudo-words prepended to benchmark questions can shift model predictions on a subset of examples, with gains and losses roughly canceling out in aggregate. The instability is modulated by context type, length, test-time compute, and model development stage, and the affected examples are largely model-specific. The authors argue this reveals 'tail risks' hidden by standard aggregate accuracy metrics, motivating per-example reliability evaluation.
Stable Recovery Manifold hypothesis: catastrophic forgetting as accessibility problem, not information destruction
A new arXiv preprint investigates the geometric structure of recoverability in continual learning using Split CIFAR-100 and a sequentially trained ResNet-18. The authors introduce Recovery Subspace Dimensionality (k_t) and find that recovery dimensionality remains stable across tasks (mean k_t = 8.0) despite substantial representational drift, with principal-angle drift strongly predicting recoverability (r = -0.862). The findings support the Stable Recovery Manifold hypothesis: forgotten knowledge remains compactly decodable, reframing catastrophic forgetting as a manifold-alignment and accessibility problem rather than true information loss.
Semantic vs. Surface Noise in LLM Agents: 68-Cell Measurement Study with Held-Out Validation
This paper documents an empirical phenomenon across 10 LLMs from 7 architecture families: meaning-bearing perturbations (paraphrase, synonym substitution) cause final-answer inconsistency ~19.69 percentage points more often than presentation-level perturbations (formatting, reordering) of comparable severity, across GSM8K, MATH, and HotpotQA benchmarks. The effect is validated on a held-out 11th model (qwen2.5-14B-Instruct) with 1,800 trajectories. Trace-level analysis supports a 'stealth-divergence' picture where semantic perturbations preserve the first action but induce divergence in intermediate reasoning steps, while two prior mechanism claims are explicitly retracted. The study is notable for its honest reporting of stress-test failures and pre-registered replication.
Linear representations of grammaticality found in neural language model internals
A new arXiv preprint investigates whether grammaticality is encoded in the internal representations of neural language models, moving beyond probability-based evaluations. Using mass-mean probing, the authors find that grammatical and ungrammatical sentences are robustly separated in representational space across a wide range of pretrained models, grammatical phenomena, and languages. The encoding is not fully explained by confounds like lexical frequency or plausibility, suggesting grammaticality constitutes a coherent representational dimension. The findings contribute to debates about syntactic knowledge in LLMs and offer a probability-independent framework for evaluating grammatical competence.
Rosetta Neurons follow sublinear power-law scaling with model size, becoming more monosemantic at scale
A new arXiv paper investigates how neuron populations evolve with scale in both language models (up to 30B parameters) and vision models (up to 5B parameters), focusing on 'Rosetta Neurons' — neurons with similar activation patterns across independently trained models. The authors find Rosetta Neurons grow in absolute count but shrink as a fraction of total neurons, and exhibit a 'Neuron Polarization Effect' where they become increasingly monosemantic while non-Rosetta neurons remain less selective. An analytical model explains the sublinear power-law scaling, and the paper demonstrates practical utility via a targeted data-filtering case study for continued pretraining. The results extend scaling laws to neuron-level interpretability structure, linking model size to systematic changes in universality and specialization.
Scaling Laws for Neural Language Models
OpenAI published foundational research establishing empirical scaling laws for neural language models, showing that model performance scales predictably with compute, data, and parameters. The work demonstrated power-law relationships between these factors and loss, providing a principled framework for allocating training resources. This paper became a cornerstone of modern large language model development strategy.
Hyperfitting Explained: Terminal Geometric Expansion in Final Transformer Layers Drives Diversity Gains
This paper investigates the 'hyperfitting' phenomenon—where fine-tuning LLMs to near-zero loss on small datasets improves open-ended generation and reduces repetition—and demonstrates it is mechanistically distinct from temperature scaling. Entropy-matched control experiments falsify both the temperature-equivalence and static vocabulary reweighting hypotheses, instead localizing the effect to a 'Terminal Expansion' in the final transformer block where feature-space dimensionality expands by ~80.8 dimensions, enabling promotion of deep-tail tokens via context-dependent rank reordering. The authors introduce Late-Stage LoRA, a targeted fine-tuning strategy updating only the final 5 layers, achieving robust generation with minimal parameter updates.