Scaling limit theory of the Random Language Model reveals condensation transition and language statistics
A new arXiv preprint develops a quantitative theory of the Random Language Model (RLM), an ensemble of stochastic context-free grammars, in a scaling limit where grammar size and temperature are jointly tuned. The authors identify a condensation phase transition at a critical parameter value and derive explicit scaling laws for entropy, rule diversity, and related observables across distinct regimes. The work claims to resolve prior ambiguities about thermodynamic transitions in language models and offers a unified framework connecting generative grammar statistics to universal properties of natural language and LLM behavior.
Related guides (1)
Related events (8)
Shannon Scaling Law: A Noisy-Channel Framework for LLM Capacity and Non-Monotonic Training Phenomena
Researchers propose the Shannon Scaling Law, a theoretical framework that models LLM training as information transmission over a noisy channel using the Shannon-Hartley theorem. By mapping model parameters to channel bandwidth and training tokens to signal power, the framework introduces a fundamental SNR-based capacity limit that explains non-monotonic phenomena like catastrophic overtraining and quantization-induced degradation that classical power-law scaling laws cannot capture. Validated on Pythia and OLMo2 under Gaussian noise, quantization, and fine-tuning perturbations, the law achieves strong R² scores and successfully extrapolates from 6.9B to 12B parameter models trained on up to 307B tokens. The framework outperforms both classical and perturbation-aware scaling laws, predicting U-shaped performance degradation when SNR is insufficient.
Language Generation in the Limit with Bounded Memory: Characterization via Sperner's Theorem
This paper studies language generation in the limit under bounded memory constraints, extending classical learning theory to the generation setting. The authors characterize when memoryless generation is possible, derive minimax density bounds using Sperner's theorem and symmetric chain decompositions, and show that adaptively chosen memory outperforms sliding-window memory. They also revisit incremental identification in the limit, finding that exact identification fails for collections of three or more languages but an approximate relaxation is achievable for all finite collections.
Scaling Laws for Neural Language Models
OpenAI published foundational research establishing empirical scaling laws for neural language models, showing that model performance scales predictably with compute, data, and parameters. The work demonstrated power-law relationships between these factors and loss, providing a principled framework for allocating training resources. This paper became a cornerstone of modern large language model development strategy.
Recursive Language Models Offer Path To Dramatically Expand Beyond the Context Window
MIT researchers Alex L. Zhang, Tim Kraska, and Omar Khattab propose Recursive Language Models (RLMs), a framework that offloads long-context processing to an external Python REPL environment, allowing models to programmatically fetch and manage text chunks via code generation. The root model spawns submodel instances to handle subtasks, aggregating their outputs recursively. Evaluated on benchmarks requiring reasoning over documents up to 11 million tokens, RLMs substantially outperform both base models and competing agentic strategies such as retrieval and summarization agents. For example, RLM-GPT-5 achieved 91.3% on BrowseComp+ versus GPT-5's inability to produce an answer, and ~50% accuracy on OOLONG-PAIRS at 1 million tokens versus near-zero for baseline approaches.
Large Language Gibbs: MCMC-based structured probabilistic inference using LLM conditionals
Researchers propose Large Language Gibbs, a structured inference scheme that uses an LLM's conditional token distributions as transition operators in a Gibbs sampling (MCMC) loop, iteratively resampling individual variables rather than generating outputs in a single autoregressive pass. The approach targets order-dependent biases in standard generation and aims to produce a stationary distribution reflecting a coherent compromise across all local conditionals. It is evaluated on synthetic distributions, consistent reasoning tasks, and Bayesian structure learning, showing MCMC-based inference is a practical alternative to one-pass generation for structured probabilistic tasks.
Space-Efficient Language Generation in the Limit: Poly-Space Algorithms with Bounded Hallucination Gap
A new arXiv preprint introduces a resource-aware theory of language generation in the limit, studying learners that must produce hallucination-free hypothesis languages from adversarial positive streams under memory constraints. The authors focus on DFA-recognizable language classes and prove a streaming algorithm using poly(s,k) space that converges with a bounded generation gap, complemented by a near-matching lower bound via communication complexity reduction. The results reveal a sharp phase transition between polynomial-space generation and exponential-space exact identification, providing theoretical grounding for memory-bounded language generation.
Predictable Confabulations: Factual Recall by LLMs Scales with Model Size and Topic Frequency
This paper establishes a quantitative scaling law linking LLM factual recall to both model parameter count and topic frequency in training data, evaluated across 38 models on 8,900+ scholarly references. Recall quality follows a sigmoid function in the log-linear combination of these two variables, explaining 60% of variance across 16 dense models from four families and 74-94% within individual families. The authors propose a superposition-inspired mechanism where recall is gated by a signal-to-noise ratio: concept frequency provides signal and model capacity sets the noise floor. This provides a predictive framework for understanding and anticipating LLM confabulation patterns.
Rosetta Neurons follow sublinear power-law scaling with model size, becoming more monosemantic at scale
A new arXiv paper investigates how neuron populations evolve with scale in both language models (up to 30B parameters) and vision models (up to 5B parameters), focusing on 'Rosetta Neurons' — neurons with similar activation patterns across independently trained models. The authors find Rosetta Neurons grow in absolute count but shrink as a fraction of total neurons, and exhibit a 'Neuron Polarization Effect' where they become increasingly monosemantic while non-Rosetta neurons remain less selective. An analytical model explains the sublinear power-law scaling, and the paper demonstrates practical utility via a targeted data-filtering case study for continued pretraining. The results extend scaling laws to neuron-level interpretability structure, linking model size to systematic changes in universality and specialization.