5arXiv cs.LG (Machine Learning)·
Unified framework for generalization and sketching across variable-size inputs via random sampling maps
A new arXiv preprint introduces a theoretical framework for understanding how ML models trained on small inputs generalize to larger, unseen input sizes — covering sequences, graphs, point clouds, and tensors. The approach uses random sampling maps (generalizing sampling with replacement, random binning, and species sampling) to compare inputs of different sizes and derive explicit generalization and sketching rates. The framework applies to transformers, graph neural networks, and moment polynomials, among other architectures. This is a foundational theoretical contribution addressing out-of-distribution generalization across input dimensionality.
Related guides (1)
Related events (8)
Theoretical analysis of generalization scaling laws in quadratic two-layer neural networks
A new arXiv preprint derives explicit characterizations of generalization error as a joint function of model width, sample count, and regularization in a quadratic two-layer network with structured data. The analysis reveals a phase diagram with distinct scaling regimes governed by data-dependent power laws tied to the spectral structure of the target function. The work extends scaling law theory beyond fixed-feature or infinite-width regimes by operating in a finite-sample, feature-learning setting, and characterizes interpolation threshold transitions.
Minimax framework establishes fundamental statistical limits for transport map estimation in generative models
A new arXiv preprint formalizes the task of estimating valid transport maps (used in diffusion models, normalizing flows, and flow matching) within a minimax statistical framework. The key result is that under standard stability assumptions from optimal transport theory, estimating any valid transport map is as statistically hard as estimating the optimal transport map itself. The authors also show that when those stability assumptions fail, alternative transport maps can be learned substantially more accurately than the OT map, clarifying when targeting sub-optimal maps yields real statistical advantages. The work provides theoretical sample complexity lower bounds applicable to a broad class of modern generative modeling methods.
Systematic framework for selecting trajectories in data augmentation evaluated across five strategies
A thesis-derived arXiv preprint proposes a framework for evaluating five trajectory selection strategies—Outlierness, Diversity, Representativeness, Uncertainty, and Random—for data augmentation in spatio-temporal ML tasks. The study tests these strategies across four datasets spanning animal behavior, maritime, and urban traffic domains using linear and non-linear models with Optuna-based hyperparameter optimization. Key findings show systematic strategies (especially Outlierness and Uncertainty) outperform random selection in sparse datasets but can degrade performance in dense, high-quality datasets, with UMAP visualization confirming topological effects.
Large deviation analysis shows most interpolating classifiers share the same generalization performance
A new arXiv preprint establishes a large deviation principle characterizing the generalization performance of interpolating linear classifiers in the overparameterized regime (n/d → α, small α). The key result is a concentration phenomenon: all but an exponentially small fraction of interpolators achieve approximately the same generalization error, determined by a unique rate-function maximizer. Empirically, gradient descent and a natural linear program both outperform this typical interpolator, providing theoretical grounding for benign overfitting in overparameterized models.
Dynamic short convolutions yield 1.33–1.60× compute advantage over standard Transformers
A new arXiv preprint introduces dynamic short convolutions as an architectural primitive for Transformers, using input-dependent filters to combine locality bias with increased expressivity. Experiments across 150M–2B parameter language models show consistent perplexity improvements over standard Transformers and static convolution variants, with scaling-law fits indicating a 1.33× compute advantage when applied to key/query/value vectors and 1.60× when added after every linear layer. The technique also improves linear RNNs (Mamba-2, Gated DeltaNet) and mixture-of-experts architectures, with custom Triton kernels making training practical.
Graph Sparse Sampling (GSS) breaks exponential horizon dependence in continuous MDP planning
Researchers introduce Graph Sparse Sampling (GSS), an online planning algorithm for continuous Markov Decision Processes that shares sampled futures across candidate decisions rather than branching separately per action, eliminating the tree structure of methods like MCTS. The approach exposes large GPU-friendly batches and uses heuristics to focus computation, with finite-sample performance guarantees showing polynomial (rather than exponential) dependence on planning horizon under suitable conditions. Empirical results on continuous-control tasks show GSS substantially outperforms tree-based planners at long horizons. The work formalizes when shared-future graph planning can avoid the curse of the horizon that afflicts sparse sampling trees.
Radial suppression of hidden representations accelerates grokking by up to 6x on algorithmic tasks
A new arXiv preprint presents a geometric analysis of the memorization-to-generalization delay (grokking) in neural networks, attributing it to radial inflation of hidden representations under cross-entropy optimization. The authors formalize a radial-angular decomposition of activation-space dynamics and derive three testable propositions about the effects of penalizing radial growth. Empirically, a single-hyperparameter norm penalty that constrains activations to a sqrt(d)-radius hypersphere accelerates grokking up to 6x on modular arithmetic tasks across MLPs and Transformers, and halves training steps for a 10M-parameter nanoGPT on 3-digit addition.
Hyperfitting Explained: Terminal Geometric Expansion in Final Transformer Layers Drives Diversity Gains
This paper investigates the 'hyperfitting' phenomenon—where fine-tuning LLMs to near-zero loss on small datasets improves open-ended generation and reduces repetition—and demonstrates it is mechanistically distinct from temperature scaling. Entropy-matched control experiments falsify both the temperature-equivalence and static vocabulary reweighting hypotheses, instead localizing the effect to a 'Terminal Expansion' in the final transformer block where feature-space dimensionality expands by ~80.8 dimensions, enabling promotion of deep-tail tokens via context-dependent rank reordering. The authors introduce Late-Stage LoRA, a targeted fine-tuning strategy updating only the final 5 layers, achieving robust generation with minimal parameter updates.