4arXiv cs.LG (Machine Learning)·
Mixed-sign spectral regularization via negative-shifted gradient descent for overparameterized linear regression
A new arXiv preprint introduces negative-shifted gradient descent as a method for mixed-sign spectral regularization in overparameterized linear regression, escaping structural limitations of the negative-ridge endpoint. The authors identify a Marchenko-Pastur barrier in a Gaussian spike-plus-flat model and prove that early-stopped paths improve on all admissible endpoints by a polynomial factor in risk under explicit conditions. The main theorem handles general high-effective-rank tails and recovers all head scales simultaneously, with technical control via localized Duhamel integrals and a finite-grid hold-out inequality for validation-selected algorithms.
Related guides (1)
Related events (8)
Open problem paper questions whether AdamW converges under heavy-tailed gradient noise
A preprint from arXiv frames as an open problem whether AdamW, the dominant optimizer for LLM pretraining, can achieve rigorous convergence guarantees under heavy-tailed stochastic gradient noise. The authors note that sign-based optimizers like Lion and Muon already have sharp heavy-tailed convergence rates, while AdamW's second-moment accumulator may create a fundamental obstruction by hiding large gradients. The paper proves a positive weighted-metric benchmark and introduces a corridor lower-bound mechanism to characterize the potential failure mode.
Complexity bounds for learning projected gradient descent solver iterates via k-neighborhood data augmentation
A new arXiv preprint derives Rademacher complexity-based generalization bounds for learning to predict intermediate iterates of projected gradient descent solvers applied to box-constrained quadratic programs. The authors propose a k-neighborhood data collection strategy that augments converged-solution datasets with intermediate solver states, increasing training data without additional solver runs. The work connects to GLENS, a data-efficient global search method, and frames the approach within the Dynamic Data-Driven Application Systems (DDDAS) paradigm for tightening data-model-optimization loops.
DRPO: Smooth divergence regularization replaces hard masking in LLM RL training
A new arXiv preprint proposes Divergence Regularized Policy Optimization (DRPO), a method that replaces the hard trust-region mask used in DPPO with a smooth advantage-weighted quadratic regularizer on policy shift. The approach addresses a known weakness in PPO and GRPO where importance ratios poorly proxy distributional shift in long-tailed vocabularies, and in DPPO where gradient signals are discarded rather than corrected at trust-region boundaries. Experiments across model scales, architectures, and precision settings show improved stability and efficiency in LLM RL post-training.
Signed-Permutation Gauge Theory for RMSNorm Transformers Improves Coordinate Transport
A new arXiv preprint formalizes the residual-stream gauge symmetry of transformer architectures, showing that RMSNorm models have a signed-permutation gauge group B_d = S_d ⋉ {±1}^d rather than the permutation-only S_d of LayerNorm models. The authors introduce sign-marginalized Hungarian matching and demonstrate that coordinate-preserving transport along fine-tuning trajectories recovers 91.1% of cross-run coordinates versus 60.3% for endpoint matching. Practical consequences include dramatically improved sparse autoencoder reconstruction (NMSE 0.004 vs 1.08), preserved steering vector effects, and correct AdamW optimizer state transfer — with implications for mechanistic interpretability, model merging, and activation engineering.
Theoretical analysis of generalization scaling laws in quadratic two-layer neural networks
A new arXiv preprint derives explicit characterizations of generalization error as a joint function of model width, sample count, and regularization in a quadratic two-layer network with structured data. The analysis reveals a phase diagram with distinct scaling regimes governed by data-dependent power laws tied to the spectral structure of the target function. The work extends scaling law theory beyond fixed-feature or infinite-width regimes by operating in a finite-sample, feature-learning setting, and characterizes interpolation threshold transitions.
Radial suppression of hidden representations accelerates grokking by up to 6x on algorithmic tasks
A new arXiv preprint presents a geometric analysis of the memorization-to-generalization delay (grokking) in neural networks, attributing it to radial inflation of hidden representations under cross-entropy optimization. The authors formalize a radial-angular decomposition of activation-space dynamics and derive three testable propositions about the effects of penalizing radial growth. Empirically, a single-hyperparameter norm penalty that constrains activations to a sqrt(d)-radius hypersphere accelerates grokking up to 6x on modular arithmetic tasks across MLPs and Transformers, and halves training steps for a 10M-parameter nanoGPT on 3-digit addition.
Kolmogorov Regression lifts diffusion policies to Cameron-Martin space for robust long-horizon control
Researchers introduce a backward Kolmogorov equation framework that reformulates diffusion policy training as a deterministic boundary-value PDE problem in Cameron-Martin space, replacing stochastic score matching. The approach uses a precision-weighted Cameron-Martin loss and a Kolmogorov residual as an inference-time failure detector, yielding convergence guarantees tied to kernel effective rank rather than action dimension. Validation on the PushT manipulation benchmark shows 17% improvement in episode reward and 67.6% reduction in inter-step drift; a 6-station manufacturing scheduling task shows 28.4% lower RMSE than LSTM baselines and 96% reduction in deadlock events via Hamilton-Jacobi reachability certification.
Probabilistic Smoothing with Ratio-Monotone Transforms for Global Optimization
This paper proposes a generalized probabilistic smoothing framework for global optimization that replaces Gaussian kernels with flexible symmetric unimodal kernels combined with monotonic ratio-based transformations. The authors prove that the smoothed objective preserves the global maximizer and that stationary points concentrate near the true optimum under large amplification, without requiring a decreasing smoothing schedule. Explicit complexity bounds for stochastic gradient ascent are derived, and a leave-one-out baseline is shown to provably reduce variance. Experiments on high-dimensional benchmarks and black-box adversarial attacks demonstrate improved robustness over existing methods.