4arXiv cs.LG (Machine Learning)·
1-Lipschitz neural networks on Hadamard manifolds via Busemann gradient flows
A new arXiv preprint constructs and analyzes 1-Lipschitz neural networks on Hadamard manifolds, including hyperbolic spaces and the manifold of symmetric positive definite (SPD) matrices. The architecture uses Busemann functions and gradient-descent-type layers that are geometry-preserving and quasi-α-firmly nonexpansive. Experiments demonstrate robust classification on the Poincaré disk and improved covariance reconstruction on the SPD manifold compared to Log-Euclidean and data-only baselines.
Related guides (1)
Related events (8)
Expressivity Limits of Congruence-Based Architectures for Neural Networks on Positive-Definite Matrices
This paper analyzes neural network architectures designed to classify symmetric positive-definite (SPD) matrices, focusing on congruence-like layers as used in SPDNet. The authors prove that imposing semi-orthogonality constraints on weight matrices limits expressivity, causing deep architectures to collapse to single-hidden-layer equivalents due to spectral diversity loss—a consequence of Poincaré's separation theorem. The work also compares Riemannian classifiers for compatibility with congruence-based feature maps.
Unified framework for Riemannian deep learning: modules, architectures, and geometries
A doctoral thesis proposes a unified framework for deep learning on manifold-valued representations, addressing limitations of existing approaches tied to specific manifolds or relying on Euclidean approximations. The work generalizes batch normalization to Lie groups and gyrogroups, extends multinomial logistic regression to SPD and general Riemannian manifolds, and introduces new architectures for hyperbolic space and correlation matrices. It also contributes adaptive, computationally efficient Riemannian metrics including learnable Log-Euclidean and Cholesky-based geometries, validated across vision, signal processing, graph learning, and genomics tasks.
Conservation laws from data symmetry in neural network gradient-flow training
A new arXiv preprint investigates whether intrinsic symmetries in training data produce conserved quantities during gradient-flow training of neural networks. The authors prove that for analytic, non-polynomial loss functions, data symmetries generically do not induce additional integrals of motion, but for MSE loss, data augmentation can yield extra conserved quantities. They introduce a framework of 'tensorizable networks'—architectures including linear, polynomial, and Lightning Attention networks—where parameter and input dependence can be separated via an intermediate representation.
Signed-Permutation Gauge Theory for RMSNorm Transformers Improves Coordinate Transport
A new arXiv preprint formalizes the residual-stream gauge symmetry of transformer architectures, showing that RMSNorm models have a signed-permutation gauge group B_d = S_d ⋉ {±1}^d rather than the permutation-only S_d of LayerNorm models. The authors introduce sign-marginalized Hungarian matching and demonstrate that coordinate-preserving transport along fine-tuning trajectories recovers 91.1% of cross-run coordinates versus 60.3% for endpoint matching. Practical consequences include dramatically improved sparse autoencoder reconstruction (NMSE 0.004 vs 1.08), preserved steering vector effects, and correct AdamW optimizer state transfer — with implications for mechanistic interpretability, model merging, and activation engineering.
PTL-Diffusion: Diffusion framework with periodic terminal laws for manifold-aware generation
PTL-Diffusion is a new diffusion modeling framework that replaces the standard single Gaussian terminal distribution with a periodic family of Gaussian terminal laws, embedding phase structure directly into the forward noising dynamics rather than only in the denoising network. The authors derive closed-form forward marginals and reverse posteriors for a periodically forced Ornstein-Uhlenbeck process, enabling standard noise-prediction training. Experiments on torus, cylinder, and face datasets show improvements in manifold-level distributional matching over DDPM baselines. The work is a proof-of-concept motivating structured terminal reference laws as a direction for geometry-aware generative modeling.
Theoretical analysis of generalization scaling laws in quadratic two-layer neural networks
A new arXiv preprint derives explicit characterizations of generalization error as a joint function of model width, sample count, and regularization in a quadratic two-layer network with structured data. The analysis reveals a phase diagram with distinct scaling regimes governed by data-dependent power laws tied to the spectral structure of the target function. The work extends scaling law theory beyond fixed-feature or infinite-width regimes by operating in a finite-sample, feature-learning setting, and characterizes interpolation threshold transitions.
Open problem paper questions whether AdamW converges under heavy-tailed gradient noise
A preprint from arXiv frames as an open problem whether AdamW, the dominant optimizer for LLM pretraining, can achieve rigorous convergence guarantees under heavy-tailed stochastic gradient noise. The authors note that sign-based optimizers like Lion and Muon already have sharp heavy-tailed convergence rates, while AdamW's second-moment accumulator may create a fundamental obstruction by hiding large gradients. The paper proves a positive weighted-metric benchmark and introduces a corridor lower-bound mechanism to characterize the potential failure mode.
Perturbation Theory for Spherical Hellinger-Kantorovich Flows with Differential Privacy Guarantees
This paper develops a perturbation theory for Spherical Hellinger-Kantorovich (SHK) gradient flows, which couple transport and reaction dynamics and coincide with birth-death Langevin dynamics. The authors derive dimension-free bounds on log-likelihood ratios and Rényi/KL divergences when two potentials differ, quantifying how perturbations propagate over time. These results are applied to differential privacy: the likelihood-ratio control yields explicit Pure-DP guarantees for SHK-based samplers implementing the exponential mechanism, while KL bounds provide Approximate-DP certificates. A utility bound is also derived that separates intrinsic exponential-mechanism suboptimality from finite-time sampling error.