4arXiv cs.LG (Machine Learning)·
SLORR: Stateless in-training low-rank regularization for compressible neural networks
Researchers introduce SLORR, a framework for in-training low-rank regularization that avoids SVD computation, architectural modifications, and stateful caching. Two variants based on Hoyer sparsity and nuclear norm are evaluated on ImageNet-1K (ResNet-50, ViT-B/16, ViT-L/16) and LLM pretraining at 135M and 560M parameter scales. SLORR-trained models compress more effectively than unregularized baselines while adding under 8% training overhead for vision models and under 1% for LLM pretraining, making post-training compression more viable.
Related guides (2)
Related events (8)
SVD-Surgeon: Training-free optimal singular value compensation for LLM compression
SVD-Surgeon applies the Optimal Brain Surgeon (OBS) framework to singular-value decomposition-based LLM compression, computing closed-form updates to retained singular values that compensate for pruned ones to second order in the model loss. The method is training-free and modular, designed to layer on top of existing SVD compressors. Applied to SVD-LLM on OPT and LLaMA 2-7B, it improves the perplexity-compression trade-off without retraining.
RELEX: Extrapolating LLM RLVR Training via Rank-1 Parameter Trajectories
This paper demonstrates that RLVR weight update trajectories are extremely low-rank and near-linearly predictable, with a rank-1 approximation capturing most downstream performance gains. The authors propose RELEX, a compute-efficient method that observes a short training window, estimates the rank-1 subspace, and extrapolates future checkpoints via linear regression—requiring no additional training. Evaluated on Qwen2.5-Math-1.5B, Qwen3-4B-Base, and Qwen3-8B-Base, RELEX matches or exceeds full RLVR performance using as few as 15% of training steps, and can extrapolate up to 10–20× beyond the observed prefix. The authors attribute the method's effectiveness to a denoising effect from rank-1 projection that discards stochastic optimization noise.
Predictor-gated bank-wise sparsity recipe for dense-to-sparse LLM upcycling from Qwen2.5-8B
A new arXiv preprint introduces a continual training recipe to convert dense LLMs into channel-sparse models without post-hoc pruning. Starting from a Qwen2.5-8B checkpoint, the method uses a low-rank predictor to gate FFN channel routing, achieving 4x sparsity in FFN intermediate activations via a bank-wise top-k rule at 32K context. The routing module is trained on the main language modeling path, making the resulting sparsity hardware-oriented rather than approximate. The authors also identify and patch a layer-local long-context failure mode on the RULER-CWE benchmark.
Using LoRA for Efficient Stable Diffusion Fine-Tuning
This Hugging Face blog post explains how Low-Rank Adaptation (LoRA) can be applied to fine-tune Stable Diffusion models efficiently. LoRA reduces the number of trainable parameters by decomposing weight updates into low-rank matrices, enabling fine-tuning on consumer hardware with significantly less memory. The post covers practical implementation details using the diffusers library.
Super Weights in LLMs are not selectively trainable despite high importance
A new arXiv paper investigates Super Weights — individual LLM parameters whose removal catastrophically degrades performance — and finds that their apparent importance does not translate into trainability. Training Super Weights in isolation (100 to 8,192 parameters) collapses accuracy to random-guessing on OLMo-1B and OLMo-7B, while training an equal number of randomly chosen parameters in the same layers improves over baseline. LoRA, which applies structured low-rank updates across entire layers, succeeds with only 0.16% of parameters, and constraining LoRA updates at Super Weight coordinates yields no benefit. The findings challenge the assumption that parameter importance implies parameter trainability and suggest effective fine-tuning requires structured decompositions over full layers rather than targeted sparse updates.
Sparsity regularizers improve interpretability of Top-k sparse autoencoders for vision models
A new arXiv preprint proposes two sparsity regularizers compatible with Top-k sparse autoencoders (SAEs), a standard tool for mechanistic interpretability of vision foundation models. The regularizers — an ℓ1 penalty on off-support units and a scale-invariant ℓ1/ℓ2-ratio penalty — are applied before Top-k selection and consistently improve monosemanticity without degrading reconstruction quality across two datasets and three vision models. The central finding is that hard architectural sparsity and soft regularization are complementary, addressing known limitations of fixed-budget Top-k SAEs such as overfitting to training k values.
HiReLC: Hierarchical Reinforcement Learning Framework for Joint Neural Network Pruning and Quantization
Researchers introduce HiReLC, a hierarchical ensemble-RL framework that automates joint quantization and structured pruning of deep neural networks. The system uses two-level agents — low-level agents selecting per-kernel compression configurations and high-level agents coordinating global budget allocation via Fisher Information-based sensitivity estimates. Experiments on Vision Transformers and CNNs achieve 5.99–6.72× parameter-storage compression with accuracy drops of 0.55–5.62% in most settings. The controller is architecture-agnostic, using a surrogate MLP and active learning loop to reduce policy evaluation cost.
GaLore: Advancing Large Model Training on Consumer-grade Hardware
GaLore (Gradient Low-Rank Projection) is a memory-efficient training technique that reduces optimizer state memory by projecting gradients into a low-rank subspace during training, enabling large model training on consumer-grade hardware. The Hugging Face blog post covers integration of GaLore into the transformers and peft ecosystems. Unlike LoRA, GaLore applies low-rank projection to the full training process rather than constraining weight updates, allowing full-parameter learning with reduced memory footprint. This makes training models like LLaMA-7B feasible on single consumer GPUs.