6arXiv cs.LG (Machine Learning)·
ISO: Isospectral Optimization framework for RLVR training efficiency and model merging
Researchers introduce Isospectral Optimization (ISO), a framework that exploits 'spectral inheritance' in RLVR-trained language models — the observation that reward-driven adaptation changes singular frames while preserving base model weight spectra. ISO has two instantiations: ISO-Merger, a data-free method for combining specialist models without gradient updates or on-policy distillation, and ISO-Optimizer, which applies standard optimizers (AdamW, Muon) only to frame variables, achieving equivalent accuracy in roughly 2.7x fewer training steps on Qwen3-8B-Base. The work proposes a principled answer to the underexplored optimization layer between reward signals and weight updates in RLVR pipelines.
Related guides (3)
Related events (8)
ACPO: Adaptive Clip Policy Optimization improves RLVR training for LLM reasoning
A new arXiv preprint provides theoretical analysis of Reinforcement Learning from Verifiable Rewards (RLVR) updates, identifying off-policy degree and gradient expectation as key factors governing update dynamics. The authors show that differences in gradient steps per rollout substantially affect importance sampling ratio distributions and which tokens dominate updates. Based on this analysis, they propose Adaptive Clip Policy Optimization (ACPO), which adjusts clipping boundaries per token group by empirical variance of importance sampling ratios, outperforming DAPO and CISPO baselines on 3B and 7B models across math, tabular QA, and logic benchmarks.
Vector Policy Optimization: Training for Diversity Improves Test-Time Search
Vector Policy Optimization (VPO) is a new RL post-training algorithm for LLMs that replaces the scalar reward paradigm with vector-valued rewards, explicitly training models to produce diverse solution sets that specialize across different reward trade-offs. VPO is designed as a near-drop-in replacement for the GRPO advantage estimator and targets inference-scaling search procedures like AlphaEvolve. Across four tasks, VPO matches or outperforms scalar RL baselines on pass@k and best@k metrics, with advantages growing as search budget increases, and unlocks evolutionary search problems that GRPO-trained models cannot solve. The paper argues that diversity-optimized post-training may need to become the default as inference-time search becomes standard.
RELEX: Extrapolating LLM RLVR Training via Rank-1 Parameter Trajectories
This paper demonstrates that RLVR weight update trajectories are extremely low-rank and near-linearly predictable, with a rank-1 approximation capturing most downstream performance gains. The authors propose RELEX, a compute-efficient method that observes a short training window, estimates the rank-1 subspace, and extrapolates future checkpoints via linear regression—requiring no additional training. Evaluated on Qwen2.5-Math-1.5B, Qwen3-4B-Base, and Qwen3-8B-Base, RELEX matches or exceeds full RLVR performance using as few as 15% of training steps, and can extrapolate up to 10–20× beyond the observed prefix. The authors attribute the method's effectiveness to a denoising effect from rank-1 projection that discards stochastic optimization noise.
GGRO: Gradient-Guided Reward Optimization for inference-time LLM alignment
Researchers introduce Gradient-Guided Reward Optimization (GGRO), an inference-time alignment method that uses gradient signals from a reward model to inject 'nudging tokens' at high-uncertainty decoding steps, rather than relying on sampling-intensive re-ranking approaches like Best-of-N. The method monitors token-level entropy to detect distribution drift and steers generation trajectories directly, claiming improved robustness to reward hacking with minimal computational overhead. Experiments show gains across safety, helpfulness, and reasoning benchmarks compared to standard inference-time alignment baselines.
GSPO: Group Sequence Policy Optimization for Scalable RL Training of Language Models
Qwen researchers introduce Group Sequence Policy Optimization (GSPO), a new RL algorithm designed to address severe training instability and model collapse observed in existing methods like GRPO during extended training runs. The core motivation is enabling stable RL scaling for language models to improve reasoning and problem-solving capabilities with increased compute. The paper targets a known bottleneck in post-training pipelines where instability prevents further performance gains.
FORCE: Efficient RL fine-tuning for Vision-Language-Action models via value-calibrated warm-up and self-distillation
Researchers introduce FORCE, a 3-stage reinforcement learning fine-tuning framework for Vision-Language-Action (VLA) models that addresses sample inefficiency caused by unstable Q-functions and low-quality exploration data. The framework uses a Value-Calibrated Warm-Up phase followed by Q-function-filtered policy updates, eliminating the need for costly human interventions during training. Evaluated on simulation and real-world robotic tasks, FORCE achieves a 79% absolute improvement in task success rates, outperforms prior RL methods by 10%, and accelerates training by 32.5%.
SCOPE-RL framework densifies reward signals in RLVR to improve reasoning accuracy and token efficiency
SCOPE-RL introduces a two-stage reinforcement learning framework that addresses sparse reward limitations in RLVR by adding prefix-decomposed sub-question rewards before success and correctness-gated process-shape rewards after success. Applied to Qwen3-8B-Instruct on math reasoning datasets, the method improves average accuracy by up to 11.2 percentage points and reduces reasoning tokens by up to 27.1% over outcome-only GRPO. The gains generalize across GSPO and a smaller Qwen3-0.6B model, suggesting reward-signal densification is broadly complementary to existing RLVR advances. Code and data are publicly released.
SLORR: Stateless in-training low-rank regularization for compressible neural networks
Researchers introduce SLORR, a framework for in-training low-rank regularization that avoids SVD computation, architectural modifications, and stateful caching. Two variants based on Hoyer sparsity and nuclear norm are evaluated on ImageNet-1K (ResNet-50, ViT-B/16, ViT-L/16) and LLM pretraining at 135M and 560M parameter scales. SLORR-trained models compress more effectively than unregularized baselines while adding under 8% training overhead for vision models and under 1% for LLM pretraining, making post-training compression more viable.