technique

rank-1 approximation

techniqueactiverank-1-approximation-4e0748ba·1 events·first seen 26d ago

Aliases: rank-1 approximation

Co-occurring entities

RLVR Qwen3-8B-Base Qwen3-4B-Base Qwen2.5-Math-PRM Reinforcement Learning with Verifiable Rewards Wei Zhepei Alibaba Qwen Team RELEX

More like this (12)

likelihood approximation Universal Approximation Theorem adaptive-rank instantiation reranking Approximate DP Top-k Accuracy Rank-to-Distill Expert Token Rank Rank-Constrained Subspace Learning (RCSL)low-rank subspace projection quantization scalar quantization

Recent events (1)

7arXiv · cs.CL·26d ago·source ↗

RELEX: Extrapolating LLM RLVR Training via Rank-1 Parameter Trajectories

This paper demonstrates that RLVR weight update trajectories are extremely low-rank and near-linearly predictable, with a rank-1 approximation capturing most downstream performance gains. The authors propose RELEX, a compute-efficient method that observes a short training window, estimates the rank-1 subspace, and extrapolates future checkpoints via linear regression—requiring no additional training. Evaluated on Qwen2.5-Math-1.5B, Qwen3-4B-Base, and Qwen3-8B-Base, RELEX matches or exceeds full RLVR performance using as few as 15% of training steps, and can extrapolate up to 10–20× beyond the observed prefix. The authors attribute the method's effectiveness to a denoising effect from rank-1 projection that discards stochastic optimization noise.

Training Infrastructure Frontier Model Releases RLVR Qwen3-8B-Base Qwen3-4B-Base +8 more