5arXiv cs.LG (Machine Learning)·
IPE method challenges Q-function pretraining convention in online RL fine-tuning
A new arXiv paper systematically examines whether pretraining Q-functions on offline data actually helps when fine-tuning a pretrained policy via online RL, finding that naive Q-function pretraining often provides little benefit over random initialization. The authors attribute this to a fundamental mismatch between the pretrained Q-function's target policy and the policy that online fine-tuning converges to. They propose Initialization via Policy Ensemble (IPE), which trains multiple diverse policies and pools their rollouts to bootstrap Q-function learning, achieving an average 1.26x improvement over naive Q-function pretraining on continuous control benchmarks.
Related guides (1)
Related events (8)
Active policy selection for offline-to-online RL fine-tuning under limited interaction budgets
A new arXiv preprint introduces an active policy selection framework for offline-to-online reinforcement learning (O2O-RL), addressing the problem of choosing which candidate policy to fine-tune when online interactions are scarce or risky. The method uses upper-confidence bounds derived from locally linear performance forecasts to balance exploration (policy evaluation) against exploitation (fine-tuning). Experiments show consistent outperformance over existing O2O-RL baselines. The authors claim this is the first work to formally address active policy selection in the O2O-RL setting.
Agency-transferring technique improves RL policy training by bootstrapping from baseline policies
A new arXiv paper proposes a model-free reinforcement learning method that embeds an existing suboptimal baseline policy into training via an arbitration mechanism, progressively transferring control from the baseline to a trainable neural network. The approach yields high goal-reaching rates from the start of training and produces a standalone policy that outperforms the baseline without requiring it at inference time. Theoretical bounds on goal-reaching probability are derived, and empirical results on continuous-control benchmarks show competitive or superior returns compared to existing methods.
FORE: Fitted occupancy-ratio evaluation for offline RL without Bellman completeness
A new arXiv preprint introduces Fitted Occupancy-Ratio Evaluation (FORE), a fixed-point method for estimating discounted occupancy ratios in offline reinforcement learning and off-policy evaluation. FORE uses an adjoint Bellman recursion and projects onto a log-ratio class in KL divergence, requiring only realizability of the occupancy ratio rather than Bellman completeness or projected-operator stability. The authors establish finite-sample regret bounds and show convergence in KL up to approximation and statistical error, supporting value estimation via reward reweighting, occupancy-weighted fitted Q-evaluation, and doubly robust estimation. The result identifies occupancy-ratio realizability as a sufficient condition for offline policy evaluation, relaxing a standard assumption in the field.
FORCE: Efficient RL fine-tuning for Vision-Language-Action models via value-calibrated warm-up and self-distillation
Researchers introduce FORCE, a 3-stage reinforcement learning fine-tuning framework for Vision-Language-Action (VLA) models that addresses sample inefficiency caused by unstable Q-functions and low-quality exploration data. The framework uses a Value-Calibrated Warm-Up phase followed by Q-function-filtered policy updates, eliminating the need for costly human interventions during training. Evaluated on simulation and real-world robotic tasks, FORCE achieves a 79% absolute improvement in task success rates, outperforms prior RL methods by 10%, and accelerates training by 32.5%.
q0: Hyper-Epoch Pretraining turns multi-epoch budgets into diverse model populations for better generalization
A new arXiv preprint introduces hyper-epoch pretraining (q0), a framework that reframes multi-epoch training as exploration of a model population rather than refinement of a single model. The approach uses three primitives—cyclic schedules with anti-correlated learning rate and weight decay, chain distillation, and a learned prior for inference-time weighting—to achieve lower validation loss than single-model training. On a 1.8B-parameter model trained on FineWeb, q0 matches a 256-epoch ensemble baseline using only ~56 epochs (~4.6× fewer), with cumulative ~12.9× data efficiency under the Slowrun setting. The work directly addresses the emerging regime where compute scales faster than high-quality data supply.
ACPO: Adaptive Clip Policy Optimization improves RLVR training for LLM reasoning
A new arXiv preprint provides theoretical analysis of Reinforcement Learning from Verifiable Rewards (RLVR) updates, identifying off-policy degree and gradient expectation as key factors governing update dynamics. The authors show that differences in gradient steps per rollout substantially affect importance sampling ratio distributions and which tokens dominate updates. Based on this analysis, they propose Adaptive Clip Policy Optimization (ACPO), which adjusts clipping boundaries per token group by empirical variance of importance sampling ratios, outperforming DAPO and CISPO baselines on 3B and 7B models across math, tabular QA, and logic benchmarks.
High offline conservatism in DPO amplifies reward hacking during online adaptation, study finds
A new arXiv paper challenges the conventional wisdom that conservative offline training (via DPO with high β) provides a safer foundation for online RL adaptation. Experiments with Qwen3-14B show that higher offline conservatism monotonically increases reward hacking damage (Goodhart gap) during online adaptation, with Spearman ρ=1.0 across conditions. The mechanistic explanation is a three-link chain: high-β DPO compresses policy entropy, reducing response diversity and concentrating outputs in a narrow reward-model region, while paradoxically increasing ensemble disagreement that gets exploited during online optimization. The authors identify a practical optimal conservatism level β* and argue the field needs calibrated rather than maximal conservatism.
Direct On-Policy Distillation transfers RL policy shifts from weak to strong models
Researchers propose Direct-OPD (Direct On-Policy Distillation), a method for transferring the policy shift induced by reinforcement learning on a small model to a larger target model, bypassing the need to run expensive RL rollouts on the stronger model. The approach uses the log-ratio between a post-RL teacher and its pre-RL reference as a dense implicit reward signal applied to the student's own on-policy states. Empirically, Direct-OPD improves Qwen3-1.7B from 48.3% to 62.4% on AIME 2024 in 4 hours on 8 A100 GPUs, outperforming step-matched direct RL. The method addresses a key scalability bottleneck in post-training as frontier models grow larger.