Entity · paper

Do You Really Need to Pretrain Q-Functions for Online RL Fine-Tuning?

paperactiveprovisionaldo-you-really-need-to-pretrain-q-functions-for-online-rl-fine-tuning--930f70c9·1 events·first seen 2d ago

Aliases: Do You Really Need to Pretrain Q-Functions for Online RL Fine-Tuning?

Co-occurring entities

Initialization via Policy Ensemble

More like this (12)

ExpRL: Exploratory RL for LLM Mid-Training Hierarchical Advantage Weighting for Online RL Fine-Tuning of VLAs from Sparse Episode Outcomes KL-regularized RL Is One Layer Enough? Training A Single Transformer Layer Can Match Full-Parameter RL Training FORCE: Efficient VLA Reinforcement Fine-Tuning via Value-Calibrated Warm-up and Self-Distillation Active Offline-to-Online Reinforcement Learning Generalization in offline RL: The structure is more important than the amount of pessimism Double Q-learning Pretraining Recurrent Networks without Recurrence Improving LLM-Generated Process Model Quality Through Reinforcement Learning: The Role of Reward Function Design Physics-EnhAnced Reinforcement Learning temporally ordered pre-training

Recent events (1)

5arXiv · cs.LG·2d ago·source ↗

IPE method challenges Q-function pretraining convention in online RL fine-tuning

A new arXiv paper systematically examines whether pretraining Q-functions on offline data actually helps when fine-tuning a pretrained policy via online RL, finding that naive Q-function pretraining often provides little benefit over random initialization. The authors attribute this to a fundamental mismatch between the pretrained Q-function's target policy and the policy that online fine-tuning converges to. They propose Initialization via Policy Ensemble (IPE), which trains multiple diverse policies and pools their rollouts to bootstrap Q-function learning, achieving an average 1.26x improvement over naive Q-function pretraining on continuous control benchmarks.

Alignment and RLHF Do You Really Need to Pretrain Q-Functions for Online RL Fine-Tuning?Initialization via Policy Ensemble