Entity · technique

Initialization via Policy Ensemble

techniqueactiveprovisionalinitialization-via-policy-ensemble-5680e04b·1 events·first seen 2d ago

Aliases: Initialization via Policy Ensemble

Co-occurring entities

Do You Really Need to Pretrain Q-Functions for Online RL Fine-Tuning?

More like this (12)

Preference Coordinated Multi-agent Policy Optimization Role-Aware Policy Optimization Privileged On-Policy Exploration APPO: Agentic Procedural Policy Optimization Graph-as-Policy Evolved Policy Gradients Patch Policy Pass the Baton: Trajectory-Relayed On-Policy Distillation LedgerAgent: Structured State for Policy-Adherent Tool-Calling Agents Routing-based On-Policy Distillation Ensemble Diversity Optimization An Agency-Transferring Model-Free Policy Enhancement Technique

Recent events (1)

5arXiv · cs.LG·2d ago·source ↗

IPE method challenges Q-function pretraining convention in online RL fine-tuning

A new arXiv paper systematically examines whether pretraining Q-functions on offline data actually helps when fine-tuning a pretrained policy via online RL, finding that naive Q-function pretraining often provides little benefit over random initialization. The authors attribute this to a fundamental mismatch between the pretrained Q-function's target policy and the policy that online fine-tuning converges to. They propose Initialization via Policy Ensemble (IPE), which trains multiple diverse policies and pools their rollouts to bootstrap Q-function learning, achieving an average 1.26x improvement over naive Q-function pretraining on continuous control benchmarks.

Alignment and RLHF Do You Really Need to Pretrain Q-Functions for Online RL Fine-Tuning?Initialization via Policy Ensemble