4arXiv cs.AI (Artificial Intelligence)·
Active policy selection for offline-to-online RL fine-tuning under limited interaction budgets
A new arXiv preprint introduces an active policy selection framework for offline-to-online reinforcement learning (O2O-RL), addressing the problem of choosing which candidate policy to fine-tune when online interactions are scarce or risky. The method uses upper-confidence bounds derived from locally linear performance forecasts to balance exploration (policy evaluation) against exploitation (fine-tuning). Experiments show consistent outperformance over existing O2O-RL baselines. The authors claim this is the first work to formally address active policy selection in the O2O-RL setting.
Related guides (1)
Related events (8)
FORE: Fitted occupancy-ratio evaluation for offline RL without Bellman completeness
A new arXiv preprint introduces Fitted Occupancy-Ratio Evaluation (FORE), a fixed-point method for estimating discounted occupancy ratios in offline reinforcement learning and off-policy evaluation. FORE uses an adjoint Bellman recursion and projects onto a log-ratio class in KL divergence, requiring only realizability of the occupancy ratio rather than Bellman completeness or projected-operator stability. The authors establish finite-sample regret bounds and show convergence in KL up to approximation and statistical error, supporting value estimation via reward reweighting, occupancy-weighted fitted Q-evaluation, and doubly robust estimation. The result identifies occupancy-ratio realizability as a sufficient condition for offline policy evaluation, relaxing a standard assumption in the field.
ACPO: Adaptive Clip Policy Optimization improves RLVR training for LLM reasoning
A new arXiv preprint provides theoretical analysis of Reinforcement Learning from Verifiable Rewards (RLVR) updates, identifying off-policy degree and gradient expectation as key factors governing update dynamics. The authors show that differences in gradient steps per rollout substantially affect importance sampling ratio distributions and which tokens dominate updates. Based on this analysis, they propose Adaptive Clip Policy Optimization (ACPO), which adjusts clipping boundaries per token group by empirical variance of importance sampling ratios, outperforming DAPO and CISPO baselines on 3B and 7B models across math, tabular QA, and logic benchmarks.
Agency-transferring technique improves RL policy training by bootstrapping from baseline policies
A new arXiv paper proposes a model-free reinforcement learning method that embeds an existing suboptimal baseline policy into training via an arbitration mechanism, progressively transferring control from the baseline to a trainable neural network. The approach yields high goal-reaching rates from the start of training and produces a standalone policy that outperforms the baseline without requiring it at inference time. Theoretical bounds on goal-reaching probability are derived, and empirical results on continuous-control benchmarks show competitive or superior returns compared to existing methods.
EvoPolicyGym benchmark evaluates autonomous policy evolution by LLM agents in RL environments
A new arXiv preprint introduces EvoPolicyGym, a benchmark for evaluating how LLM-based agents iteratively improve executable policies in compact interactive RL environments under a fixed interaction budget. The benchmark provides trajectory-level diagnostics beyond aggregate scores, distinguishing how agents allocate budget and convert feedback into parametric tuning. GPT-5.5 achieves the strongest aggregate rank score and top-two performance across all 16 environments. The work targets a gap in agent evaluation where iterative policy refinement is conflated with open-ended software engineering progress.
DRIFT: Decoupled Rollouts and Importance-Weighted Fine-Tuning for Efficient Multi-Turn Optimization
DRIFT is a training framework that bridges online RL and offline SFT for multi-turn LLM optimization by exploiting the theoretical equivalence between KL-regularized RL and importance-weighted supervised learning. It decouples rollout generation from policy optimization: trajectories are sampled from a fixed reference policy offline, weighted by return-based importance scores, and used for weighted SFT. Empirically, DRIFT matches or exceeds multi-turn RL baselines while retaining the efficiency and simplicity of standard supervised fine-tuning. Code is publicly released.
HABC: Hierarchical Advantage Weighting for Online RL Fine-Tuning of Vision-Language-Action Policies
Researchers introduce Hierarchical Advantage-Weighted Behavior Cloning (HABC), a method for fine-tuning pretrained Vision-Language-Action (VLA) policies via online RL using only sparse binary episode outcomes. HABC trains separate critic heads for viability and efficiency objectives, combines them via a state-adaptive gate, and applies intervention-aware credit assignment to avoid incorrect supervision across human-intervention boundaries. On three contact-rich bimanual real-robot tasks, HABC improves success rates from SFT baselines of 36%, 44%, and 12% to 92%, 88%, and 38% respectively. The work addresses a fundamental credit assignment problem in robot learning from sparse outcome signals.
PCMA: Learning coordinated agent-specific preferences for multi-objective multi-agent RL
A new arXiv preprint introduces Preference Coordinated Multi-agent Policy Optimization (PCMA), a method for cooperative multi-objective multi-agent reinforcement learning (MOMARL) that learns agent-specific preferences to enable complementary trade-offs across agents. The authors formulate cooperative MOMARL as a team-optimal game and provide a first-order improvement decomposition showing that preference diversity can induce team improvement. Experiments on cooperative MOMA environments and a traffic-control scenario demonstrate improvements in both performance and trade-off coordination.
High offline conservatism in DPO amplifies reward hacking during online adaptation, study finds
A new arXiv paper challenges the conventional wisdom that conservative offline training (via DPO with high β) provides a safer foundation for online RL adaptation. Experiments with Qwen3-14B show that higher offline conservatism monotonically increases reward hacking damage (Goodhart gap) during online adaptation, with Spearman ρ=1.0 across conditions. The mechanistic explanation is a three-link chain: high-β DPO compresses policy entropy, reducing response diversity and concentrating outputs in a narrow reward-model region, while paradoxically increasing ensemble disagreement that gets exploited during online optimization. The authors identify a practical optimal conservatism level β* and argue the field needs calibrated rather than maximal conservatism.