4arXiv cs.AI (Artificial Intelligence)·
PAC-ACT: Reinforcement learning post-training framework for Action Chunking Transformer robot policies
PAC-ACT is a reinforcement learning post-training framework that fine-tunes pretrained Action Chunking Transformer (ACT) policies for precision industrial contact manipulation tasks. The method reformulates policy optimization at the chunk level, introduces an actor-critic architecture adapted for ACT, and uses a hybrid behavior-prior constraint to prevent distribution shift during online RL fine-tuning. Experiments on industrial contact benchmarks show significant improvements in task success, contact stability, and force safety — including a 46x reduction in force readings above 60 N on a contour-following task — while preserving low latency and GPU memory usage.
Related guides (1)
Related events (8)
HABC: Hierarchical Advantage Weighting for Online RL Fine-Tuning of Vision-Language-Action Policies
Researchers introduce Hierarchical Advantage-Weighted Behavior Cloning (HABC), a method for fine-tuning pretrained Vision-Language-Action (VLA) policies via online RL using only sparse binary episode outcomes. HABC trains separate critic heads for viability and efficiency objectives, combines them via a state-adaptive gate, and applies intervention-aware credit assignment to avoid incorrect supervision across human-intervention boundaries. On three contact-rich bimanual real-robot tasks, HABC improves success rates from SFT baselines of 36%, 44%, and 12% to 92%, 88%, and 38% respectively. The work addresses a fundamental credit assignment problem in robot learning from sparse outcome signals.
PACT: Hybrid SLM deliberation architecture improves reactive RL policies in unfamiliar environments
Researchers propose PACT (Plan, Align, Commit, Think), a hybrid architecture pairing a fast reactive RL policy with an asynchronous small language model planner for deliberation. The SLM generates and validates candidate action plans via simulation before committing to execution, bypassing the RL policy without retraining. Evaluated on FrozenLake configurations of increasing difficulty, PACT outperforms baselines using only a 2B-parameter SLM, suggesting complementary strengths between deliberative planning and reactive execution.
FORCE: Efficient RL fine-tuning for Vision-Language-Action models via value-calibrated warm-up and self-distillation
Researchers introduce FORCE, a 3-stage reinforcement learning fine-tuning framework for Vision-Language-Action (VLA) models that addresses sample inefficiency caused by unstable Q-functions and low-quality exploration data. The framework uses a Value-Calibrated Warm-Up phase followed by Q-function-filtered policy updates, eliminating the need for costly human interventions during training. Evaluated on simulation and real-world robotic tasks, FORCE achieves a 79% absolute improvement in task success rates, outperforms prior RL methods by 10%, and accelerates training by 32.5%.
Agency-transferring technique improves RL policy training by bootstrapping from baseline policies
A new arXiv paper proposes a model-free reinforcement learning method that embeds an existing suboptimal baseline policy into training via an arbitration mechanism, progressively transferring control from the baseline to a trainable neural network. The approach yields high goal-reaching rates from the start of training and produces a standalone policy that outperforms the baseline without requiring it at inference time. Theoretical bounds on goal-reaching probability are derived, and empirical results on continuous-control benchmarks show competitive or superior returns compared to existing methods.
RECALL: Active continual learning for Vision-Language-Action models via uncertainty-guided recovery data collection
Researchers propose RECALL, an active continual learning paradigm for Vision-Language-Action (VLA) robot models that uses uncertainty-guided data collection to target states where the policy struggles, rather than passively collecting demonstrations after failures. The paper demonstrates improved fine-tuning efficiency over passive imitation learning but identifies catastrophic forgetting as a key challenge when incorporating recovery data. The authors evaluate continual learning mitigations including replay-based data mixing and elastic weight consolidation, characterizing tradeoffs between plasticity and retention in large autoregressive robot policies.
ACPO: Adaptive Clip Policy Optimization improves RLVR training for LLM reasoning
A new arXiv preprint provides theoretical analysis of Reinforcement Learning from Verifiable Rewards (RLVR) updates, identifying off-policy degree and gradient expectation as key factors governing update dynamics. The authors show that differences in gradient steps per rollout substantially affect importance sampling ratio distributions and which tokens dominate updates. Based on this analysis, they propose Adaptive Clip Policy Optimization (ACPO), which adjusts clipping boundaries per token group by empirical variance of importance sampling ratios, outperforming DAPO and CISPO baselines on 3B and 7B models across math, tabular QA, and logic benchmarks.
OpenPipe ART: Agent Reinforcement Trainer for Multi-Step Agents via GRPO
OpenPipe has released ART (Agent Reinforcement Trainer), an open-source Python library for training multi-step agents on real-world tasks using GRPO (Group Relative Policy Optimization). The framework supports multiple model families including Qwen3, GPT-OSS, and Llama. With nearly 10k GitHub stars and 66 gained today, it is gaining notable community traction as a practical RL fine-tuning tool for agentic workflows.
AXPO: Agent Explorative Policy Optimization Addresses Thinking-Acting Gap in Multimodal Agentic Reasoning
This paper identifies a structural asymmetry in agentic reasoning called the 'Thinking-Acting Gap,' where tool use is attempted in only ~30% of rollouts under standard RL training (GRPO), and all-wrong tool-using subgroups suppress learning signals. The authors propose AXPO (Agent eXplorative Policy Optimization), which fixes the thinking prefix and resamples tool calls for all-wrong subgroups, combined with uncertainty-based prefix selection. Evaluated across nine multimodal benchmarks on Qwen3-VL-Thinking at multiple scales, SFT+AXPO outperforms SFT+GRPO by +1.8pp on both Pass@1 and Pass@4 at 8B, with the 8B model surpassing the 32B baseline on Pass@4 using 4× fewer parameters.