3arXiv cs.LG (Machine Learning)·
Physics-aware policy distillation makes deep RL agents interpretable via decision tree surrogates
A new arXiv preprint presents a policy distillation framework that transfers a high-performance TD3 reinforcement learning agent into a shallow decision tree surrogate for interpretable continuous control, demonstrated on the Inverted Pendulum benchmark. The method uses physics-aware features and 'Noisy Oracle Rollouts' to match teacher performance while providing global and local interpretability. A key finding is a fundamental trade-off: discretizing control induces Bang-Bang actuation and a bimodal limit cycle, though BIBO stability is preserved. The work targets safety-critical deployment contexts where regulatory compliance and human-agent trust require explainability.
Related guides (1)
Related events (8)
Direct On-Policy Distillation transfers RL policy shifts from weak to strong models
Researchers propose Direct-OPD (Direct On-Policy Distillation), a method for transferring the policy shift induced by reinforcement learning on a small model to a larger target model, bypassing the need to run expensive RL rollouts on the stronger model. The approach uses the log-ratio between a post-RL teacher and its pre-RL reference as a dense implicit reward signal applied to the student's own on-policy states. Empirically, Direct-OPD improves Qwen3-1.7B from 48.3% to 62.4% on AIME 2024 in 4 hours on 8 A100 GPUs, outperforming step-matched direct RL. The method addresses a key scalability bottleneck in post-training as frontier models grow larger.
Agency-transferring technique improves RL policy training by bootstrapping from baseline policies
A new arXiv paper proposes a model-free reinforcement learning method that embeds an existing suboptimal baseline policy into training via an arbitration mechanism, progressively transferring control from the baseline to a trainable neural network. The approach yields high goal-reaching rates from the start of training and produces a standalone policy that outperforms the baseline without requiring it at inference time. Theoretical bounds on goal-reaching probability are derived, and empirical results on continuous-control benchmarks show competitive or superior returns compared to existing methods.
On-policy self-distillation reduces output diversity compared to RL, paper shows
A new arXiv paper analyzes on-policy self-distillation, where a single model serves as both teacher and student conditioned on correct demonstrations, finding it achieves strong pass@1 accuracy but at the cost of reduced rollout diversity and flattened pass@k curves. The authors trace this to compounding biases: teacher feedback is channeled through the model's own biases, amplifying probability mass on already-dominant modes rather than preserving diversity across equally correct solutions. Theoretical analysis shows the self-distillation policy tilts the base distribution by pointwise conditional mutual information, unlike ideal on-policy RL which preserves probability ratios among correct rollouts. Empirical results on graph path-finding and science QA benchmarks confirm self-distilled models match RL on average performance but fail on out-of-distribution settings requiring diverse strategies.
PEARL: Physics-enhanced reinforcement learning for high-dimensional optimal control
Researchers introduce PEARL (Physics-EnhAnced Reinforcement Learning), a hybrid paradigm combining RL with classical optimal control techniques for high-dimensional dynamical systems. The method uses an actor-adjoint algorithm that leverages automatic differentiation and adjoint-based sensitivity computation to dramatically reduce environment interactions and mitigate long-term gradient instabilities. Demonstrated on parametric navigation problems in unsteady flows, PEARL outperforms standard RL baselines while generalizing across scenarios and scaling to high-dimensional state/action spaces without requiring dimensionality reduction.
Unified study of multi-turn long-horizon planning across pre-training and post-training stages via agentic distillation
A new arXiv preprint introduces a controlled multi-turn environment to systematically study how long-horizon planning ability is acquired, shaped, and integrated in foundation model agents across three stages: pre-training data design, post-training via GRPO and on-policy distillation (OPD), and multi-teacher on-policy distillation (MOPD). Key findings include that explicit world model construction via chain-of-thought state transition modeling improves generalization, suboptimal trajectories severely degrade performance over long horizons, and OPD outperforms GRPO in low-quality and long-horizon settings. The multi-teacher distillation analysis reveals that compatible planning patterns enable cross-environment generalization while conflicting patterns cause interference.
MOPD: Multi-Teacher On-Policy Distillation for integrating multiple RL-trained capabilities in LLMs
Researchers propose Multi-teacher On-Policy Distillation (MOPD), a post-training paradigm that first trains domain-specialized RL teacher models, then distills them into a student model using on-policy rollouts to eliminate exposure bias. Evaluated on Qwen3-30B-A3B, MOPD outperforms Mix-RL, Cascade RL, Off-Policy Finetune, and Param-Merge baselines while preserving nearly all per-domain capability. The method has been deployed in production for MiMo-V2-Flash, an industrial-scale frontier model, validating its practical applicability. The approach also enables parallel, decoupled development of domain teachers, reducing cross-domain interference in multi-capability post-training.
On-Policy Delta Distillation improves reasoning transfer via teacher-base model difference signal
Researchers from NAVER AI introduce On-Policy Delta Distillation (OPD²), a new post-training method that replaces direct imitation of a teacher model's output distribution with a 'delta signal' — the difference between the teacher and its pre-instruction-tuning base model. This delta signal isolates the reasoning capability changes induced by instruction tuning, providing a more targeted supervision signal for student models. Experiments across math, science, and code-reasoning benchmarks show OPD² consistently outperforms conventional on-policy distillation with shorter post-training periods.
ACPO: Adaptive Clip Policy Optimization improves RLVR training for LLM reasoning
A new arXiv preprint provides theoretical analysis of Reinforcement Learning from Verifiable Rewards (RLVR) updates, identifying off-policy degree and gradient expectation as key factors governing update dynamics. The authors show that differences in gradient steps per rollout substantially affect importance sampling ratio distributions and which tokens dominate updates. Based on this analysis, they propose Adaptive Clip Policy Optimization (ACPO), which adjusts clipping boundaries per token group by empirical variance of importance sampling ratios, outperforming DAPO and CISPO baselines on 3B and 7B models across math, tabular QA, and logic benchmarks.