Entity · paper

Weak-to-Strong Generalization via Direct On-Policy Distillation

paperactiveprovisionalweak-to-strong-generalization-via-direct-on-policy-distillation-9754854b·1 events·first seen 6d ago

Aliases: Weak-to-Strong Generalization via Direct On-Policy Distillation

Co-occurring entities

on-policy distillation AIME 2026 Qwen3-1.7B Reinforcement Learning with Verifiable Rewards

More like this (12)

on-policy distillation On-Policy Distillation (OPD)Dense Supervision, Sparse Updates: On the Sparsity and Geometry of On-Policy Distillation Multi-Teacher On-Policy Distillation weak-to-strong generalization On-Policy Co-Distillation on-policy self-distillation Weak-to-Strong Distillation Learning from the Self-future: On-policy Self-distillation for dLLMs Canonical-Context On-Policy Distillation (CCOPD)On-Policy Self-Distillation with Sampled Demonstrations Reduces Output Diversity Generalized Distillation

Recent events (1)

6arXiv · cs.CL·6d ago·source ↗

Direct On-Policy Distillation transfers RL policy shifts from weak to strong models

Researchers propose Direct-OPD (Direct On-Policy Distillation), a method for transferring the policy shift induced by reinforcement learning on a small model to a larger target model, bypassing the need to run expensive RL rollouts on the stronger model. The approach uses the log-ratio between a post-RL teacher and its pre-RL reference as a dense implicit reward signal applied to the student's own on-policy states. Empirically, Direct-OPD improves Qwen3-1.7B from 48.3% to 62.4% on AIME 2024 in 4 hours on 8 A100 GPUs, outperforming step-matched direct RL. The method addresses a key scalability bottleneck in post-training as frontier models grow larger.

Training Infrastructure Frontier Model Releases on-policy distillation AIME 2026 Weak-to-Strong Generalization via Direct On-Policy Distillation +5 more