5arXiv cs.LG (Machine Learning)·
X³-OPD: Cross-modal on-policy distillation transfers reasoning from text LLMs to audio-language models
X³-OPD is a new training framework that distills reasoning capabilities from a powerful text-based teacher model into an audio-language student model via on-policy alignment. The approach generates reasoning trajectories conditioned on the student's own acoustic perception while the teacher provides token-level guidance from matched textual inputs. A three-tier symmetric corpus covers speech-rendered text reasoning, audio-event reasoning, and paralinguistic spoken-dialogue reasoning. Evaluations on MMSU, MMAU, BIG Bench Audio, and MMAR show substantial improvements in audio-grounded reasoning and chain-of-thought quality.
Related guides (2)
Related events (8)
Purified OPSD fixes on-policy self-distillation failures in long chain-of-thought reasoning models
A new arXiv preprint identifies why on-policy self-distillation (OPSD) consistently degrades long chain-of-thought reasoning models: the teacher's supervision signal is dominated by reference-induced shortcuts rather than question-conditioned, transferable corrections. The authors propose a two-step fix using a reference-only teacher to isolate the non-transferable component and pointwise mutual information (PMI) to construct a cleaner distillation target. Experiments across four long-CoT models on two datasets show consistent improvements over both the base model and standard OPSD while preserving reflective reasoning behavior.
On-Policy Delta Distillation improves reasoning transfer via teacher-base model difference signal
Researchers from NAVER AI introduce On-Policy Delta Distillation (OPD²), a new post-training method that replaces direct imitation of a teacher model's output distribution with a 'delta signal' — the difference between the teacher and its pre-instruction-tuning base model. This delta signal isolates the reasoning capability changes induced by instruction tuning, providing a more targeted supervision signal for student models. Experiments across math, science, and code-reasoning benchmarks show OPD² consistently outperforms conventional on-policy distillation with shorter post-training periods.
AudioDER: Deduplication-enhanced reasoning dataset for post-training large audio-language models
Researchers introduce AudioDER, a ~191k-sample post-training dataset for Large Audio-Language Models (LALMs) built via an acoustic similarity-based deduplication pipeline to reduce redundancy and improve corpus diversity. Each sample pairs an audio clip with a multiple-choice question, answer candidates, a caption, and a chain-of-thought rationale generated by Qwen3-30B. Post-training Qwen2-Audio-7B-Instruct on AudioDER yields consistent gains on audio reasoning benchmarks including MMAU-mini, MMSU, and MMAR. The work addresses a data quality gap in audio-language training rather than proposing a new model architecture.
MOPD: Multi-Teacher On-Policy Distillation for integrating multiple RL-trained capabilities in LLMs
Researchers propose Multi-teacher On-Policy Distillation (MOPD), a post-training paradigm that first trains domain-specialized RL teacher models, then distills them into a student model using on-policy rollouts to eliminate exposure bias. Evaluated on Qwen3-30B-A3B, MOPD outperforms Mix-RL, Cascade RL, Off-Policy Finetune, and Param-Merge baselines while preserving nearly all per-domain capability. The method has been deployed in production for MiMo-V2-Flash, an industrial-scale frontier model, validating its practical applicability. The approach also enables parallel, decoupled development of domain teachers, reducing cross-domain interference in multi-capability post-training.
Relay-OPD addresses prefix failure in on-policy distillation via teacher-student trajectory handoff
Researchers introduce Relay On-Policy Distillation (Relay-OPD), a training method that addresses 'prefix failure' in on-policy knowledge distillation, where student models compound early reasoning errors throughout a trajectory. The approach detects divergence points where teacher and student continuations asymmetrically diverge, then briefly hands generation to the teacher to produce a corrective 'relay leg' before the student resumes. Evaluated on eight mathematical reasoning benchmarks using Qwen3-4B-Instruct-2507 as teacher and Qwen3-0.6B/1.7B as students, Relay-OPD outperforms standard OPD by +5.73% and the strongest baseline FastOPD by +1.49% on average for the 1.7B model, while also reducing training trajectory length by over 50%.
Vision-OPD: On-Policy Self-Distillation for Fine-Grained Visual Understanding in MLLMs
Vision-OPD addresses a 'regional-to-global perception gap' in multimodal LLMs, where models answer fine-grained visual questions more accurately when given cropped evidence regions than full images. The method instantiates a crop-conditioned teacher and full-image-conditioned student from the same MLLM, minimizing token-level divergence along on-policy rollouts to transfer regional perception to the full-image policy. This self-distillation requires no external teacher models, ground-truth labels, reward verifiers, or inference-time tools. Benchmarks show competitive or superior performance against larger open-source, closed-source, and agentic 'Thinking-with-Images' models.
OPCoD: On-Policy Co-Distillation for Mutual LLM Improvement via Peer Feedback
Researchers introduce On-Policy Co-Distillation (OPCoD), a training framework where two LLMs, each stronger in a different domain, iteratively tutor each other using on-policy rollouts and peer feedback. The method uses cognizance-based gating to control when feedback is given and feedback anchoring to ground it in the problem context. On Science Q&A tasks, OPCoD achieves Pareto improvement for both models across all evaluated domain pairs, outperforming one-way distillation and single-model fine-tuning baselines.
BIRD: Bootstrapped Iterative Self-Reasoning Distillation reduces LLM chain-of-thought length while improving accuracy
Researchers introduce BIRD (Bootstrapped Iterative Self-Reasoning Distillation), a two-stage method for compressing chain-of-thought reasoning in large language models without sacrificing accuracy. The approach first fine-tunes a model on brevity-instructed correct traces to warm-start the rollout distribution, then applies on-policy reverse-KL distillation against a concise self-teacher. On Qwen3-8B, BIRD improves MATH-500 accuracy from 86.2% to 92.0% while cutting average response length from 3,099 to 1,115 tokens, outperforming cold-start on-policy distillation baselines.