6arXiv cs.CL (Computation and Language)·
CANON: Label-free self-distillation using consensus as dense token-level supervision
Researchers introduce CANON (Consensus-ANchored self-distillatiON), a label-free training method that converts majority-vote consensus from multiple sampled solutions into dense token-level supervision for LLM self-improvement. Unlike prior approaches that use consensus only as a filter, preference signal, or scalar reward, CANON conditions a frozen model snapshot on a consensus-reaching solution to supervise the model's own rollouts at every token. On mathematical and scientific reasoning benchmarks, CANON improves pass@1 by up to 12 points, outperforming label-free RL by 6 points at one-seventh the compute, and approaches performance of methods using gold labels. The method also transfers to held-out benchmarks when trained on pooled unlabeled data.
Related guides (2)
Related events (8)
Canonical-Context On-Policy Distillation (CCOPD) for Multi-Turn LLM Consistency
This paper identifies 'self-anchored drift' as a key failure mode in multi-turn LLMs: when information is revealed incrementally across turns, models produce unsupported assumptions that distort final answers, even when the total evidence is identical to a single-prompt setting. The authors propose Canonical-Context On-Policy Distillation (CCOPD), which trains a student model on incremental multi-turn conversations to match the output distribution of a frozen teacher conditioned on the full clean prompt. Trained only on math conversations, CCOPD achieves a 32% average relative improvement on multi-turn (RAW-SHARDED) tasks and generalizes zero-shot to five out-of-domain task families while preserving single-prompt performance.
Neuron-OPSD: Annotation-free LLM self-distillation guided by internal neuron activations
Researchers introduce Neuron On-Policy Self-Distillation (Neuron-OPSD), a data-centric framework for post-training LLMs without human annotations or real-world interaction feedback. The method uses internal neuron activations to guide training-data selection and teacher context construction, then trains via on-policy distillation from the teacher distribution. Evaluated on specialized-domain benchmarks, Neuron-OPSD improves in-domain performance while preserving cross-domain generalization and avoiding the calibration collapse seen in prior SFT-, GRPO-, and reward-RL-based annotation-free approaches.
Future Confidence Distillation enables pre-solution confidence estimation in LLMs
Researchers introduce 'future confidence distillation,' a method that trains predictors on pre-solution hidden representations using post-solution correctness probes as teacher signals, enabling reliable confidence estimation before answer generation completes. The paper compares pre-solution Feeling-of-Knowing (FOK) and post-solution Judgement-of-Learning (JOL) confidence estimates across frontier and open-source LLMs, finding post-solution estimates are better calibrated and that linear probes recover richer confidence information than models verbalize. Distilled predictors recover much of the post-solution calibration improvement while remaining sample-efficient and domain-transferable. This has practical implications for confidence-aware systems involving retrieval, tool use, and adaptive computation.
Step-aligned critique outperforms GRPO and reference-solution conditioning in self-distillation
A new arXiv paper investigates context design for self-distillation of language models, comparing binary reward (GRPO), reference solutions, and step-by-step critiques aligned to the solver's reasoning trace. Step-aligned critique yields the largest gains, outperforming GRPO by 16.11 points and reference-solution conditioning by 5.27 points on Avg@12. Per-token advantage analysis shows that step-aligned feedback targets only failing tokens, avoiding unnecessary pressure on already-correct reasoning steps. The findings suggest structural alignment between feedback and the model's reasoning trace is a key driver of self-distillation effectiveness.
Rubric-Conditioned Self-Distillation: structured feedback for reasoning model post-training
A new arXiv preprint proposes Rubric-Conditioned Self-Distillation (RCSD), a post-training framework that replaces scalar reward signals and noisy chain-of-thought annotations with structured rubrics for fine-grained credit assignment. The method conditions a teacher model on criterion-level rubrics to provide token-level guidance on the student's own sampled trajectories, avoiding reliance on a single reference rationale. Evaluated on science reasoning benchmarks, RCSD outperforms GRPO by 1.0 points and OPSD by 0.9 points on average.
BIRD: Bootstrapped Iterative Self-Reasoning Distillation reduces LLM chain-of-thought length while improving accuracy
Researchers introduce BIRD (Bootstrapped Iterative Self-Reasoning Distillation), a two-stage method for compressing chain-of-thought reasoning in large language models without sacrificing accuracy. The approach first fine-tunes a model on brevity-instructed correct traces to warm-start the rollout distribution, then applies on-policy reverse-KL distillation against a concise self-teacher. On Qwen3-8B, BIRD improves MATH-500 accuracy from 86.2% to 92.0% while cutting average response length from 3,099 to 1,115 tokens, outperforming cold-start on-policy distillation baselines.
Reinforcement Learning with Metacognitive Feedback (RLMF) improves LLM calibration and uncertainty expression
Researchers introduce Reinforcement Learning with Metacognitive Feedback (RLMF), a training paradigm that refines preference optimization using a model's self-judgments of its own performance quality. The method is applied to faithful calibration — aligning a model's expressed confidence with its intrinsic uncertainty — and achieves state-of-the-art results across diverse tasks while outperforming standard RL by up to 63%. A companion technique, metacognitive data selection, uses similar self-judgments to identify high-value training examples, outperforming naive active learning baselines. The work positions metacognitive performance as a novel and effective RL signal for improving LLM reliability and alignment.
LLM-as-a-Verifier: Training-free verification framework scales along granularity, repetition, and criteria decomposition
Researchers introduce LLM-as-a-Verifier, a general-purpose verification framework that treats verification as a new scaling axis for LLMs, computing continuous scores from token logit distributions rather than discrete judge outputs. The framework scales along three dimensions—score granularity, repeated evaluation, and criteria decomposition—and achieves state-of-the-art results on Terminal-Bench V2 (86.5%), SWE-Bench Verified (78.2%), RoboRewardBench (87.4%), and MedAgentBench (73.3%) without requiring additional training. The authors also demonstrate that the framework's fine-grained signals can serve as dense RL feedback, improving sample efficiency for SAC and GRPO on robotics and math benchmarks, and build a Claude Code extension for monitoring agentic systems.