technique
Pointwise Mutual Information
techniqueactiveprovisional
pointwise-mutual-information-02dcbd30·1 events·first seen 12h agoAliases: Pointwise Mutual Information
Co-occurring entities
More like this (12)
mutual informationKullback-Leibler divergenceVariational Information BottleneckPair Opt-distJensen-Shannon divergenceGeneralized Method of MomentsPair M-distMarker Internal Confidence (MIC)AIR: Adaptive Interleaved Reasoning with Code in MLLMsdifference-in-meansProbabilistic Smoothing with Ratio-Monotone Transformsoutcome indistinguishability
Recent events (1)
Purified OPSD fixes on-policy self-distillation failures in long chain-of-thought reasoning models
A new arXiv preprint identifies why on-policy self-distillation (OPSD) consistently degrades long chain-of-thought reasoning models: the teacher's supervision signal is dominated by reference-induced shortcuts rather than question-conditioned, transferable corrections. The authors propose a two-step fix using a reference-only teacher to isolate the non-transferable component and pointwise mutual information (PMI) to construct a cleaner distillation target. Experiments across four long-CoT models on two datasets show consistent improvements over both the base model and standard OPSD while preserving reflective reasoning behavior.