Byte-Prefix Marginalization

Byte-Prefix Marginalization enables cross-tokenizer on-policy distillation between heterogeneous LLM families

A new arXiv preprint introduces Byte-Prefix Marginalization (BPM), a technique for distilling knowledge from teacher LLMs into a student model when the two use different tokenizers. BPM re-expresses the teacher's next-token distribution in a shared byte space, preserving probability mass and producing a vocabulary-complete alignment target. Evaluated with Qwen3-32B, GLM-Z1-9B-0414, and MiniMax-M2.7 as teachers, BPM improves six-benchmark average scores by 3.7–6.6 points over the strongest cross-tokenizer baselines on math and programming tasks. The method addresses a practical bottleneck in consolidating complementary open-weight models into compact students.