ViRL39K

Visual Contrastive Self-Distillation (VCSD) improves multimodal LLM training without external teachers

Researchers propose Visual Contrastive Self-Distillation (VCSD), a training method for vision-language models that creates an on-policy self-distillation signal purely through input conditioning — specifically by contrasting token distributions from an EMA teacher conditioned on the original image versus a content-erased control. The approach eliminates the need for external teachers, privileged answers, visual evidence signals, or reasoning traces. Evaluated on Qwen3-VL and Qwen3.5 models using the ViRL39K dataset, VCSD consistently outperforms matched on-policy self-distillation baselines, with aggregate benchmark gains of up to ~3.75 percentage points at the 8B scale.