Vision-Default, Prior-Override: Causal Mechanisms of Perception-Knowledge Conflict in Vision-Language Models

Causal circuit analysis reveals how vision-language models resolve perception-knowledge conflicts

A new arXiv preprint uses activation patching and ablation studies to identify the mechanistic basis of perception-knowledge conflict in vision-language models across three VLM families. The authors find that visual grounding is the default behavior, while knowledge-grounded responses depend on a small set of attention heads (2.5–4.8% of total) concentrated in the network's second half. Ablating these heads flips knowledge-grounded predictions to visually grounded ones in 68–96% of cases while barely affecting visually grounded predictions, revealing an asymmetric causal structure. The identified heads decompose into routing heads and writing heads, and the circuit is consistent across model families and scales.