The Count Is There, but Misaligned: Understanding and Correcting Counting Failures in VLMs

Probing VLM activations reveals counting knowledge is encoded but misaligned with outputs

Researchers find that vision-language models often internally encode correct object counts even when they verbalize wrong answers, diagnosing the failure as a misalignment between internal representations and output directions rather than missing knowledge. Using nonlinear probes and SVCCA analysis across four VLMs and five counting datasets, they identify a partially shared but misaligned activation subspace between ground-truth and model-output probes. A causal steering intervention confirms the diagnosis, and a detector-guided self-correction method that re-prompts only on predicted failures improves counting accuracy by up to 15.6 absolute percentage points at inference time with no parameter updates.