5arXiv cs.CL (Computation and Language)·
VLMs overestimate common ground in asymmetric dialogue, conflating potential with established shared understanding
A new arXiv paper investigates whether vision-language models can distinguish between what could be shared versus what has actually been established as shared between dialogue participants. Using 13,077 annotated reference expressions from HCRC MapTask dialogues, the authors find that VLMs systematically over-predict alignment when given task-relevant map content—whether presented visually or as text—suggesting the bias stems from static referential cues rather than tracking grounding through dialogue history. The effect is observed most strongly in Qwen3-VL-8B-Instruct and replicated across four additional models from two architecture families, revealing a fundamental limitation in how current VLMs model collaborative dialogue.
Related guides (2)
Related events (8)
Real Images, Worse Judgments: Evaluating VLMs on Concreteness and Imagery
This paper evaluates whether vision-language models (VLMs) benefit from real image context when making lexical judgments about word concreteness and imagery. The authors find that real-image contexts frequently hurt alignment with human ratings, especially when visual evidence is least relevant to the word being judged. Probing and canonical correlation analysis reveal that real images cause representational shifts and increased sensitivity to spurious visual cues. Instructing models to focus on text-only content at inference time partially mitigates this degradation.
VLMs May Not Globally Enhance Human Alignment over LLMs During Natural Reading
This paper compares matched LLM and VLM pairs in a text-only setting to isolate the effect of multimodal training history on human-like language processing. Using whole-cortex fMRI and eye-tracking data from natural reading, the authors find that multimodal pretraining does not confer a uniform global advantage in human alignment. However, VLMs show selective advantages when sentences contain stronger visual semantic content, with converging evidence from both neural and behavioral measures. The findings suggest language-internal representations remain the primary driver of human text processing alignment.
Vision Language Models Explained
A Hugging Face blog post providing a technical overview of vision language models (VLMs), covering their architecture, training approaches, and capabilities. The post serves as an educational resource explaining how VLMs combine visual and language understanding. As a tier-2 commentary piece, it synthesizes existing knowledge rather than presenting new research findings.
Act2Answer: Benchmarking commonsense and world knowledge retention in Vision-Language-Action models
Researchers introduce Act2Answer, a protocol for evaluating how much commonsense and factual knowledge VLA models retain after fine-tuning on robotics data. The approach converts knowledge benchmark questions into tabletop object-placement episodes, yielding action-grounded success rates that reduce confounds from low-level control failures. A large-scale study of 7 VLA models and 9 VLM baselines finds that VLAs retain solid performance on simple concepts but show larger gaps on richer semantic categories compared to their source VLMs, and that VQA co-training is associated with better knowledge retention.
Vision Language Model Alignment in TRL
Hugging Face's TRL library has added support for aligning Vision Language Models (VLMs), extending existing RLHF and preference optimization tooling to multimodal settings. The blog post covers the new capabilities for training VLMs with alignment techniques such as DPO and related methods. This expands the open-source ecosystem for multimodal model fine-tuning and alignment.
Vision-Language Models Suppress Female Representations Under Ambiguous Input
This paper investigates gender bias in vision-language models (VLMs) when inputs are ambiguous (e.g., workers in full gear or seen from behind), finding that models default to male outputs even for strongly female-stereotyped occupations. The authors introduce LALS (Latent Association Leaning Score), a zero-shot metric that probes internal visual-token activations to measure concept associations across layers. Across 15 occupations, 800+ ambiguous images, and four VLMs, they find a systematic decoupling: models internally encode female associations but suppress them before generation, with male signals amplifying end-to-end while female signals peak mid-network and are filtered out. Cultural visual cues like clothing color further modulate these internal associations.
Causal circuit analysis reveals how vision-language models resolve perception-knowledge conflicts
A new arXiv preprint uses activation patching and ablation studies to identify the mechanistic basis of perception-knowledge conflict in vision-language models across three VLM families. The authors find that visual grounding is the default behavior, while knowledge-grounded responses depend on a small set of attention heads (2.5–4.8% of total) concentrated in the network's second half. Ablating these heads flips knowledge-grounded predictions to visually grounded ones in 68–96% of cases while barely affecting visually grounded predictions, revealing an asymmetric causal structure. The identified heads decompose into routing heads and writing heads, and the circuit is consistent across model families and scales.
Vision Language Models (Better, faster, stronger)
A Hugging Face blog post surveys the state of vision-language models (VLMs) in 2025, covering advances in architecture, training, efficiency, and deployment. The post reviews progress across major open and closed VLMs, highlighting trends in multimodal capability, speed improvements, and practical deployment patterns. As a tier-2 commentary piece, it synthesizes the current landscape rather than announcing new research.