4arXiv cs.CL (Computation and Language)·
ENTRAP-VL: A benchmark for measuring contextual entrainment in vision-language models
Researchers introduce ENTRAP-VL, a manually curated 1,500-item benchmark designed to probe contextual entrainment in vision-language models (VLMs) — the tendency of models to let irrelevant or false auxiliary context distort their outputs. The benchmark is taxonomically structured across two axes (context association and truth relationship) and two modality streams (textual and visual entrainment), covering eight categories. The paper argues that entrainment in VLMs is a substantively new dual-modality phenomenon not captured by porting unimodal text benchmarks, and the dataset will be released publicly.
Related guides (2)
Related events (8)
Real Images, Worse Judgments: Evaluating VLMs on Concreteness and Imagery
This paper evaluates whether vision-language models (VLMs) benefit from real image context when making lexical judgments about word concreteness and imagery. The authors find that real-image contexts frequently hurt alignment with human ratings, especially when visual evidence is least relevant to the word being judged. Probing and canonical correlation analysis reveal that real images cause representational shifts and increased sensitivity to spurious visual cues. Instructing models to focus on text-only content at inference time partially mitigates this degradation.
AnyGroundBench: Domain-adaptation benchmark for spatio-temporal video grounding in specialized fields
Researchers introduce AnyGroundBench, a benchmark designed to evaluate Vision-Language Models on Spatio-Temporal Video Grounding (STVG) across five specialized domains: animal behavior, industry, sports, surgery, and public security. Unlike existing benchmarks that focus on zero-shot general-domain evaluation, AnyGroundBench includes dedicated training subsets to measure domain adaptability and in-context learning. Evaluation of 15 state-of-the-art VLMs reveals significant failures in both zero-shot and ICL-based adaptation, exposing weaknesses in spatio-temporal reasoning under domain shift.
IndicContextEval: Benchmark for context utilisation in Audio LLMs across 8 Indic languages
Researchers introduce IndicContextEval, a 56-hour multilingual speech benchmark covering 555 speakers across 8 Indian languages and 23 professional domains, designed to test whether Audio LLMs genuinely use textual context (domain descriptions, entity lists) or rely on parametric knowledge. The benchmark employs a 7-level prompting framework that progressively introduces contextual signals including adversarial prompts with incorrect entities. Evaluation of five models reveals substantial variation in context utilisation behaviour, exposing a gap in existing ASR benchmarks that test only fixed prompting conditions.
VisAnomReasoner: Efficient VLM for Time-Series Anomaly Detection via VisAnomBench
Researchers introduce VisAnomBench, a curated benchmark augmenting public time-series anomaly datasets with natural-language rationales generated and selected from multiple large VLMs using task-specific rewards. Fine-tuning on this benchmark produces VisAnomReasoner, a parameter-efficient vision-language model that outperforms all baselines by at least 21.23 and 23.87 percentage points in precision and F1 on VisAnomBench. Cross-benchmark evaluation on TSB-AD-U shows further generalization gains of 9.57 and 13.39 percentage points in precision and F1, respectively.
Act2Answer: Benchmarking commonsense and world knowledge retention in Vision-Language-Action models
Researchers introduce Act2Answer, a protocol for evaluating how much commonsense and factual knowledge VLA models retain after fine-tuning on robotics data. The approach converts knowledge benchmark questions into tabletop object-placement episodes, yielding action-grounded success rates that reduce confounds from low-level control failures. A large-scale study of 7 VLA models and 9 VLM baselines finds that VLAs retain solid performance on simple concepts but show larger gaps on richer semantic categories compared to their source VLMs, and that VQA co-training is associated with better knowledge retention.
VLMs overestimate common ground in asymmetric dialogue, conflating potential with established shared understanding
A new arXiv paper investigates whether vision-language models can distinguish between what could be shared versus what has actually been established as shared between dialogue participants. Using 13,077 annotated reference expressions from HCRC MapTask dialogues, the authors find that VLMs systematically over-predict alignment when given task-relevant map content—whether presented visually or as text—suggesting the bias stems from static referential cues rather than tracking grounding through dialogue history. The effect is observed most strongly in Qwen3-VL-8B-Instruct and replicated across four additional models from two architecture families, revealing a fundamental limitation in how current VLMs model collaborative dialogue.
Test-time training corrects modality order sensitivity in vision-language models
Researchers find that vision-language models consistently perform better when images are presented before questions rather than after, a semantically irrelevant prompt-order artifact that persists across three models and three benchmarks. The authors propose an order-consistent test-time training method that closes this gap and also improves the stronger image-first baseline. Activation patching localizes the failure to a narrow mid-network region where representations diverge between orderings, framing modality-order sensitivity as a circuit-level failure. The work contributes both a diagnostic finding about VLM robustness and a lightweight mitigation technique.
Introducing HELMET: Holistically Evaluating Long-context Language Models
HELMET is a new benchmark designed to holistically evaluate long-context language models across diverse real-world tasks rather than synthetic needle-in-a-haystack tests. The benchmark covers multiple task categories including retrieval, reasoning, summarization, and code, aiming to provide more reliable and comprehensive assessment of long-context capabilities. It is introduced via the Hugging Face blog, suggesting an open release with associated tooling for the community.