6arXiv cs.CL (Computation and Language)·
Quote-and-retrieve pipeline reduces attribution hallucination in visual document understanding without coordinate supervision
Researchers present a study showing that replacing coordinate-based evidence attribution with a language interface—where vision-language models quote evidence verbatim and a multimodal retriever locates the quote—substantially reduces attribution hallucination in visual document understanding. Across six open VLMs on a bilingual CiteVQA subset, evidence recall rises from at most 8 points to 26–47 points and hallucination rate roughly halves. The authors extend this into a GRPO-based training recipe that uses retrieved region crops as reward signal, lifting an 8B model's strict attributed accuracy from 22.4 to 33.8 without any region-level labels. The work offers a practical path to grounded document QA without expensive spatial annotation.
Related guides (3)
Related events (8)
The Abstraction Gap in Vision-Language Causal Reasoning
Researchers introduce a dual-probe methodology and the CAGE benchmark (49,500 questions across 5,500 images) to distinguish linguistic plausibility from faithful causal reasoning in vision-language models. An Abstraction Gap (AG) metric quantifies the normalized performance difference between text-only and chain-of-reasoning probes. Evaluating eight VLMs, seven exhibit AG exceeding 0.50—generating fluent causal text but failing structured causal chain tasks—while one model achieves near-zero AG, suggesting architectural and pretraining choices are decisive. Fine-tuning on 45,000 chain-annotated examples fails to close the gap, pointing to a fundamental capability distinction.
ACL-Verbatim: Hallucination-Free Extractive QA System for Research Papers
The paper introduces ACL-Verbatim, an extractive question answering system built on VerbatimRAG that maps user queries directly to verbatim text spans in ACL Anthology papers, eliminating hallucination by design. The authors contribute a new ground-truth benchmark dataset created via human NLP-researcher annotation over synthetic queries generated using a ScIRGen-based pipeline. A 150M-parameter ModernBERT token classifier trained on silver supervision achieves the best word-level F1 of 53.6, outperforming the strongest LLM-based extractor at 48.7. The work demonstrates that smaller extractive models can outperform large generative LLMs on precision-critical retrieval tasks.
MedFocus: Causal Visual Attribution Framework for Chest X-ray Reasoning in Large Vision-Language Models
This paper addresses the faithfulness of visual attribution methods in Large Vision-Language Models (LVLMs) applied to chest X-ray (CXR) analysis. The authors develop a causal evaluation framework using counterfactual editing to verify whether expert-annotated regions are causally responsible for model predictions, testing 11 attribution methods across six open-source LVLMs. Finding that existing attribution methods frequently fail to identify the actual visual evidence used by models, they propose MedFocus, a concept-based attribution method using unbalanced optimal transport to localize anatomical regions and measure their causal effect on outputs. MedFocus substantially outperforms prior methods and provides spatial, concept-level, and token-level attributions.
EG-VQA benchmark exposes gap between answer correctness and evidence grounding in Video-LLMs
Researchers introduce EG-VQA, a benchmark of 2,067 videos and 11,838 QA pairs designed to evaluate Video Large Language Models on both answer correctness and temporal evidence grounding simultaneously. A new metric, Evidence-Grounded F1 (EG-F1), jointly measures temporal alignment and semantic consistency against annotated ground-truth evidence. Experiments show that even strong proprietary models fail to reliably localize supporting evidence, revealing a fundamental gap between surface-level accuracy and faithful reasoning. The authors also propose EG-Reasoner, an evidence-supervised open-source model that achieves competitive results against proprietary systems.
Act2Answer: Benchmarking commonsense and world knowledge retention in Vision-Language-Action models
Researchers introduce Act2Answer, a protocol for evaluating how much commonsense and factual knowledge VLA models retain after fine-tuning on robotics data. The approach converts knowledge benchmark questions into tabletop object-placement episodes, yielding action-grounded success rates that reduce confounds from low-level control failures. A large-scale study of 7 VLA models and 9 VLM baselines finds that VLAs retain solid performance on simple concepts but show larger gaps on richer semantic categories compared to their source VLMs, and that VQA co-training is associated with better knowledge retention.
LAVE: Zero-shot VQA Evaluation on Docmatix with LLMs - Do We Still Need Fine-Tuning?
This Hugging Face blog post introduces LAVE (LLM-Assisted Visual Evaluation), a zero-shot VQA evaluation methodology applied to the Docmatix dataset. The post investigates whether large vision-language models can perform document visual question answering without task-specific fine-tuning by leveraging LLM-based evaluation metrics. The analysis probes the gap between zero-shot and fine-tuned performance on document understanding tasks, raising questions about the continued necessity of supervised adaptation for VQA.
E-VQA: Evidence-Backed Video Question Answering with Spatio-Temporal Grounding
Researchers from Salesforce AI Research propose Evidence-Backed Video Question Answering (E-VQA), a task requiring Video LLMs to jointly produce semantic answers and precise spatio-temporal evidence via dense tracked object segmentation masklets. They introduce ST-Evidence, the first human-verified pixel-level grounding benchmark for video QA, and ST-Evidence-Instruct, a 160k-scale training dataset. Fine-tuning on this data yields substantial gains over size-matched baselines (+27.2 t-mean, +13.8 J&F on a 7B model), revealing a critical gap between QA accuracy and true visual perception that scaling alone does not close.
LoMo: Local Modality Substitution for Deeper Vision-Language Fusion
This paper identifies a 'carrier sensitivity' problem in Vision-Language Models (VLMs), where replacing textual queries with rendered-image equivalents causes significant performance degradation due to asymmetric roles of text and images in training data. The authors propose Local Modality Substitution (LoMo), a data curation paradigm that reformulates single-modality prompts into interleaved multimodal sequences by dynamically rendering text spans as images, enforcing cross-modal representational invariance. Evaluated across 13 multimodal benchmarks, LoMo improves over standard supervised fine-tuning by 2.67 points on LLaVA-OneVision-1.5-8B and 2.82 points on Qwen3.5-9B. The approach is architecture-agnostic and lightweight, requiring no changes to model architecture.