5arXiv cs.CL (Computation and Language)·
Small VLMs have usable internal confidence signals that their verbalized outputs fail to express
A new arXiv paper evaluates two small open-weight vision-language models (Qwen2-VL-2B-Instruct and SmolVLM-Instruct) under six realistic image degradations, comparing verbalized confidence against internal token probability as uncertainty signals. Verbalized confidence is found to be nearly constant and near-chance at error detection (AUROC ~0.50), while internal token probability reliably separates correct from incorrect answers (AUROC 0.92–0.99). Both signals fail under severe underexposure, where accuracy collapses but confidence barely moves. The authors conclude that internal probability is the superior deferral signal for constrained deployments, and that small VLMs encode self-knowledge they cannot articulate.
Related guides (2)
Related events (8)
Real Images, Worse Judgments: Evaluating VLMs on Concreteness and Imagery
This paper evaluates whether vision-language models (VLMs) benefit from real image context when making lexical judgments about word concreteness and imagery. The authors find that real-image contexts frequently hurt alignment with human ratings, especially when visual evidence is least relevant to the word being judged. Probing and canonical correlation analysis reveal that real images cause representational shifts and increased sensitivity to spurious visual cues. Instructing models to focus on text-only content at inference time partially mitigates this degradation.
Self-improving VLMs can silently regress when verifier quality is task-mismatched
A new arXiv paper demonstrates that verifier-driven self-DPO, a common recipe for self-improving visual-language models, can silently degrade student model performance when the verifier's task-rubric accuracy is insufficient for the target task. Experiments on Qwen-3-VL-2B and Qwen-2.5-VL-3B across MathVista, MMMU, and BLINK show regressions of 3.4–10.9 percentage points below frozen baselines, with the counterintuitive finding that more accurate-but-still-wrong verifiers cause larger regressions than near-random ones. The authors provide a mechanistic explanation via a variance theorem for progress-gated replay and offer operational guidance: measure target-task rubric accuracy before running any verifier-driven loop and rank verifiers by task-specific quality rather than parameter count.
VLMs overestimate common ground in asymmetric dialogue, conflating potential with established shared understanding
A new arXiv paper investigates whether vision-language models can distinguish between what could be shared versus what has actually been established as shared between dialogue participants. Using 13,077 annotated reference expressions from HCRC MapTask dialogues, the authors find that VLMs systematically over-predict alignment when given task-relevant map content—whether presented visually or as text—suggesting the bias stems from static referential cues rather than tracking grounding through dialogue history. The effect is observed most strongly in Qwen3-VL-8B-Instruct and replicated across four additional models from two architecture families, revealing a fundamental limitation in how current VLMs model collaborative dialogue.
Test-time scaling for small VLMs on multilingual visual MCQ: conditions matter more than methods
A new arXiv paper examines whether test-time scaling (TTS) transfers to small open vision-language models using EXAMS-V, a multilingual visual multiple-choice benchmark. The study compares self-consistency, describe-then-reason with PRM-guided beam search, and post-hoc selectors across Qwen2.5-VL-7B-Instruct and Qwen3.5-4B. Key findings: prompt parseability and decoding budget (token limit) dominate gains, while elaborate search/verification methods like PRM-guided beam search underperform plain majority vote at 8x the cost. The best configuration achieves 84.1% on ImageCLEF 2026 test split, ranking first on the Visual MCQ leaderboard.
Training framework reduces calibration error 60%+ in Medical VQA multimodal LLMs
A new arXiv preprint proposes a finetuning framework to improve verbalized uncertainty calibration in multimodal LLMs applied to Medical Visual Question Answering. The composite loss function combines Brier-style calibration, anchor regularization, contrastive image-text alignment, and KL-based stabilization, evaluated on MedGemma 4B IT and Qwen2-VL 7B Instruct across three medical VQA benchmarks. The method reduces calibration error by 60% or more and improves discrimination by 26% or more while preserving predictive accuracy, outperforming prompting-, sampling-, and training-based baselines.
SmolVLM - Small Yet Mighty Vision Language Model
Hugging Face introduces SmolVLM, a compact vision-language model designed to deliver strong multimodal performance at small parameter counts. The model targets edge and resource-constrained deployment scenarios while maintaining competitive capabilities relative to its size. The announcement highlights efficiency improvements in both training and inference for small-scale VLMs.
Confidence and Calibration of Activation Oracles for Reliable Interpretation of Language Model Internals
This paper investigates uncertainty quantification (UQ) for activation oracles—systems that make LLM internal activations human-legible—by evaluating 6 confidence estimation methods across 6,000 samples per oracle. The authors find that bootstrap mode frequency achieves the best calibration (ECE 5.7% vs. 25.5% for log-probability baseline on Qwen3-8B), while the log-prob baseline remains useful as a cheap triage signal. Experiments vary verbalizer and context prompts across two Qwen3 model sizes. Code and a patched trainer are released publicly.
Visual Contrastive Self-Distillation (VCSD) improves multimodal LLM training without external teachers
Researchers propose Visual Contrastive Self-Distillation (VCSD), a training method for vision-language models that creates an on-policy self-distillation signal purely through input conditioning — specifically by contrasting token distributions from an EMA teacher conditioned on the original image versus a content-erased control. The approach eliminates the need for external teachers, privileged answers, visual evidence signals, or reasoning traces. Evaluated on Qwen3-VL and Qwen3.5 models using the ViRL39K dataset, VCSD consistently outperforms matched on-policy self-distillation baselines, with aggregate benchmark gains of up to ~3.75 percentage points at the 8B scale.