5arXiv cs.CL (Computation and Language)·
Scale vs. quantization tradeoffs for uncertainty signals in vision-language models under image degradation
A new arXiv paper evaluates how model scale and 4-bit quantization affect two confidence signals — verbalized confidence and mean token probability — in the Qwen2-VL family across 5,700 predictions under six photographic degradation types. Key findings: scaling from 2B to 7B sharply improves internal uncertainty (AUROC 0.80→0.98) while verbalized confidence remains weak; 4-bit quantization costs little in accuracy (-1.6 pts) but degrades the internal confidence signal (AUROC 0.95→0.80) and collapses verbalized-confidence parse rate from 99% to 64%. The practical recommendation is to prefer a larger quantized model over a smaller full-precision one within a fixed memory budget, and to use error-detection AUROC rather than calibration error as the primary metric.
Related guides (3)
Related events (8)
Small VLMs have usable internal confidence signals that their verbalized outputs fail to express
A new arXiv paper evaluates two small open-weight vision-language models (Qwen2-VL-2B-Instruct and SmolVLM-Instruct) under six realistic image degradations, comparing verbalized confidence against internal token probability as uncertainty signals. Verbalized confidence is found to be nearly constant and near-chance at error detection (AUROC ~0.50), while internal token probability reliably separates correct from incorrect answers (AUROC 0.92–0.99). Both signals fail under severe underexposure, where accuracy collapses but confidence barely moves. The authors conclude that internal probability is the superior deferral signal for constrained deployments, and that small VLMs encode self-knowledge they cannot articulate.
Confidence and Calibration of Activation Oracles for Reliable Interpretation of Language Model Internals
This paper investigates uncertainty quantification (UQ) for activation oracles—systems that make LLM internal activations human-legible—by evaluating 6 confidence estimation methods across 6,000 samples per oracle. The authors find that bootstrap mode frequency achieves the best calibration (ECE 5.7% vs. 25.5% for log-probability baseline on Qwen3-8B), while the log-prob baseline remains useful as a cheap triage signal. Experiments vary verbalizer and context prompts across two Qwen3 model sizes. Code and a patched trainer are released publicly.
Test-time scaling for small VLMs on multilingual visual MCQ: conditions matter more than methods
A new arXiv paper examines whether test-time scaling (TTS) transfers to small open vision-language models using EXAMS-V, a multilingual visual multiple-choice benchmark. The study compares self-consistency, describe-then-reason with PRM-guided beam search, and post-hoc selectors across Qwen2.5-VL-7B-Instruct and Qwen3.5-4B. Key findings: prompt parseability and decoding budget (token limit) dominate gains, while elaborate search/verification methods like PRM-guided beam search underperform plain majority vote at 8x the cost. The best configuration achieves 84.1% on ImageCLEF 2026 test split, ranking first on the Visual MCQ leaderboard.
New paper exposes behavioral divergence in quantized LLMs beyond accuracy/perplexity metrics
A new arXiv preprint argues that standard accuracy and perplexity metrics fail to capture behavioral changes induced by post-training quantization. The authors introduce 'correctness agreement', a decision-level metric measuring overlap in correct predictions between base and quantized models, and find behavioral divergence emerges even when task performance appears preserved. Analysis of attention weight distortions reveals non-linear breakpoints at low bit-widths and that query/key projections are more sensitive to quantization than value/output projections. The findings challenge the assumption that quantized models are behaviorally equivalent to their base counterparts.
Training framework reduces calibration error 60%+ in Medical VQA multimodal LLMs
A new arXiv preprint proposes a finetuning framework to improve verbalized uncertainty calibration in multimodal LLMs applied to Medical Visual Question Answering. The composite loss function combines Brier-style calibration, anchor regularization, contrastive image-text alignment, and KL-based stabilization, evaluated on MedGemma 4B IT and Qwen2-VL 7B Instruct across three medical VQA benchmarks. The method reduces calibration error by 60% or more and improves discrimination by 26% or more while preserving predictive accuracy, outperforming prompting-, sampling-, and training-based baselines.
LogbQuant: Logarithmic quantization with adjustable bases for language models
A new arXiv preprint introduces LogbQuant, a logarithmic quantization scheme with tunable bases designed to better capture common weight distributions in language models. The method targets the known weakness of uniform quantization in handling low-frequency, high-magnitude weights. At 4-bit precision, LogbQuant claims superior performance over asymmetric linear quantization at tensor-wise granularity, with moderate speedup and high memory savings suitable for consumer-grade GPU deployment.
Mapping the Schedule × Bit-Width Boundary in Sub-100M Quantisation-Aware Training
A large factorial grid study (1345 total runs across two phases) tests whether optimal learning-rate schedules differ by bit-width during from-scratch quantisation-aware training (QAT) for sub-100M decoder language models. The primary hypothesis—that INT6 QAT requires a different schedule than FP16/INT8—is falsified; a 33% warmdown fraction is optimal across all precisions and model sizes from 5M to 350M. For INT4, a regime boundary is identified near 50M parameters: above it, wd33 is decisively optimal; below it, schedule choice falls within seed-level noise. The study also establishes a log-linear scaling law for the INT6 quantisation penalty that successfully predicts held-out model sizes.
Framework for quantifying faithful confidence expression in large reasoning models
A new arXiv preprint introduces a framework to measure faithful calibration (FC) in large reasoning models (LRMs)—the alignment between a model's intrinsic confidence and its linguistically expressed confidence. The authors analyze linguistic decisiveness against three internal uncertainty sources (token probabilities, hidden states, sampled response consistency) and introduce prefix-conditioned sampling to handle structural variation in chain-of-thought traces. Applying the framework across leading models, they find FC is a significant and distinct failure mode for LRMs: extended reasoning traces do not automatically improve calibration, prompt interventions that help non-reasoning models fail in the reasoning setting, and different confidence estimators produce divergent assessments of the same traces.