5arXiv cs.AI (Artificial Intelligence)·
Subjective Risk Decomposition: A unified theoretical framework for uncertainty quantification
A new arXiv preprint proposes deriving epistemic and aleatoric uncertainty measures as consequences of decomposing a subjective risk based on a strictly proper loss, rather than treating them as axiomatic primitives. The framework unifies numerous previously proposed UQ measures under a common theoretical foundation, with reverse cross-entropy recovering classic information-theoretic uncertainty terms as a special case. The authors extend the view to learning theory, introducing subjective risk analogues of excess risk, approximation error, and estimation error, positioning this as a first step toward a full learning-theoretic framework for UQ.
Related guides (2)
Related events (8)
Three-axis uncertainty estimation framework for code generation outperforms NL-derived baselines
A new arXiv preprint argues that uncertainty estimation (UE) for code generation requires code-specific design rather than methods ported from natural language. The authors propose three orthogonal uncertainty axes—lexical (token entropy), algorithmic (pseudo-code consistency), and functional (behavioral consistency)—grounded in properties unique to code: token fragility, intent-code gap, and executability. Evaluated across five code LLMs, their ensemble improves average AUROC from 0.696 to 0.776 (+8.1 points) over the strongest NL-derived baseline, with a single-pass token entropy method on Qwen3-14B matching multi-pass baselines at 3x lower cost. The work is directly relevant to safe deployment of LLMs in agentic coding pipelines.
Uncertainty-aware decision-making algorithms for LLMs using Bayesian and risk-averse methods
A new arXiv preprint proposes and evaluates uncertainty-aware decision-making algorithms for LLMs grounded in Bayesian decision theory and risk-averse decision making, applied to tutoring and automatic peer review tasks. The authors incorporate conformal prediction to provide formal guarantees over strategy and score outputs. Empirical results show Bayesian methods outperform risk-averse rules, which can degrade to generic outputs under high ambiguity. The work highlights a gap in decision-making algorithm research relative to model training advances.
Reverse Probing: Supervised Token-level Uncertainty Quantification for LLMs in Clinical Text
The paper introduces Reverse Probing, a novel uncertainty quantification framework designed specifically for clinical text summarization that estimates token-level uncertainty from pre-existing labeled summaries rather than sampling new outputs. It extracts uncertainty signals from four categories of internal model activations, treating text as a probe into the model's internal state. Evaluated on two expert-annotated clinical datasets, it outperforms eight adapted baselines on all metrics, achieving up to 4× higher AUPRC while reducing inference time and compute. Feature analysis identifies delta energy and neighborhood context as the most consistent predictors of uncertainty across models.
Clustered Self-Assessment: LLM uncertainty quantification via semantic clustering and multiple-choice self-evaluation
A new arXiv preprint proposes Clustered Self-Assessment, a method for uncertainty quantification in LLMs that groups sampled generations into semantically distinct clusters, reformats them as multiple-choice options, and uses the model's own probability assignments as confidence estimates. The approach outperforms entropy-based baselines across multiple models and datasets, achieving competitive performance with as few as two additional samples. The method is notable for directly leveraging the model's self-assessment capability rather than relying on indirect distributional signals.
Reward uncertainty as a principled mechanism for diverse RL behaviour
A new arXiv preprint proposes replacing the scalar reward in RL with a distribution over reward functions, applying a non-linear objective over sets of actions to induce calibrated behavioural diversity without sacrificing expected reward. The authors derive a principled gradient estimator in the contextual bandit setting and prove the formulation generalizes vanilla policy gradient and action-set approaches. The work is motivated by applications like language model fine-tuning where diversity is desirable but entropy regularization and diversity bonuses introduce fragile trade-offs. Empirical results support the framework as a theoretically grounded alternative to heuristic diversity methods.
Uncertainty-Based Decontamination (UBD) framework for removing benchmark contamination from LLMs
Researchers propose Uncertainty-Based Decontamination (UBD), a method that uses deep ensembles of a contaminated model to estimate per-sample memorization and correct for benchmark data contamination without requiring access to an uncontaminated reference model. The approach introduces a sample-level evaluation framework using distributional distance metrics alongside aggregate accuracy to better characterize decontamination quality. Experiments on MMLU-Pro and MATH-MCQA show UBD produces output distributions closer to uncontaminated baselines than paraphrasing or choice-permutation methods. The work addresses a significant validity concern in LLM evaluation, where contamination inflates reported benchmark performance.
Scale vs. quantization tradeoffs for uncertainty signals in vision-language models under image degradation
A new arXiv paper evaluates how model scale and 4-bit quantization affect two confidence signals — verbalized confidence and mean token probability — in the Qwen2-VL family across 5,700 predictions under six photographic degradation types. Key findings: scaling from 2B to 7B sharply improves internal uncertainty (AUROC 0.80→0.98) while verbalized confidence remains weak; 4-bit quantization costs little in accuracy (-1.6 pts) but degrades the internal confidence signal (AUROC 0.95→0.80) and collapses verbalized-confidence parse rate from 99% to 64%. The practical recommendation is to prefer a larger quantized model over a smaller full-precision one within a fixed memory budget, and to use error-detection AUROC rather than calibration error as the primary metric.
Large-scale study of uncertainty estimation across 22 languages in LLMs finds English reasoning improves low-resource UE
A new arXiv preprint presents the first large-scale evaluation of uncertainty estimation (UE) methods across 22 languages, covering high-, mid-, and low-resource settings using nine open- and closed-box UE methods. Key findings include that prompting models to reason in English while keeping questions in low-resource languages substantially improves UE performance, suggesting the reliability bottleneck is in generation rather than comprehension. The study also finds that optimal UE method choice depends on model scale: probability-based open-box methods win at smaller scales, while self-verbalized closed-box uncertainty dominates at larger scales. Practical guidance on threshold selection for abstention in multilingual settings is also provided.