4arXiv cs.CL (Computation and Language)·
Noisy-channel decomposition of Minimum Bayes Risk decoding addresses asymmetric metric bias
A new arXiv preprint proposes a noisy-channel decomposition of Minimum Bayes Risk (MBR) decoding that breaks the process into four components: hypothesis-to-reference likelihood, reference-to-hypothesis likelihood, hypothesis prior, and reference prior. The decomposition addresses a known asymmetry problem in MBR decoding caused by directional evaluation metrics like BLEU and COMET. The framework unifies existing MBR variants under a single interpretation and suggests that channel-specific weighting could improve over standard MBR decoding.
Related guides (1)
Related events (8)
Variance-Calibrated Modulation (VCM): training-free decoding intervention to address LLM likelihood trap
Researchers propose Variance-Calibrated Modulation (VCM), a training-free pre-decoding method that reshapes LLM probability distributions before truncation to combat repetitive degeneration and vocabulary dullness. VCM combines two mechanisms: Contextual Searchlight via PMI (suppressing stopwords, elevating context-relevant tokens) and Adaptive Self-Debiasing (scale-invariant penalization using real-time logit standard deviation). Evaluated across open-ended generation, factual QA, and mathematical reasoning, VCM improves diversity, coherence, and reasoning accuracy at higher temperatures with negligible overhead. The method is compatible with existing decoding strategies like Top-p and Min-p.
Theory of acceptance criteria in speculative decoding for greedy and relaxed regimes
A new arXiv preprint develops a theoretical framework for speculative decoding acceptance criteria beyond the standard stochastic, distribution-preserving setting. The authors characterize rejection regions for greedy decoding, additive/multiplicative relaxed acceptance, top-m criteria, and entropy-thresholded acceptance in terms of KL divergence and margin-based bounds. The framework is extended to greedy tree decoding and validated empirically on Qwen3 models, showing relaxed and tree-based criteria substantially expand certified acceptance regions. The work fills a gap between existing theory and practical inference systems that use non-exact acceptance rules.
CTC oracle gap anatomy: acoustic scoring saturates, linguistic MBR decoding recovers WER
A new arXiv paper systematically diagnoses why CTC-internal N-best rescoring fails to improve over greedy decoding on LibriSpeech, showing that blank-path proliferation causes a 53% degradation in rank correlation between CTC scores and WER as beam size grows. The authors demonstrate that the bottleneck is linguistic rather than acoustic: MBR decoding with RoBERTa pseudo-log-likelihood achieves 9% relative WER reduction on LibriSpeech test-other and generalizes across two architectures and three domains. The paper also analyzes MWER sequence-level fine-tuning failure at near-converged checkpoints, attributing collapse to a vanishingly small training oracle gap.
Unified MAIR framework bridges GP-UCB and DEC approaches in kernel bandits
A new arXiv preprint unifies two major theoretical frameworks for frequentist RKHS bandits — Gaussian-process upper confidence bound (GP-UCB) and decision-estimation-coefficient (DEC) methods — under a common algorithmic-information language called MAIR. The paper generalizes both the GP-UCB analysis and the MAMS algorithm, proposes a safeguarded master algorithm combining their advantages, and demonstrates that algorithmic complexity can be more informative than class-wide minimax certificates in overparameterized models. The work clarifies a foundational distinction between algorithmic information and minimax coefficients in bandit theory.
PAC-Bayes analysis establishes formal expressivity and alignment floors for prompt-conditioned LLMs
A new arXiv preprint models user-LLM interaction as a bilevel cheap-talk game and derives PAC-Bayes bounds showing two irreducible limitations: an 'expressivity floor' where language's finite channel capacity makes distinct tasks indistinguishable, and an 'objective-misalignment floor' where alignment constraints prevent reaching user-ideal outputs. The authors prove that prompt-conditioned LLMs cannot be universal problem solvers, as correct behavior on certain task families is provably unattainable even with infinite data, optimal training, or model scaling. The work suggests multimodal inputs and external memory as potential mitigations by increasing task-relevant information bandwidth.
ASRD: Training-free anchor-guided revocable decoding for diffusion LLMs improves accuracy and throughput
A new arXiv preprint introduces ASRD (Anchor Supervised Revocable Decoding), a training-free framework for improving decoding quality in diffusion large language models. The method addresses error propagation and local error reinforcement in revocable decoding by separating trusted 'anchor tokens' (identified via temporal consistency) from uncertain candidates, then applying anchor-guided generation and anchor-perturbed verification. Experiments on math and coding benchmarks show up to 6.4% accuracy improvement and 7.2× inference throughput gains over remasking baselines.
ADAS: Attention-Discounted Adaptive Sampler improves parallel decoding for masked diffusion language models
Researchers propose ADAS, a training-free reranking rule for masked diffusion language model decoding that addresses token interaction failures in parallel token commitment. The method greedily penalizes candidates that attend strongly to already-selected uncertain positions, using attention weights as soft marginal penalties rather than hard constraints. Evaluated on LLaDA-8B-Base and Dream-7B-Base across GSM8K, MATH500, HumanEval, and MBPP, ADAS improves low-NFE performance by 9–10 percentage points on average when plugged into existing samplers with only 3.1% runtime overhead.
Uncertainty-aware decision-making algorithms for LLMs using Bayesian and risk-averse methods
A new arXiv preprint proposes and evaluates uncertainty-aware decision-making algorithms for LLMs grounded in Bayesian decision theory and risk-averse decision making, applied to tutoring and automatic peer review tasks. The authors incorporate conformal prediction to provide formal guarantees over strategy and score outputs. Empirical results show Bayesian methods outperform risk-averse rules, which can degrade to generic outputs under high ambiguity. The work highlights a gap in decision-making algorithm research relative to model training advances.