5arXiv cs.LG (Machine Learning)·
Score accuracy in forward diffusion does not guarantee numerical stability in reverse sampling
A new arXiv paper demonstrates that small forward-marginal L2 error in score matching does not certify numerical stability for discretized reverse-time diffusion samplers. The authors construct adversarial examples where Euler-Maruyama discretizations converge weakly yet all Wasserstein distances diverge, and show this failure can occur within a fixed finite neural architecture. A positive result is also provided: projecting the learned denoiser onto a bounded convex set containing the data support restores Wasserstein convergence under mild regularity. Experiments with a DiT-style network confirm the instability and its suppression via denoiser projection.
Related guides (1)
Related events (8)
Exact Posterior Score (EPS): Closed-form posterior sampling for linear inverse problems with diffusion models
A new arXiv preprint derives the exact posterior score in closed form for linear Gaussian inverse problems under general Gaussian interpolants, showing that posterior sampling reduces to a denoising problem at an operator-dependent shifted pivot under anisotropic noise covariance. The authors convert this identity into a training objective called Exact Posterior Score (EPS) that preserves the input/output structure of standard diffusion pretraining, enabling training from scratch or fine-tuning from a pretrained denoiser. EPS is evaluated on five linear inverse problems across FFHQ and ImageNet, outperforming both training-free and training-based baselines while requiring roughly an order of magnitude fewer denoiser evaluations than gradient-based posterior samplers.
Finite-Sample Lens for Understanding Diffusion Posterior Sampler Failures
This paper introduces a finite-sample theoretical framework for analyzing diffusion model posterior samplers used in imaging inverse problems. The authors show that popular likelihood approximations at intermediate timesteps systematically under- or over-estimate posterior spread, leading to failure modes including sensitivity to early stopping, incorrect weighting of posterior modes, and hallucination of prior or likelihood modes. Crucially, they demonstrate these failures can arise from a multimodal prior alone, without requiring nonlinear measurement models or multimodal posteriors. The framework is model-agnostic and can serve as a diagnostic tool for evaluating existing and future posterior samplers.
Theoretical unification of discrete diffusion model objectives via Oracle Distance theorem
A new arXiv preprint provides a rigorous theoretical framework for understanding what discrete diffusion models learn, proving the 'Oracle Distance' theorem: the negative ELBO exactly equals data entropy plus the path KL from the oracle reverse process to the learned one. The work shows that denoiser, score ratio, and bridge plug-in parameterizations are the same object in different coordinates, with closed-form conversions among them. It unifies several existing discrete diffusion losses (MDM, UDM, SEDD, GIDD) as special cases and identifies practical consequences, such as why denoiser parameterization causes the uniform ELBO to diverge at initialization. All identities are verified numerically on an exactly solvable model.
LESS: Adaptive mutual-stability sampling cuts diffusion LLM decoding steps by 72%
Researchers introduce LESS, a training-free adaptive sampler for diffusion large language models that treats token commitment as an online stopping problem. The method uses a joint stability rule combining confidence, persistence, and distributional stability to decide when to unmask tokens, avoiding wasted computation on already-stable positions. Evaluated on Dream-7B, LLaDA-8B, and LLaDA-1.5-8B across seven benchmarks, LESS reduces reverse denoising steps by 72.1% versus fixed-budget decoding while improving accuracy over prior adaptive samplers. The step reductions translate directly to fewer Transformer forward passes and lower wall-clock latency.
Kolmogorov Regression lifts diffusion policies to Cameron-Martin space for robust long-horizon control
Researchers introduce a backward Kolmogorov equation framework that reformulates diffusion policy training as a deterministic boundary-value PDE problem in Cameron-Martin space, replacing stochastic score matching. The approach uses a precision-weighted Cameron-Martin loss and a Kolmogorov residual as an inference-time failure detector, yielding convergence guarantees tied to kernel effective rank rather than action dimension. Validation on the PushT manipulation benchmark shows 17% improvement in episode reward and 67.6% reduction in inter-step drift; a 6-station manufacturing scheduling task shows 28.4% lower RMSE than LSTM baselines and 96% reduction in deadlock events via Hamilton-Jacobi reachability certification.
Open problem paper questions whether AdamW converges under heavy-tailed gradient noise
A preprint from arXiv frames as an open problem whether AdamW, the dominant optimizer for LLM pretraining, can achieve rigorous convergence guarantees under heavy-tailed stochastic gradient noise. The authors note that sign-based optimizers like Lion and Muon already have sharp heavy-tailed convergence rates, while AdamW's second-moment accumulator may create a fundamental obstruction by hiding large gradients. The paper proves a positive weighted-metric benchmark and introduces a corridor lower-bound mechanism to characterize the potential failure mode.
Selective timestep weighting and advantage-based replay improve diffusion RLHF sample efficiency by up to 6×
A new arXiv preprint proposes two complementary techniques to improve feedback efficiency in diffusion model RLHF: a per-timestep weighting scheme grounded in PPO convergence theory, and a replay mechanism that prioritizes informative trajectories to reduce redundant reward queries. Together, the methods achieve up to 6× improvement in sample efficiency over standard diffusion RLHF baselines under identical hyperparameter settings. The work addresses a practical bottleneck—feedback cost—that limits real-world deployment of RLHF-aligned diffusion models.
GADD: Gibbs-Accelerated Discrete Diffusion Achieves Polylog Sampling Complexity
This paper introduces Gibbs-Accelerated Discrete Diffusion (GADD), a corrector method for uniform-rate discrete diffusion models that constructs Gibbs posterior likelihoods directly from the concrete score function without additional training. GADD achieves O(polylog(ε⁻¹)) sampling complexity, the first such rate for diffusion-based samplers in this setting. Experiments on synthetic data, zero-shot text sampling, and zero-shot conditional music generation show consistent improvements in sample quality and wall-clock efficiency over Euler and CTMC baselines. The work also introduces a novel induction-based theoretical framework for analyzing predictor-corrector methods in discrete diffusion.