Entity · paper

What Does a Discrete Diffusion Model Learn?

paperactiveprovisionalwhat-does-a-discrete-diffusion-model-learn--9a91f4ab·1 events·first seen 40h ago

Aliases: What Does a Discrete Diffusion Model Learn?

Co-occurring entities

More like this (12)

discrete diffusion models Masked Diffusion Models Representation-Conditioned Diffusion Models Diffusion Models Diffusion Language Models Denoising Diffusion Probabilistic Models Gibbs-Accelerated Discrete Diffusion (GADD)Survival Diffusion Probabilistic Model (SDPM)continuous diffusion language model latent diffusion model Knowledge Editing in Masked Diffusion Language Models Block-Size Curriculum Learning for Diffusion Reasoning Models

Recent events (1)

5arXiv · cs.CL·40h ago·source ↗

Theoretical unification of discrete diffusion model objectives via Oracle Distance theorem

A new arXiv preprint provides a rigorous theoretical framework for understanding what discrete diffusion models learn, proving the 'Oracle Distance' theorem: the negative ELBO exactly equals data entropy plus the path KL from the oracle reverse process to the learned one. The work shows that denoiser, score ratio, and bridge plug-in parameterizations are the same object in different coordinates, with closed-form conversions among them. It unifies several existing discrete diffusion losses (MDM, UDM, SEDD, GIDD) as special cases and identifies practical consequences, such as why denoiser parameterization causes the uniform ELBO to diverge at initialization. All identities are verified numerically on an exactly solvable model.

Evaluation and Benchmarking SEDD MDM GIDD +2 more