technique

continuous diffusion language model

techniqueactivecontinuous-diffusion-language-model-4d682d15·1 events·first seen 29d ago

Aliases: continuous diffusion language model

Co-occurring entities

RePlaid PLAID OpenWebText Duo MDLM ELBO variance minimization

More like this (12)

Diffusion Language Models latent diffusion model Diffusion Models discrete diffusion models LESS: Mutual-Stability Sampling for Diffusion Language Models diffusion-based generative models Self-Augmenting Retrieval for Diffusion Language Models Character-level Language Model Representation-Conditioned Diffusion Models text-to-image diffusion model mRNA Language Model Continuous-Time Consistency Models

Recent events (1)

6arXiv · cs.CL·29d ago·source ↗

RePlaid: Continuous Diffusion Language Models Scale Competitively with Discrete Diffusion

This paper revisits continuous diffusion language models (DLMs) by introducing RePlaid, an updated version of Plaid that aligns its architecture with modern discrete DLMs. RePlaid establishes the first scaling law for continuous DLMs competitive with discrete approaches, achieving a compute gap of only 20× versus autoregressive models and a state-of-the-art perplexity bound of 22.1 on OpenWebText among continuous DLMs. The authors provide theoretical analysis showing that likelihood-based training naturally yields linear cross-entropy over time and creates structured embedding geometries, explaining the performance gains.

Frontier Model Releases Evaluation and Benchmarking RePlaid PLAID OpenWebText +4 more