5arXiv cs.CL (Computation and Language)·
Mechanistic analysis reveals bidirectional induction circuits in diffusion language models
A new arXiv preprint performs mechanistic interpretability analysis on masked diffusion language models (DLMs), identifying a bidirectional induction circuit that enables in-context learning by leveraging both past and future context around masked tokens. The study compares attention-only autoregressive transformers with absorbing-mask DLMs under matched architectures, finding that DLMs do not outperform AR models when only left context is visible but show stronger induction when both sides are accessible. The authors also provide causal evidence that DLMs implicitly compute the global fraction of masked tokens as an internal timestep signal, without explicit timestep embeddings.
Related guides (1)
Related events (8)
Trajectory Analysis of Masked Diffusion LMs for Graph-to-Text Generation with Lambda-Scaled Structural Decoding
This paper presents the first systematic study of masked diffusion language models (MDLMs) for graph-to-text generation, analyzing the order in which tokens are unmasked during iterative decoding. The authors find MDLMs naturally unmask entities first, then relational/function words, then structural tokens—a pattern disrupted by supervised fine-tuning, which prematurely anchors structural tokens and causes hallucination or omission. They propose lambda-scaled structural decoding, a training-free inference-time fix that recovers +9.4 BLEU-4, and introduce Graph-LLaDA, which integrates a Graph Transformer encoder into LLaDA's decoding process. Cross-dataset evaluation on the LAGRANGE benchmark shows prior baselines overfit to dataset-specific patterns while MDLM-based approaches generalize better.
Knowledge editing via locate-then-edit transferred to masked diffusion language models, revealing multi-token failure mode
A new arXiv paper investigates whether locate-then-edit knowledge editing methods, developed for autoregressive models, transfer to masked diffusion language models (MDMs) such as LLaDA and Dream. The authors find that causal tracing identifies the same early-to-mid-layer MLP location in both paradigms, but MDMs degrade systematically on multi-token edits due to partially unmasked intermediate states that the edit was never optimized for. A correction targeting these intermediate states substantially restores multi-token editing performance. The work is the first systematic comparison of knowledge editing across autoregressive and diffusion-based language model paradigms.
Mask-Aware Policy Gradients improve RL training for Masked Diffusion Language Models
A new arXiv preprint introduces a two-stage action MDP formalization for applying reinforcement learning to Masked Diffusion Language Models (MDLMs), decomposing the policy gradient into a token prediction term and a masking order term. Prior approaches ignored the position-unmasking decision, leading to intractable log-likelihood estimates; the proposed method optimizes both terms jointly. The approach achieves 87.1% on GSM8K and 53.4% on MBPP, claiming state-of-the-art results for MDLM-based reasoning and coding.
Survey on efficient inference techniques for masked diffusion large language models
A new arXiv survey introduces a unified latency decomposition framework for diffusion large language models (dLLMs) to systematically analyze inference efficiency. The authors categorize acceleration techniques across three axes: algorithmic innovations, architectural and system optimizations, and inference-time scaling. The paper addresses the gap between dLLMs' theoretical parallel-generation advantage and practical deployment speedups, and provides benchmarking guidelines for reproducible comparisons.
AdaLook: Adaptive multi-step lookahead decoding for masked diffusion language models
Researchers introduce AdaLook, an adaptive lookahead decoding framework for masked diffusion language models (DLMs) that dynamically determines rollout depth based on candidate-score variance rather than using fixed-depth lookahead. The method also enables branch expansion at informative intermediate states, avoiding unnecessary computation while improving exploration. Experiments across multiple benchmarks show AdaLook achieves a better accuracy-to-decoding-steps trade-off than existing one-step lookahead methods. The work addresses a known limitation of parallel text generation via DLMs, which are a non-autoregressive alternative to standard transformer decoding.
AGDO: Attention-guided denoising and optimization framework improves diffusion language model reasoning
Researchers propose AGDO, a framework that replaces random masking in diffusion large language models (dLLMs) with attention-guided denoising order and token weighting during fine-tuning and reinforcement learning. The work is motivated by an empirical finding that tokens with stronger attention to unmasked context are more stable and critical for reasoning. Experiments on math and coding benchmarks show AGDO outperforms existing post-training methods for dLLMs, advancing the case for attention-aware training in parallel-decoding language models.
Looped Diffusion Language Models (LoopMDM): Depth Scaling via Layer Looping
LoopMDM introduces selective looping of early-middle transformer layers in masked diffusion language models, achieving a depth-scaling effect without adding parameters. The approach matches same-size MDM performance with up to 3.3× fewer training FLOPs and outperforms deeper non-looped MDMs on reasoning benchmarks, including up to 8.5 points improvement on GSM8K. Inference-time compute scaling is enabled by varying loop counts, with adaptive loop scheduling providing additional efficiency gains. Attention analysis suggests looping works by promoting interactions among masked token positions.
Theoretical framework reveals low-dimensional invariant manifolds governing Transformer inductive reasoning dynamics
A new arXiv preprint presents a theoretical framework explaining how Transformers develop inductive reasoning abilities by proving that training dynamics of attention models are confined to a low-dimensional invariant manifold. The framework unifies several synthetic inductive tasks (in-context n-grams, multi-hop reasoning) and characterizes how data statistics govern competition between in-context and in-weights learning. The authors show that random initializations determine which circuit 'wins' when multiple solutions exist, and that the manifold's coordinate frame can automatically detect learned circuits in trained models. The work advances mechanistic interpretability by casting circuit formation as a tractable low-dimensional dynamical phenomenon.