induction-in-both-directions-a-mechanistic-analysis-of-in-context-learning-in-masked-diffusion-language-models-bf674c22·1 events·first seen Aliases: Induction in Both Directions: A Mechanistic Analysis of In-Context Learning in Masked Diffusion Language Models
A new arXiv preprint performs mechanistic interpretability analysis on masked diffusion language models (DLMs), identifying a bidirectional induction circuit that enables in-context learning by leveraging both past and future context around masked tokens. The study compares attention-only autoregressive transformers with absorbing-mask DLMs under matched architectures, finding that DLMs do not outperform AR models when only left context is visible but show stronger induction when both sides are accessible. The authors also provide causal evidence that DLMs implicitly compute the global fraction of masked tokens as an internal timestep signal, without explicit timestep embeddings.