5arXiv cs.CL (Computation and Language)·
Mobius Learning: cyclic depth folding enables depth-role superposition in Transformers
Researchers introduce Mobius Learning, a training architecture where different data streams follow cyclically shifted block orders in a Transformer, forcing each block group to be optimized in both shallow and deep representational roles — a property they call depth-role superposition. Experiments with a modified GPT-2 small (124M) trained on 2.5B FineWeb tokens show lower validation loss than a fixed-order looped Transformer at larger numbers of block-sequence passes. The architecture is also naturally suited to memory-constrained distributed training, as each worker stores only one block group rather than the full model stack.
Related guides (1)
Related events (8)
Loopie: Looped MoE Transformer achieves gold-medal performance at IMO and IPhO
Researchers introduce Loopie, a looped Transformer series using Mixture-of-Experts architecture in two sizes (20B/2B active and 6B/0.6B active parameters). The work directly addresses the longstanding challenge that scaling parameters outperforms looping under equal compute budgets, demonstrating that Loopie substantially outperforms vanilla Transformer baselines at the same compute. A novel post-training pipeline yields strong reasoning capabilities, with Loopie achieving gold-medal performance at the 2025 IMO and IPhO competitions without tool use.
Looped Diffusion Language Models (LoopMDM): Depth Scaling via Layer Looping
LoopMDM introduces selective looping of early-middle transformer layers in masked diffusion language models, achieving a depth-scaling effect without adding parameters. The approach matches same-size MDM performance with up to 3.3× fewer training FLOPs and outperforms deeper non-looped MDMs on reasoning benchmarks, including up to 8.5 points improvement on GSM8K. Inference-time compute scaling is enabled by varying loop counts, with adaptive loop scheduling providing additional efficiency gains. Attention analysis suggests looping works by promoting interactions among masked token positions.
Theoretical analysis reframes Transformer architecture design as rank-preservation tradeoffs across depth
A new arXiv preprint investigates how components of the Transformer feedforward block — skip connections, normalization placement, and two-matrix expansion/contraction structure — determine how much rank survives across depth at initialization. The authors reinterpret skip connections and normalization not merely as magnitude controllers but as mechanisms for preserving gradient rank, unifying the Pre-Norm vs. Post-Norm literature under a single rank-collapse framework. They derive that width expansion between the two MLP matrices follows a Marchenko-Pastur law to maintain full branch Jacobian rank, and show that the initialization rank of the input-output Jacobian predicts trainability on CIFAR-10. The work recasts deep network architecture design as navigating a three-way tradeoff among rank collapse, ensemble-like behavior, and parameter count.
2D-RoPE positional encoding enables frontier-length exact copying in Transformers
A new arXiv preprint demonstrates that frontier LLMs systematically fail at exact string copying within their context windows, attributing the failure to 1D positional encodings that encourage shortcut matching of local contexts rather than precise position retrieval. The authors introduce 2D-RoPE, which assigns each token a row and column ID in a 2D grid, making copying equivalent to a fixed column-offset lookup. Shallow Transformers with 2D-RoPE achieve perfect copying at lengths hundreds of times beyond training distribution, and the advantage persists in large-scale pretraining on DCLM up to 1.4B parameters. The result challenges assumptions about what frontier models can reliably do and proposes a concrete architectural modification to positional encoding.
Training-Free Looped Transformers: Inference-Time Recurrence via ODE-Motivated Layer Reapplication
The paper introduces a method to retrofit recurrence onto frozen pretrained transformer checkpoints at inference time by looping a contiguous mid-stack block of layers without any fine-tuning or architectural changes. Naive block reapplication degrades performance, so the authors motivate their approach by treating pre-norm transformer blocks as forward Euler ODE steps and replacing one large update with smaller damped sub-steps. Evaluated across seven model families including dense, sparse MoE, and MLA+MoE architectures, the method yields consistent benchmark improvements (e.g., +2.64 pp on MMLU-Pro for Qwen3-4B-Instruct) at no training cost.
The Reformer - Pushing the limits of language modeling
This Hugging Face blog post covers the Reformer, a memory-efficient transformer architecture that uses locality-sensitive hashing (LSH) attention and reversible residual layers to handle very long sequences. The post explains the technical mechanisms that allow Reformer to process sequences up to 1 million tokens with significantly reduced memory footprint compared to standard transformers. It serves as an educational deep-dive into the architectural innovations introduced in the original Reformer paper by Kitaev et al.
FlashMorph: Learned layer selection for converting Transformers to hybrid attention models
This arXiv paper introduces FlashMorph, a method for converting standard Transformer models into hybrid attention architectures by optimally selecting which layers retain full attention versus linear attention. Rather than using heuristic placement patterns, FlashMorph frames layer selection as a budget-constrained subset optimization, jointly learning layerwise gates on synthetic long-context retrieval data with a linearization regularization term. Experiments show FlashMorph finds more effective hybrid configurations that preserve long-context recall and general benchmark performance while reducing selection cost compared to prior methods. The work addresses a practical efficiency problem in deploying long-context models at scale.
Theoretical framework reveals low-dimensional invariant manifolds governing Transformer inductive reasoning dynamics
A new arXiv preprint presents a theoretical framework explaining how Transformers develop inductive reasoning abilities by proving that training dynamics of attention models are confined to a low-dimensional invariant manifold. The framework unifies several synthetic inductive tasks (in-context n-grams, multi-hop reasoning) and characterizes how data statistics govern competition between in-context and in-weights learning. The authors show that random initializations determine which circuit 'wins' when multiple solutions exist, and that the manifold's coordinate frame can automatically detect learned circuits in trained models. The work advances mechanistic interpretability by casting circuit formation as a tractable low-dimensional dynamical phenomenon.