6arXiv cs.AI (Artificial Intelligence)·
Theoretical analysis reframes Transformer architecture design as rank-preservation tradeoffs across depth
A new arXiv preprint investigates how components of the Transformer feedforward block — skip connections, normalization placement, and two-matrix expansion/contraction structure — determine how much rank survives across depth at initialization. The authors reinterpret skip connections and normalization not merely as magnitude controllers but as mechanisms for preserving gradient rank, unifying the Pre-Norm vs. Post-Norm literature under a single rank-collapse framework. They derive that width expansion between the two MLP matrices follows a Marchenko-Pastur law to maintain full branch Jacobian rank, and show that the initialization rank of the input-output Jacobian predicts trainability on CIFAR-10. The work recasts deep network architecture design as navigating a three-way tradeoff among rank collapse, ensemble-like behavior, and parameter count.
Related guides (2)
Related events (8)
Theoretical framework reveals low-dimensional invariant manifolds governing Transformer inductive reasoning dynamics
A new arXiv preprint presents a theoretical framework explaining how Transformers develop inductive reasoning abilities by proving that training dynamics of attention models are confined to a low-dimensional invariant manifold. The framework unifies several synthetic inductive tasks (in-context n-grams, multi-hop reasoning) and characterizes how data statistics govern competition between in-context and in-weights learning. The authors show that random initializations determine which circuit 'wins' when multiple solutions exist, and that the manifold's coordinate frame can automatically detect learned circuits in trained models. The work advances mechanistic interpretability by casting circuit formation as a tractable low-dimensional dynamical phenomenon.
Review Residuals: input-dependent gating for transformer residual connections shows scale-emergent gains
A new arXiv preprint introduces Review Residuals, a mechanism that replaces fixed-coefficient residual connections in transformers with a learned, input-dependent gate conditioned on both the current hidden state and the proposed sublayer update. Experiments across five model sizes (60M–1B parameters) find no advantage at small scale but statistically significant improvements over standard residuals and Highway-style gates at 590M and 1B parameters, with the benefit growing with scale. A key finding is that convex (Highway-style) gating reintroduces vanishing gradients beyond ~20 layers, while the additive identity-preserving form trains stably at all tested depths.
Analysis-driven transformer linearization outperforms prior baselines on LLaMA and Qwen up to 32B
A new arXiv paper analyzes why post-hoc linearization of causal self-attention degrades model quality, identifying key-dependent rank-1 orthogonal projections as the mechanism softmax relies on and explaining why delta-style networks outperform gated accumulation. The authors introduce structural interventions—sink tokens, short convolutions, and fixed-budget cache routing—applied in a frozen-backbone regime. Scaling across LLaMA and Qwen models up to 32B parameters, the approach outperforms prior post-hoc linearization baselines on MMLU and matches long-context retrieval of adaptive-caching frameworks.
Single transformer layer training can match full-parameter RL post-training in LLMs
A new arXiv paper challenges the assumption that all transformer layers contribute equally during RL post-training, finding that training a single layer can recover most or all of the gains from full-parameter RL. The authors introduce a 'layer contribution' metric and evaluate across seven models from the Qwen2.5 and Qwen3 families, three RL algorithms (GRPO, GiGPO, Dr. GRPO), and tasks including math reasoning, code, and agentic decision-making. A consistent structural pattern emerges: high-contribution layers concentrate in the middle of the transformer stack, and this ranking is stable across datasets, tasks, and algorithms.
State-Prediction Separation Hypothesis proposes dual-stream Transformer variant with consistent efficiency gains
A new arXiv preprint introduces the state-prediction separation hypothesis, arguing that Transformers conflate two distinct roles—next-token prediction and state storage for future predictions—within a single computation stream. The authors design a dual-stream Transformer variant that disentangles these functions and report consistent improvements in data and compute efficiency across multiple pretraining scales, outperforming standard Transformers by 2–3 percentage points on average on downstream tasks. Gradient analysis is provided to argue the architectural difference is fundamental rather than incidental.
Dynamic short convolutions yield 1.33–1.60× compute advantage over standard Transformers
A new arXiv preprint introduces dynamic short convolutions as an architectural primitive for Transformers, using input-dependent filters to combine locality bias with increased expressivity. Experiments across 150M–2B parameter language models show consistent perplexity improvements over standard Transformers and static convolution variants, with scaling-law fits indicating a 1.33× compute advantage when applied to key/query/value vectors and 1.60× when added after every linear layer. The technique also improves linear RNNs (Mamba-2, Gated DeltaNet) and mixture-of-experts architectures, with custom Triton kernels making training practical.
Local linear structures in LLM weights and activations are dynamic, not fixed global directions
A new arXiv paper investigates the nature of linear structures in transformer weights and activations, finding strong local low-rank task-gradient structure but rejecting the hypothesis that fixed task planes exist. The authors show that useful bases drift substantially within 100 optimization steps, yet early recovery updates form a trajectory-prefix basis capturing 77% of LoRA recovery displacement. They also establish a formal connection between parameter perturbations and activation steering, finding a 0.58 cosine similarity between gradient-step-induced activation shifts and CAA steering vectors, suggesting linear structures are evolving local geometries rather than stable global task directions.
Variable-Width Transformers: X-shaped architecture outperforms uniform-width baselines with 22% fewer FLOPs
Researchers propose the ><former (X-shaped transformer), a decoder-only architecture that uses wider early and late layers with narrower middle layers, implemented via a parameter-free residual resizing mechanism. Evaluated on models from 200M to 2B dense parameters and 3B MoE, the architecture consistently outperforms parameter-matched uniform-width baselines on language modeling loss. The design yields a 22% reduction in FLOPs and 15% reduction in KV cache memory under fitted scaling curves, suggesting nonuniform width allocation is a viable path to more compute-efficient language models.