4arXiv cs.CL (Computation and Language)·
L1 Augmented Attention improves Transformer similarity metric with 14.5% perplexity reduction
A new arXiv preprint proposes L1 augmented attention, a modification to scaled dot product attention that subtracts a learned, head-specific L1 distance between queries and keys from the dot product score. The hybrid metric captures complementary geometric information — directional alignment from dot product and coordinate deviation penalties from L1 — using low-dimensional projections to keep computation parallelizable. Evaluated on WikiText-2 with a compact transformer, the method achieves up to 14.5% perplexity reduction over the standard baseline and outperforms an RBF/L2 kernel alternative. Analysis reveals distinct geometric roles across layers and strong head-level specialization.
Related guides (1)
Related events (8)
Analysis-driven transformer linearization outperforms prior baselines on LLaMA and Qwen up to 32B
A new arXiv paper analyzes why post-hoc linearization of causal self-attention degrades model quality, identifying key-dependent rank-1 orthogonal projections as the mechanism softmax relies on and explaining why delta-style networks outperform gated accumulation. The authors introduce structural interventions—sink tokens, short convolutions, and fixed-budget cache routing—applied in a frozen-backbone regime. Scaling across LLaMA and Qwen models up to 32B parameters, the approach outperforms prior post-hoc linearization baselines on MMLU and matches long-context retrieval of adaptive-caching frameworks.
FourierQK: Spectral preprocessing of query-key projections improves transformer attention by up to 79%
A new arXiv preprint introduces FourierQK, a method applying FFT-based spectral preprocessing to learned query-key projections in transformer attention. On the TinyShakespeare character-level benchmark, four learned frequencies spanning paragraph-to-word scales achieve a 79% validation loss reduction over standard dot-product attention. The paper distinguishes itself from FNet by preserving the full attention score structure and only modifying Q/K projections, and identifies an architectural boundary between bilateral spectral attention and causal spectral attention. A companion paper (MorletQK) addresses the causal variant for word-scale tokenization.
Lie-Algebra Attention: tokens as bare matrix Lie group elements with closed-form geometric scores
A new arXiv preprint introduces Lie-Algebra Attention, an attention mechanism where tokens are elements of a matrix Lie group rather than feature vectors, with pairwise attention scores computed as the closed-form algebra norm of the relative pose (log of the group inverse product). The construction achieves equivariance tautologically without representation-theoretic machinery such as irreps, spherical harmonics, or Clebsch-Gordan products, and extends to non-compact affine groups that existing methods cannot handle. Experiments on SE(2), SO(3), and Aff(2) sequence-completion tasks show the closed-form score matches or outperforms learned MLP kernels while using 50–80x fewer score parameters.
Differential Transformer V2
Microsoft has published a blog post on Hugging Face introducing Differential Transformer V2, an updated version of their differential attention mechanism for transformers. The differential attention architecture aims to reduce attention noise by computing attention as a difference between two softmax attention maps. This post likely covers improvements to the original design, training dynamics, or scaling behavior of the V2 iteration.
Dynamic short convolutions yield 1.33–1.60× compute advantage over standard Transformers
A new arXiv preprint introduces dynamic short convolutions as an architectural primitive for Transformers, using input-dependent filters to combine locality bias with increased expressivity. Experiments across 150M–2B parameter language models show consistent perplexity improvements over standard Transformers and static convolution variants, with scaling-law fits indicating a 1.33× compute advantage when applied to key/query/value vectors and 1.60× when added after every linear layer. The technique also improves linear RNNs (Mamba-2, Gated DeltaNet) and mixture-of-experts architectures, with custom Triton kernels making training practical.
HOLA adds hippocampal exact KV cache to linear attention, closing gap with full-attention Transformers
HOLA (Hippocampal Linear Attention) augments linear-attention and state-space models with a bounded exact key-value cache inspired by Complementary Learning Systems theory, addressing the lossy compression problem that causes earlier facts to be overwritten in recurrent states. The cache uses a residual-based eviction criterion (large beta * ||e||) without a learned eviction module, and a decoupled RMSNorm-gamma read for sharp retrieval. At 340M parameters trained on 15B SlimPajama tokens, HOLA reduces Wikitext perplexity from 27.32 to 22.92, falling below a full-attention Transformer++ baseline, and shows strong needle-in-a-haystack recall out to 32k tokens despite training only at 2k. The work is directly relevant to the open question of whether linear-attention models can match full-attention on long-context retrieval tasks.
FlashMorph: Learned layer selection for converting Transformers to hybrid attention models
This arXiv paper introduces FlashMorph, a method for converting standard Transformer models into hybrid attention architectures by optimally selecting which layers retain full attention versus linear attention. Rather than using heuristic placement patterns, FlashMorph frames layer selection as a budget-constrained subset optimization, jointly learning layerwise gates on synthetic long-context retrieval data with a linearization regularization term. Experiments show FlashMorph finds more effective hybrid configurations that preserve long-context recall and general benchmark performance while reducing selection cost compared to prior methods. The work addresses a practical efficiency problem in deploying long-context models at scale.
Functional Attention: Reinterpreting Attention as Functional Correspondences for Operator Learning
This paper introduces Functional Attention, a novel attention mechanism for operator learning that replaces standard softmax token-wise affinities with structured linear operators inspired by geometric functional maps. The approach treats attention as a correspondence between adaptive bases rather than discrete tokens, yielding a resolution-invariant, globally-aware representation. Experiments show competitive or state-of-the-art performance on PDE solving, 3D segmentation, and regression tasks, with robustness to varying discretizations.