Entity · paper

The Key to Going Linear: Analysis-Driven Transformer Linearization

paperactiveprovisionalthe-key-to-going-linear-analysis-driven-transformer-linearization-4d470b28·1 events·first seen 2d ago

Aliases: The Key to Going Linear: Analysis-Driven Transformer Linearization

Co-occurring entities

Qwen Llama MMLU

More like this (12)

Training-Free Looped Transformers Fixed-Point Reasoners: Stable and Adaptive Deep Looped Transformers Linear Diffusion Transformer Bridging the Gap Between Latent and Explicit Reasoning with Looped Transformers Variable-Width Transformers Review Residuals: Update-Conditioned Residual Gating for Transformers Transformer Language Models The Energy Consumption of Transformer Fine-Tuning: A Roofline-Inspired Scaling Model Feature-wise Linear Modulation Dynamic Short Convolutions Improve Transformers transformer architecture Graph Transformer

Recent events (1)

6arXiv · cs.LG·2d ago·source ↗

Analysis-driven transformer linearization outperforms prior baselines on LLaMA and Qwen up to 32B

A new arXiv paper analyzes why post-hoc linearization of causal self-attention degrades model quality, identifying key-dependent rank-1 orthogonal projections as the mechanism softmax relies on and explaining why delta-style networks outperform gated accumulation. The authors introduce structural interventions—sink tokens, short convolutions, and fixed-budget cache routing—applied in a frozen-backbone regime. Scaling across LLaMA and Qwen models up to 32B parameters, the approach outperforms prior post-hoc linearization baselines on MMLU and matches long-context retrieval of adaptive-caching frameworks.

Long Context Evolution Inference Economics Qwen The Key to Going Linear: Analysis-Driven Transformer Linearization Llama +1 more