Express: Efficient causal attention approximation with formal guarantees and FlashAttention 2 speedups
A new tool called Express converts non-causal attention approximations into causal ones with matching theoretical guarantees, achieving log^(3/2)(n)/s approximation error with O(s) memory. Combined with the Thinformer approximation and an I/O-aware Triton implementation, it demonstrates substantial speedups over FlashAttention 2. The work targets four practical bottlenecks: long-context prefill, KV cache compression, and both memory- and compute-constrained long-form decoding.
Related guides (3)
Related events (8)
Nyströmformer: Approximating Self-Attention in Linear Time and Memory via the Nyström Method
This Hugging Face blog post covers Nyströmformer, a transformer variant that approximates standard self-attention using the Nyström method to achieve linear time and memory complexity. The approach addresses the quadratic scaling bottleneck of standard attention, enabling processing of longer sequences at reduced computational cost. The post likely covers the model's integration into the Hugging Face ecosystem and its practical use cases.
Improving Hugging Face Training Efficiency Through Packing with Flash Attention 2
Hugging Face published a blog post describing a technique for improving training efficiency by packing multiple short sequences into a single batch using Flash Attention 2. The approach reduces padding waste and improves GPU utilization during LLM fine-tuning. This is a practical infrastructure optimization relevant to practitioners training models on datasets with variable-length sequences.
CLSA: Cross-Layer Sparse Attention with Shared Routing for Efficient Long-Context Inference
Researchers propose Cross-Layer Sparse Attention (CLSA), a method that builds on KV-sharing architectures (like YOCO) to share both the KV cache and the routing index across decoder layers. A single indexer computes token-level top-k selection once and reuses it across layers, reducing routing overhead while preserving fine-grained selectivity. Experiments on short- and long-context benchmarks show up to 7.6x decoding speedup and 17.1x overall throughput improvement at 128K context, addressing pre-filling, KV-cache storage, and decoding bottlenecks simultaneously.
A Failed Experiment: Infini-Attention, and Why We Should Keep Trying?
A Hugging Face blog post documents an attempt to implement and validate Infini-Attention, a technique proposed to extend transformer context length by combining local and compressed global memory. The experiment reportedly failed to reproduce the claimed benefits, raising questions about the reproducibility and practical viability of the approach. The post frames the failure as instructive and argues for continued experimentation with long-context architectures.
Functional Attention: Reinterpreting Attention as Functional Correspondences for Operator Learning
This paper introduces Functional Attention, a novel attention mechanism for operator learning that replaces standard softmax token-wise affinities with structured linear operators inspired by geometric functional maps. The approach treats attention as a correspondence between adaptive bases rather than discrete tokens, yielding a resolution-invariant, globally-aware representation. Experiments show competitive or state-of-the-art performance on PDE solving, 3D segmentation, and regression tasks, with robustness to varying discretizations.
DashAttention: Differentiable and Adaptive Sparse Hierarchical Attention for Long-Context LLMs
DashAttention introduces a two-stage hierarchical sparse attention mechanism that replaces the fixed top-k block selection used in methods like NSA and InfLLMv2 with an adaptive α-entmax transformation, allowing a variable number of KV blocks to be selected per query. The approach keeps the full hierarchy differentiable by using the first-stage selection as a prior for second-stage softmax attention. Experiments show comparable accuracy to full attention at 75% sparsity with a better Pareto frontier than competing methods, and a Triton GPU implementation achieves meaningful speedup over FlashAttention-3 at inference time.
VaSE: Value-Aware Stochastic KV Cache Eviction improves reasoning model efficiency
A new arXiv preprint introduces Value-aware Stochastic KV Cache Eviction (VaSE), a training-free method for compressing KV caches in long-chain-of-thought reasoning models. The authors identify two key failure modes in prior eviction approaches — catastrophic repetition loops caused by evicting high-magnitude value states, and low cache diversity — and address both with targeted protections and stochastic eviction. On six reasoning tasks with Qwen3 models at 4x compression, VaSE outperforms the current best selection-based sparse attention method and exceeds the strongest eviction baseline by over 4%, while supporting FlashAttention2 and maintaining a static memory footprint.
COMPACT-VA: Planning-aligned token compression for long-context autonomous driving
Researchers introduce COMPACT-VA, a working memory framework using conditional VQ-VAE to compress extended temporal context in vision-action autonomous driving models. Compression is conditioned on historical trajectory and a learned planning intent derived from future trajectories during training, enabling end-to-end optimization without backbone modifications. On high-signal dynamic scenarios, the method achieves 68.3% success rate (>6% improvement) with 3.3x speedup and 2.7x memory reduction over uncompressed processing.