Entity · paper

FourierQK: Spectral Preprocessing of Query-Key Projections Improves Transformer Attention

paperactiveprovisionalfourierqk-spectral-preprocessing-of-query-key-projections-improves-transformer-attention-8fb9b1e8·1 events·first seen 4d ago

Aliases: FourierQK: Spectral Preprocessing of Query-Key Projections Improves Transformer Attention

Co-occurring entities

MorletQK FNet

More like this (12)

Spectral Attention Do transformers need three projections? Systematic study of QKV variants Dynamic Short Convolutions Improve Transformers Quantisation-Aware Training (QAT)Listening with Attention: Entropy-Guided Explainability for Transformer-Based Audio Models The Key to Going Linear: Analysis-Driven Transformer Linearization Sparse Transformer Graph Convolutional Attention: A Spectral Perspective on Graph Denoising and Diffusion Graph Convolutional Attention: A Spectral Perspective on Graph Denoising and Diffusion A Physics-Informed Fourier-Wavelet Transformer for Multiscale Computational Fluid Dynamics Surrogate Modeling Attention Expansion: Enhancing Keyphrase Extraction from Long Documents with Attention-Augmented Contextualized Embeddings permutation-invariant transformers

Recent events (1)

4arXiv · cs.CL·4d ago·source ↗

FourierQK: Spectral preprocessing of query-key projections improves transformer attention by up to 79%

A new arXiv preprint introduces FourierQK, a method applying FFT-based spectral preprocessing to learned query-key projections in transformer attention. On the TinyShakespeare character-level benchmark, four learned frequencies spanning paragraph-to-word scales achieve a 79% validation loss reduction over standard dot-product attention. The paper distinguishes itself from FNet by preserving the full attention score structure and only modifying Q/K projections, and identifies an architectural boundary between bilateral spectral attention and causal spectral attention. A companion paper (MorletQK) addresses the causal variant for word-scale tokenization.

Evaluation and Benchmarking MorletQK FNet FourierQK: Spectral Preprocessing of Query-Key Projections Improves Transformer Attention