MorletQK

FourierQK: Spectral preprocessing of query-key projections improves transformer attention by up to 79%

A new arXiv preprint introduces FourierQK, a method applying FFT-based spectral preprocessing to learned query-key projections in transformer attention. On the TinyShakespeare character-level benchmark, four learned frequencies spanning paragraph-to-word scales achieve a 79% validation loss reduction over standard dot-product attention. The paper distinguishes itself from FNet by preserving the full attention score structure and only modifying Q/K projections, and identifies an architectural boundary between bilateral spectral attention and causal spectral attention. A companion paper (MorletQK) addresses the causal variant for word-scale tokenization.