4arXiv cs.CL (Computation and Language)·
FourierQK: Spectral preprocessing of query-key projections improves transformer attention by up to 79%
A new arXiv preprint introduces FourierQK, a method applying FFT-based spectral preprocessing to learned query-key projections in transformer attention. On the TinyShakespeare character-level benchmark, four learned frequencies spanning paragraph-to-word scales achieve a 79% validation loss reduction over standard dot-product attention. The paper distinguishes itself from FNet by preserving the full attention score structure and only modifying Q/K projections, and identifies an architectural boundary between bilateral spectral attention and causal spectral attention. A companion paper (MorletQK) addresses the causal variant for word-scale tokenization.
Related guides (1)
Related events (8)
Systematic study questions whether transformers need all three QKV projections
An arXiv preprint investigates whether the standard query, key, and value projections in transformer attention are all necessary, conducting a systematic study of QKV variants. The work has attracted moderate community engagement on Hacker News (168 points, 34 comments). Results could inform more efficient attention architectures by potentially reducing parameter counts or computation.
QK-Restore: Fixing long-context recall degradation caused by CoT fine-tuning in hybrid LLMs
Researchers find that chain-of-thought supervised fine-tuning systematically degrades long-context recall in hybrid linear-attention models (HypeNet, Jet-Nemotron), with Needle-In-A-Haystack performance collapsing dramatically—e.g., HypeNet-9B dropping from 67.2% to 9.4% at 256K context. The root cause is identified as CoT-SFT biasing attention gradients toward short-range patterns, corrupting the query-key projections responsible for long-range routing. The paper proposes QK-Restore, a training-free fix that restores only W_Q and W_K from the pre-SFT checkpoint, recovering long-context capability while preserving reasoning gains.
Physics-informed Fourier-wavelet transformer improves multiscale CFD surrogate modeling
A new arXiv preprint introduces a physics-informed Fourier-wavelet transformer for next-step velocity-field reconstruction in computational fluid dynamics, combining hybrid spectral encoding with PDE-residual-guided self-attention and self-supervised pretraining. The model is evaluated on cylinder-wake and fluid-structure interaction benchmarks, achieving best-in-class normalized mean-squared error on both tasks and stronger recovery of localized flow structures compared to spectral, transformer, and physics-informed neural network baselines. The work targets the persistent gap between global flow pattern accuracy and fine-grained multiscale structure recovery in surrogate models.
Analysis-driven transformer linearization outperforms prior baselines on LLaMA and Qwen up to 32B
A new arXiv paper analyzes why post-hoc linearization of causal self-attention degrades model quality, identifying key-dependent rank-1 orthogonal projections as the mechanism softmax relies on and explaining why delta-style networks outperform gated accumulation. The authors introduce structural interventions—sink tokens, short convolutions, and fixed-budget cache routing—applied in a frozen-backbone regime. Scaling across LLaMA and Qwen models up to 32B parameters, the approach outperforms prior post-hoc linearization baselines on MMLU and matches long-context retrieval of adaptive-caching frameworks.
KnowsTFM: Knowledge graph-informed fine-tuning of small tabular foundation models
A new arXiv preprint introduces KnowsTFM, a method for fine-tuning small tabular foundation models (nanoscale TabPFN and TabICL variants) using structural attention priors derived from knowledge graphs and parameter-efficient low-rank updates. The approach targets niche domains with scarce, high-dimensional data shifted from pretraining distributions, showing meaningful gains in specialist settings but marginal gains on general tasks. The paper also reports that continual fine-tuning of frontier tabular models can trigger collapse of pretrained knowledge, a notable failure mode.
Q-target framework unifies supervised fine-tuning variants through target distribution design
A new arXiv preprint reframes supervised fine-tuning (SFT) as a problem of target distribution design rather than loss objective selection, introducing the Q-target framework that decomposes SFT supervision into two explicit choices: reliance on the observed token and allocation of remaining probability mass. The authors show that many existing SFT variants can be understood as implicit choices of this target distribution. They propose Target-SFT, which constructs training objectives directly from the desired target distribution, and report consistent improvements across ten reasoning dataset-model settings. The work offers a unifying theoretical lens and opens a broader design space for SFT objectives.
FreqDepthKV: Frequency-guided KV cache compression for long-context LLM inference
A new arXiv preprint introduces FreqDepthKV, an inference-time KV cache compression method that factorizes adjacent-layer KV states into shared low-frequency depth components and sparse high-frequency residuals. A lightweight online probe dynamically assigns attention heads to different caching modes without retraining. On a 32k-token prefill window, the method achieves a 3.9x compression ratio while closely matching full-KV accuracy across QA, retrieval, summarization, and code generation benchmarks, improving decoding throughput to 70.4 tokens/s and reducing peak KV memory to 6.2 GB.
FlashMorph: Learned layer selection for converting Transformers to hybrid attention models
This arXiv paper introduces FlashMorph, a method for converting standard Transformer models into hybrid attention architectures by optimally selecting which layers retain full attention versus linear attention. Rather than using heuristic placement patterns, FlashMorph frames layer selection as a budget-constrained subset optimization, jointly learning layerwise gates on synthetic long-context retrieval data with a linearization regularization term. Experiments show FlashMorph finds more effective hybrid configurations that preserve long-context recall and general benchmark performance while reducing selection cost compared to prior methods. The work addresses a practical efficiency problem in deploying long-context models at scale.