3arXiv cs.CL (Computation and Language)·
Spectral initialization schemes for LM pretraining show diagnostic value but no performance gain
Researchers analyze weight spectra across eleven pretrained GPT-2-style checkpoints varying in size, language, and training corpus, finding consistent depth-wise patterns in Frobenius norm and effective-rank entropy. They construct initialization schemes that mimic these spectral profiles and compare them against standard initialization methods. Despite visibly altering structural spectral patterns, the proposed initializers do not yield performance improvements over pretrained-weight reuse. The results suggest pretrained spectra are useful diagnostics of model structure but that coarse spectral matching is insufficient for effective transfer.
Related guides (1)
Related events (8)
PC Layer: Polynomial weight preconditioning for stable LLM pre-training
Researchers propose a PC (preconditioning) layer that applies polynomial preconditioning to reshape the singular-value spectrum of weight matrices during LLM training, improving conditioning stability. The preconditioned weights merge back into the original architecture at inference time with no overhead. Experiments on Llama-1B pre-training show advantages over standard transformers for both AdamW and Muon optimizers, with theoretical convergence guarantees for deep linear networks.
Super Weights in LLMs are not selectively trainable despite high importance
A new arXiv paper investigates Super Weights — individual LLM parameters whose removal catastrophically degrades performance — and finds that their apparent importance does not translate into trainability. Training Super Weights in isolation (100 to 8,192 parameters) collapses accuracy to random-guessing on OLMo-1B and OLMo-7B, while training an equal number of randomly chosen parameters in the same layers improves over baseline. LoRA, which applies structured low-rank updates across entire layers, succeeds with only 0.16% of parameters, and constraining LoRA updates at Super Weight coordinates yields no benefit. The findings challenge the assumption that parameter importance implies parameter trainability and suggest effective fine-tuning requires structured decompositions over full layers rather than targeted sparse updates.
PEFT-Arena: Benchmarking Parameter-Efficient Finetuning via Stability-Plasticity Trade-offs
PEFT-Arena is a new benchmark that evaluates parameter-efficient finetuning methods jointly on downstream task performance and retention of pretrained general capabilities, framing the problem as a stability-plasticity dilemma. Across methods tested under comparable parameter budgets, orthogonal finetuning achieves the best Pareto frontier. The paper provides geometric analyses in both weight space (spectral/singular-value structure) and activation space (representation distortion metrics) to explain why different PEFT methods differ in forgetting behavior. A practical finding is that final SFT checkpoints often overshoot an optimal retention operating point, motivating path-wise rewinding as a post-hoc correction.
Super-Tuning: Pruning saliency signals repurposed for sparse parameter-efficient fine-tuning
Researchers propose Super and Supra, two sparse PEFT methods that reuse activation-weighted magnitude scores (Wanda-style) originally developed for pruning to select which parameters to update during fine-tuning. Supra combines this sparse update with LoRA under a fixed parameter budget via a budget-splitting rule. Experiments on Llama-3.2-1B and Llama-3-8B on a Math17K arithmetic task show the best Super/Supra variants outperform other tested adapter configurations. The work suggests pruning-inspired orderings are a useful, low-cost signal for identifying effective sparse fine-tuning supports.
FourierQK: Spectral preprocessing of query-key projections improves transformer attention by up to 79%
A new arXiv preprint introduces FourierQK, a method applying FFT-based spectral preprocessing to learned query-key projections in transformer attention. On the TinyShakespeare character-level benchmark, four learned frequencies spanning paragraph-to-word scales achieve a 79% validation loss reduction over standard dot-product attention. The paper distinguishes itself from FNet by preserving the full attention score structure and only modifying Q/K projections, and identifies an architectural boundary between bilateral spectral attention and causal spectral attention. A companion paper (MorletQK) addresses the causal variant for word-scale tokenization.
TailLoR: Spectral-domain continual learning via protected principal components
TailLoR is a new parameter-efficient finetuning method for continual learning that uses the singular value decomposition of pre-trained weights as a fixed reference frame, applying low-rank updates only to the singular value matrix. A soft spectral penalty discourages updates aligned with dominant singular directions, reducing catastrophic interference while routing adaptation into long-tail spectral coordinates. The approach targets the forgetting problem in continual learning through a principled spectral lens.
Hyperfitting Explained: Terminal Geometric Expansion in Final Transformer Layers Drives Diversity Gains
This paper investigates the 'hyperfitting' phenomenon—where fine-tuning LLMs to near-zero loss on small datasets improves open-ended generation and reduces repetition—and demonstrates it is mechanistically distinct from temperature scaling. Entropy-matched control experiments falsify both the temperature-equivalence and static vocabulary reweighting hypotheses, instead localizing the effect to a 'Terminal Expansion' in the final transformer block where feature-space dimensionality expands by ~80.8 dimensions, enabling promotion of deep-tail tokens via context-dependent rank reordering. The authors introduce Late-Stage LoRA, a targeted fine-tuning strategy updating only the final 5 layers, achieving robust generation with minimal parameter updates.
Predictor-gated bank-wise sparsity recipe for dense-to-sparse LLM upcycling from Qwen2.5-8B
A new arXiv preprint introduces a continual training recipe to convert dense LLMs into channel-sparse models without post-hoc pruning. Starting from a Qwen2.5-8B checkpoint, the method uses a low-rank predictor to gate FFN channel routing, achieving 4x sparsity in FFN intermediate activations via a bank-wise top-k rule at 32K context. The routing module is trained on the main language modeling path, making the resulting sparsity hardware-oriented rather than approximate. The authors also identify and patch a layer-local long-context failure mode on the RULER-CWE benchmark.