Entity · paper

Super Weights in LLMs and the Failure of Selective Training

paperactiveprovisionalsuper-weights-in-llms-and-the-failure-of-selective-training-13a495a8·1 events·first seen 6h ago

Aliases: Super Weights in LLMs and the Failure of Selective Training

Co-occurring entities

More like this (12)

Fast Weights Continual LLM Upcycling: A Predictor-Gated Bank-Wise Sparsity Training Recipe for Dense-to-Sparse LLMs Training LLMs to Enforce Multi-Level Instruction Hierarchies via Gravity-Weighted Direct Preference Optimization PC Layer: Polynomial Weight Preconditioning for Improving LLM Pre-Training Program-as-Weights Program-as-Weights On The Effectiveness-Fluency Trade-Off In LLM Conditioning: A Systematic Study Backdoor Unlearning Generalization: A Path Toward the Removal of Unknown Triggers in LLMs Mechanism-Driven Monitors for Preemptive Detection of LLM Training Instability Which Models Are Our Models Built On? Auditing Invisible Dependencies in Modern LLMs The Illusion of Equivalency: Statistical Characterization of Quantization Effects in LLMs LearnWeak

Recent events (1)

6arXiv · cs.LG·6h ago·source ↗

Super Weights in LLMs are not selectively trainable despite high importance

A new arXiv paper investigates Super Weights — individual LLM parameters whose removal catastrophically degrades performance — and finds that their apparent importance does not translate into trainability. Training Super Weights in isolation (100 to 8,192 parameters) collapses accuracy to random-guessing on OLMo-1B and OLMo-7B, while training an equal number of randomly chosen parameters in the same layers improves over baseline. LoRA, which applies structured low-rank updates across entire layers, succeeds with only 0.16% of parameters, and constraining LoRA updates at Super Weight coordinates yields no benefit. The findings challenge the assumption that parameter importance implies parameter trainability and suggest effective fine-tuning requires structured decompositions over full layers rather than targeted sparse updates.

Evaluation and Benchmarking Alignment and RLHF OLMoE-1B-7B LoRA OLMo-1B +1 more