Entity · model

OLMo-1B

modelactiveprovisionalolmo-1b-943b4596·1 events·first seen 6h ago

Aliases: OLMo-1B

Co-occurring entities

OLMoE-1B-7B LoRA Super Weights in LLMs and the Failure of Selective Training

More like this (12)

OLMo-3 OLMoE-1B-7B OLMoE-1B-7B-0924 OLMo2 OLMo OLMoE LM1B LLaMA-Omni LLaVA-1.5-7B LLaVA-1.5-13B CO-LMLM LLaDA-1.5-8B

Recent events (1)

6arXiv · cs.LG·6h ago·source ↗

Super Weights in LLMs are not selectively trainable despite high importance

A new arXiv paper investigates Super Weights — individual LLM parameters whose removal catastrophically degrades performance — and finds that their apparent importance does not translate into trainability. Training Super Weights in isolation (100 to 8,192 parameters) collapses accuracy to random-guessing on OLMo-1B and OLMo-7B, while training an equal number of randomly chosen parameters in the same layers improves over baseline. LoRA, which applies structured low-rank updates across entire layers, succeeds with only 0.16% of parameters, and constraining LoRA updates at Super Weight coordinates yields no benefit. The findings challenge the assumption that parameter importance implies parameter trainability and suggest effective fine-tuning requires structured decompositions over full layers rather than targeted sparse updates.

Evaluation and Benchmarking Alignment and RLHF OLMoE-1B-7B LoRA OLMo-1B +1 more