Entity · model

Llama 1B

modelactivellama-1b-a373f7aa·1 events·first seen Jun 5, 2026

Aliases: Llama 1B

Co-occurring entities

AdamW PC Layer: Polynomial Weight Preconditioning for Improving LLM Pre-Training Muon

More like this (12)

Llama 3.2 Llama-3 Llama Guard 3 1B Llama3-8B Llama Llama 2 70B Llama-3.2-1B-Instruct Llama 2 Llama 3.1 70B Llama 3.3 70B Instruct Llama-3.1-8B Llama 3

Recent events (1)

4arXiv · cs.LG·Jun 5, 2026·source ↗

PC Layer: Polynomial weight preconditioning for stable LLM pre-training

Researchers propose a PC (preconditioning) layer that applies polynomial preconditioning to reshape the singular-value spectrum of weight matrices during LLM training, improving conditioning stability. The preconditioned weights merge back into the original architecture at inference time with no overhead. Experiments on Llama-1B pre-training show advantages over standard transformers for both AdamW and Muon optimizers, with theoretical convergence guarantees for deep linear networks.

Training Infrastructure AdamW PC Layer: Polynomial Weight Preconditioning for Improving LLM Pre-Training Llama 1B +1 more