paper

PC Layer: Polynomial Weight Preconditioning for Improving LLM Pre-Training

paperactiveprovisionalpc-layer-polynomial-weight-preconditioning-for-improving-llm-pre-training-26751bd8·1 events·first seen 12d ago

Aliases: PC Layer: Polynomial Weight Preconditioning for Improving LLM Pre-Training

Co-occurring entities

AdamW Llama 1B Muon

More like this (12)

Continual LLM Upcycling: A Predictor-Gated Bank-Wise Sparsity Training Recipe for Dense-to-Sparse LLMs LLM Pretraining TailLoR: Protecting Principal Components in Parameter-Efficient Continual Learning CLP: Collocation-Length Prediction for Zero-Loss Adaptive Multi-Token Inference Leveraging Audio-LLMs to Filter Speech-to-Speech Training Data Alternating Token-Weighted Unlearning Training LLMs to Enforce Multi-Level Instruction Hierarchies via Gravity-Weighted Direct Preference Optimization q0: Primitives for Hyper-Epoch Pretraining KL-Cov regularization Layer-Adaptive Expert Pruning A sleep-like consolidation mechanism for LLMs On The Effectiveness-Fluency Trade-Off In LLM Conditioning: A Systematic Study

Recent events (1)

4arXiv · cs.LG·12d ago·source ↗

PC Layer: Polynomial weight preconditioning for stable LLM pre-training

Researchers propose a PC (preconditioning) layer that applies polynomial preconditioning to reshape the singular-value spectrum of weight matrices during LLM training, improving conditioning stability. The preconditioned weights merge back into the original architecture at inference time with no overhead. Experiments on Llama-1B pre-training show advantages over standard transformers for both AdamW and Muon optimizers, with theoretical convergence guarantees for deep linear networks.

Training Infrastructure AdamW PC Layer: Polynomial Weight Preconditioning for Improving LLM Pre-Training Llama 1B +1 more