Entity · paper

PC Layer: Polynomial Weight Preconditioning for Improving LLM Pre-Training

paperactivepc-layer-polynomial-weight-preconditioning-for-improving-llm-pre-training-26751bd8·1 events·first seen Jun 5, 2026

Aliases: PC Layer: Polynomial Weight Preconditioning for Improving LLM Pre-Training

Co-occurring entities

AdamW Llama 1B Muon

More like this (12)

PALS: Percentile-Aware Layerwise Sparsity for LLM Pruning Continual LLM Upcycling: A Predictor-Gated Bank-Wise Sparsity Training Recipe for Dense-to-Sparse LLMs In-Place Tokenizer Expansion for Pre-trained LLMs LLM Pretraining TailLoR: Protecting Principal Components in Parameter-Efficient Continual Learning One-Step Gradient Delay is Not a Barrier for Large-Scale Asynchronous Pipeline Parallel LLM Pretraining CLP: Collocation-Length Prediction for Zero-Loss Adaptive Multi-Token Inference Super Weights in LLMs and the Failure of Selective Training The Illusion of Equivalency: Statistical Characterization of Quantization Effects in LLMs AIR: Adaptive Interleaved Reasoning with Code in MLLMs Leveraging Audio-LLMs to Filter Speech-to-Speech Training Data Forecasting With LLMs: Improved Generalization Through Feature Steering

Recent events (1)

4arXiv · cs.LG·Jun 5, 2026·source ↗

PC Layer: Polynomial weight preconditioning for stable LLM pre-training

Researchers propose a PC (preconditioning) layer that applies polynomial preconditioning to reshape the singular-value spectrum of weight matrices during LLM training, improving conditioning stability. The preconditioned weights merge back into the original architecture at inference time with no overhead. Experiments on Llama-1B pre-training show advantages over standard transformers for both AdamW and Muon optimizers, with theoretical convergence guarantees for deep linear networks.

Training Infrastructure AdamW PC Layer: Polynomial Weight Preconditioning for Improving LLM Pre-Training Llama 1B +1 more