What it is
AdamW is an adaptive gradient optimizer that has become the de facto standard for training large language models. It extends Adam by decoupling weight decay from the gradient update: rather than folding L2 regularization into the adaptive gradient term (which distorts the effective decay rate), AdamW applies weight decay directly to the parameters. This seemingly small change substantially improves regularization behavior at scale and is the primary reason AdamW displaced vanilla Adam as the LLM training default.
The core mechanism is otherwise identical to Adam: it maintains per-parameter running estimates of the first moment (mean gradient) and second moment (uncentered variance), uses these to normalize the update step, and applies a global learning rate schedule on top. The result is an optimizer that adapts to the local curvature of each parameter independently, tolerates sparse gradients, and is relatively robust to learning rate choice — properties that matter enormously when training models with billions of heterogeneous parameters.
How it works
`` For each parameter θ at step t: m_t = β₁·m_{t-1} + (1-β₁)·g_t # first moment v_t = β₂·v_{t-1} + (1-β₂)·g_t² # second moment θ_t = θ_{t-1} - α·m̂_t/√(v̂_t+ε) # Adam update - α·λ·θ_{t-1} # decoupled weight decay ``
The decoupled weight decay term α·λ·θ is applied after the gradient step, not inside the adaptive normalization. This is the entire difference from Adam — but it means weight decay behaves as true L2 regularization regardless of the gradient magnitude, which matters most for parameters with small or sparse gradients (including embedding layers).
Why it matters — and where it's being stress-tested
AdamW's dominance is not in question, but its theoretical foundations and practical limits are under active scrutiny across several fronts.
Heavy-tailed gradient noise
A June 2026 preprint formally frames as an open problem whether AdamW can achieve rigorous convergence guarantees under heavy-tailed stochastic gradient noise — the kind of noise distribution that appears in practice during LLM pretraining. The concern is mechanistic: AdamW's second-moment accumulator may hide large gradient spikes by averaging them into the normalization denominator, potentially masking the signal needed for convergence. Sign-based optimizers like Lion and Muon already have sharp convergence rates under heavy-tailed noise; AdamW does not yet have an equivalent result, and the paper introduces a corridor lower-bound mechanism to characterize the potential failure mode.
Asynchronous pipeline parallelism
A separate line of work shows that gradient staleness — unavoidable in asynchronous pipeline parallelism (e.g., PipeDream-2BW) — is not a fundamental obstacle, but its impact is optimizer-dependent. Under one-step gradient delay, AdamW training degrades significantly, while Muon remains stable. Experiments at up to 10B parameters confirm this gap. An optimizer-agnostic Error Feedback correction can partially close it, but the finding establishes a concrete regime where AdamW is not the right default.
Hyperparameter transfer across scales
The well-known advantage of Maximal Update Parameterization (μP) over standard parameterization (SP) with AdamW has been traced to a single dominant factor: the embedding layer learning rate. Under SP, the embedding layer acts as a training bottleneck — its effective learning rate does not scale correctly with model width, causing instabilities that propagate through training. Scaling the embedding learning rate by model width to match μP's prescription substantially stabilizes AdamW training and improves hyperparameter transfer from small to large models. Weight decay, separately, affects scaling law fit quality and extrapolation robustness in opposite directions, creating a tradeoff practitioners must navigate explicitly.
Optimizer state transfer and gauge symmetry
Correctly transferring AdamW's optimizer state (its accumulated first and second moments) across fine-tuning runs or model merges requires respecting the underlying symmetry of the architecture. For RMSNorm transformers, the relevant symmetry group is the signed-permutation group B_d = S_d ⋉ {±1}^d — larger than the permutation-only S_d of LayerNorm models. Ignoring this when aligning checkpoints before state transfer leads to incorrect momentum estimates; sign-marginalized Hungarian matching recovers 91.1% of cross-run coordinates versus 60.3% for naive endpoint matching.
Variants and adjacent work
AdamO (2026) extends the AdamW design pattern to continual learning: it decouples isometry regularization — a mechanism for keeping layer-wise Jacobian singular values near one, which preserves plasticity under non-stationarity — from the gradient update, in the same spirit that AdamW decoupled weight decay from Adam. It consistently matches or outperforms prior plasticity-preserving methods on supervised and reinforcement-learning continual-learning benchmarks.
PC Layer (polynomial preconditioning) is a complementary approach that reshapes the singular-value spectrum of weight matrices before the optimizer sees them, improving conditioning for both AdamW and Muon on Llama-1B pretraining.
Quantisation-aware training (QAT) studies confirm that AdamW's learning rate schedule preferences are largely precision-agnostic for sub-100M decoder models: a 33% warmdown fraction is optimal across FP16, INT8, and INT6, with INT4 showing a regime boundary near 50M parameters.
When to use AdamW — and when to reconsider
AdamW remains the correct default for most LLM training workloads: it is well-understood, well-tuned in existing codebases, and its failure modes are now better characterized rather than newly discovered. Reconsider it when:
- Asynchronous pipeline parallelism is required for GPU utilization — Muon with Error Feedback correction is more robust under gradient delay.
- Heavy-tailed gradient noise is a concern and theoretical convergence guarantees matter — sign-based optimizers have sharper rates.
- Continual learning with non-stationary data is the setting — AdamO's isometry regularization addresses plasticity loss that AdamW does not.
- Hyperparameter transfer across scales is needed — audit the embedding layer learning rate first before switching parameterization schemes.
