Why does AdamW work better than Adam + L2 weight decay for transformers?
hardAnswer
- In Adam, adding λ*θ inside the gradient means weight decay gets rescaled by the per-parameter learning rate .
- Parameters with small gradients get almost no decay; those with large ones get too much.
- AdamW decouples: apply weight decay directly to parameters (θ ← ) — every parameter shrinks by η*λ uniformly.
- Critical for transformer training.
Check yourself — multiple choice
- AdamW is identical to Adam
- AdamW decouples weight decay from the gradient — uniform shrinkage, works well for transformers
- AdamW removes momentum
- AdamW uses no ε
AdamW: apply weight decay directly to θ, not through the gradient — uniform, transformer-friendly.
#optimizers#regularization
Practise Deep Learning
214 interview questions in this topic.
Related questions
- How does weight decay change the objective and the update?
- What does SAM (Sharpness-Aware Minimization) do?
- Why is weight decay via L2-in-loss different from decoupled weight decay in Adam?
- Why does AdamW handle weight decay differently from Adam, and why does it matter?
- How does dropout work and when is it applied?
- SGD vs Adam vs AdamW — how do you choose?