Why is weight decay via L2-in-loss different from decoupled weight decay in Adam?
hardAnswer
- L2 in loss: gradient becomes g + λ*θ, then Adam divides by .
- Parameters with large gradients get smaller effective weight decay — inverted relative to what you want.
- Decoupled (AdamW): weight decay applied directly to θ as θ ← θ - η*λ*θ, independent of the adaptive scale.
- Every parameter shrinks uniformly.
- Empirically much better generalization on transformers.
Check yourself — multiple choice
- They are identical in Adam
- L2-in-loss gets rescaled by 1/√v → inconsistent decay; AdamW decouples for uniform shrinkage
- L2 in loss is always better
- Weight decay is disabled in Adam
AdamW's decoupled weight decay avoids the 1/√v rescaling issue of L2-in-loss.
#regularization#optimizers
Practise Deep Learning
214 interview questions in this topic.
Related questions
- Why does AdamW work better than Adam + L2 weight decay for transformers?
- How does weight decay change the objective and the update?
- What does SAM (Sharpness-Aware Minimization) do?
- Why does AdamW handle weight decay differently from Adam, and why does it matter?
- How does dropout work and when is it applied?
- SGD vs Adam vs AdamW — how do you choose?