How does weight decay change the objective and the update?
mediumAnswer
- Adds an L2 penalty λ/2 * ||θ||² to the loss.
- In SGD: θ ← θ - η * (∇L + λ*θ) = (1 - η*λ) * θ - η * ∇L — shrinks weights each step.
- Effects: prevents overfitting (Occam-style bias toward small weights), reduces effective model capacity, and improves generalization.
- In AdamW, applied directly to θ (not through g), which is why AdamW works better than 'Adam + L2 in loss'.
Check yourself — multiple choice
- Weight decay adds noise to gradients
- L2 penalty on θ → SGD becomes θ ← (1-ηλ)·θ - η·∇L → smaller weights, better generalization
- Weight decay only affects biases
- Removes the loss entirely
Weight decay = L2 penalty → weights shrink every step → better generalization.
#regularization#optimizers
Practise Deep Learning
214 interview questions in this topic.
Related questions
- Why does AdamW work better than Adam + L2 weight decay for transformers?
- What does SAM (Sharpness-Aware Minimization) do?
- Why is weight decay via L2-in-loss different from decoupled weight decay in Adam?
- Why does AdamW handle weight decay differently from Adam, and why does it matter?
- How does dropout work and when is it applied?
- SGD vs Adam vs AdamW — how do you choose?