Why does AdamW handle weight decay differently from Adam, and why does it matter?

hard

Answer

$In Adam, weight decay is added into the gradient, so it passes through the adaptive scaling.$
$Parameters with a large accumulated second moment get their decay divided down, which means the effective regularization differs per parameter and does not match what you asked for.$
$AdamW decouples it: the decay is applied directly to the weights, outside the adaptive step, so every parameter shrinks at the same relative rate.$
$In practice this makes the weight decay hyperparameter behave predictably and transfer between learning rates, and it measurably improves generalization on transformers.$
$It is also why decay is usually excluded from bias and normalization parameters, where shrinking towards zero has no regularizing meaning.$

Check yourself — multiple choice

There is no real difference
Adam pushes decay through the adaptive scaling so regularization varies per parameter; AdamW applies it directly to the weights, making the hyperparameter predictable
AdamW removes weight decay
Only the learning rate changes

Decoupling decay from the adaptive denominator makes regularization uniform and tunable.

#regularization#optimizers

214 interview questions in this topic.