SGD vs Adam vs AdamW — how do you choose?
mediumAnswer
- SGD with momentum is simple and often generalizes better on vision when tuned carefully with LR schedules.
- Adam adapts per-parameter learning rates using first and second moment estimates — great default, converges fast, robust to hyperparameters.
- AdamW decouples weight decay from the gradient update — the standard for training large transformers and modern LLMs. Rule of thumb: AdamW for transformers, SGD+momentum for classic CNNs.
Check yourself — multiple choice
- Adam and AdamW handle weight decay identically
- AdamW decouples weight decay from the gradient update
- SGD adapts per-parameter learning rates
- Adam is deterministic
AdamW's decoupled weight decay is why it's used for transformers.
#optimization#optimizers
Practise Deep Learning
214 interview questions in this topic.
Related questions
- Write the vanilla SGD update rule and explain each term.
- What is Shampoo and when does second-order optimization pay off?
- When do you prefer SGD with momentum over Adam?
- What is the 'implicit bias' of SGD?
- How does classical momentum modify SGD?
- What is Nesterov momentum and how is it different from classical momentum?