When do you prefer SGD with momentum over Adam?
mediumAnswer
- For convolutional vision models trained from scratch to their best possible accuracy, tuned SGD with momentum still tends to generalize slightly better than Adam, which is why many image classification recipes use it.
- Adam wins where gradient scales differ wildly across parameters: transformers, embeddings with sparse updates, and anything with attention.
- Adam is also far more forgiving of a poorly chosen learning rate, which makes it the right default when you cannot afford a sweep.
- Memory matters too, since Adam stores two extra states per parameter, which is significant at scale.
- The honest summary is that Adam, or AdamW, is the default for language models and SGD remains competitive for vision, and the difference shrinks once both are properly tuned.
Check yourself — multiple choice
- SGD is always better
- Tuned SGD with momentum can generalize slightly better for vision models from scratch; Adam wins on transformers and sparse gradients and tolerates untuned learning rates
- Adam never converges
- They are interchangeable
The choice tracks gradient scale heterogeneity, tuning budget and memory, not a universal ranking.
#optimizers#optimization
Practise Deep Learning
214 interview questions in this topic.
Related questions
- SGD vs Adam vs AdamW — how do you choose?
- Write the vanilla SGD update rule and explain each term.
- What is Shampoo and when does second-order optimization pay off?
- What is the 'implicit bias' of SGD?
- How does classical momentum modify SGD?
- What is Nesterov momentum and how is it different from classical momentum?