EasyDeepLearn
Deep Learning · section 5 of 19

Optimizers

17 interview questions on optimizers, each answered in full. Free to read, no account needed.

SGD vs Adam vs AdamW — how do you choose?

medium
  • SGD with momentum is simple and often generalizes better on vision when tuned carefully with LR schedules.
  • Adam adapts per-parameter learning rates using first and second moment estimates — great default, converges fast, robust to hyperparameters.
  • AdamW decouples weight decay from the gradient update — the standard for training large transformers and modern LLMs. Rule of thumb: AdamW for transformers, SGD+momentum for classic CNNs.
#optimization#optimizersPermalink & quiz →

Write the vanilla SGD update rule and explain each term.

easy
  • θt+1  =  θt  −  η  ⋅  gt{\theta}_{t} + 1\; = \;{\theta}_{t}\; - \;{\eta}\; \cdot \;g_{t}, where θ is the parameter vector, η the learning rate, and gtg_{t} = ∇_θ L(θt  batcht)L({\theta}_{t}\;\mathrm{batch}_{t}) is the stochastic gradient on the current mini-batch.
  • Each step is a noisy estimate of the true gradient — the noise itself acts as regularizer and helps escape saddle points.
  • Convergence rate for convex problems: O(1/√T).
#optimizers#optimizationPermalink & quiz →

How does classical momentum modify SGD?

easy
  • Maintain a velocity: vt  =  β  ⋅  vt−1  +  gtv_{t}\; = \;{\beta}\; \cdot \;v_{t} - 1\; + \;g_{t}; then θt+1  =  θt  −  η  ⋅  vt{\theta}_{t} + 1\; = \;{\theta}_{t}\; - \;{\eta}\; \cdot \;v_{t}.
  • Typical β = 0.9.
  • Momentum accumulates gradients over recent steps, damping oscillations along high-curvature directions and accelerating along consistent ones.
  • Roughly equivalent to averaging gradients over the last ~1/(1-β) steps — often improves convergence 2-10x on ill-conditioned problems.
#optimizersPermalink & quiz →

What is Nesterov momentum and how is it different from classical momentum?

medium
  • Look-ahead trick: compute the gradient at the parameters shifted by the current velocity, not at the current parameters.
  • Update: vt  =  β  ⋅  vt−1v_{t}\; = \;{\beta}\; \cdot \;v_{t} - 1 + ∇L(θt  −  η  ⋅  β  ⋅  vt−1)L({\theta}_{t}\; - \;{\eta}\; \cdot \;{\beta}\; \cdot \;v_{t} - 1); θt+1  =  θt  −  η  ⋅  vt{\theta}_{t} + 1\; = \;{\theta}_{t}\; - \;{\eta}\; \cdot \;v_{t}.
  • Effectively 'peek' where momentum would take you and correct.
  • Theoretical acceleration in convex settings (O(1/T2)  instead  of  O(1/T))(O(1 / T^{2})\;\mathrm{instead}\;\mathrm{of}\;O(1 / T)), modest improvement in deep learning practice.
#optimizersPermalink & quiz →

What is AdaGrad and its main drawback?

medium
  • Per-parameter learning rate: θ ← θ - η * g / (sqrt(sum of squared past gradients) + ε).
  • Parameters with large historical gradients get smaller effective LRs — great for sparse features (NLP with one-hot inputs).
  • Drawback: accumulator grows monotonically, so effective LR → 0.
  • Model stops learning after enough steps.
  • RMSProp and Adam fix this with exponential moving averages.
#optimizersPermalink & quiz →

How does RMSProp fix AdaGrad?

medium
  • Replaces the running sum of squared gradients with an exponential moving average: vt  =  β  ⋅  vt−1  +  (1−β)  ⋅  gt2v_{t}\; = \;{\beta}\; \cdot \;v_{t} - 1\; + \;(1 - {\beta})\; \cdot \;g_{t}^{2}.
  • Update: θ ← θ  −  η  ⋅  g  /  (sqrt(vt)  +  ε){\theta}\; - \;{\eta}\; \cdot \;g\; / \;(\mathrm{sqrt}(v_{t})\; + \;{\varepsilon}).
  • Typical β = 0.9-0.99.
  • LR no longer decays to zero — the running variance forgets old gradients.
  • Good default for RNNs before Adam took over.
#optimizersPermalink & quiz →

Write Adam's full update rule.

medium
  • mt  =  β1  ⋅  mt−1  +  (1−β1)  ⋅  gtm_{t}\; = \;{\beta}_{1}\; \cdot \;m_{t} - 1\; + \;(1 - {\beta}_{1})\; \cdot \;g_{t} (first moment); vt  =  β2  ⋅  vt−1  +  (1−β2)  ⋅  gt2v_{t}\; = \;{\beta}_{2}\; \cdot \;v_{t} - 1\; + \;(1 - {\beta}_{2})\; \cdot \;g_{t}^{2} (second moment).
  • Bias-corrected: mhat  =  mt  /  (1  −  β1t)m_{\mathrm{hat}}\; = \;m_{t}\; / \;(1\; - \;{\beta}_{1}^{t}), vhat  =  vt  /  (1  −  β2t)v_{\mathrm{hat}}\; = \;v_{t}\; / \;(1\; - \;{\beta}_{2}^{t}).
  • Update: θ ← θ  −  η  ⋅  mhat  /  (sqrt(vhat)  +  ε){\theta}\; - \;{\eta}\; \cdot \;m_{\mathrm{hat}}\; / \;(\mathrm{sqrt}(v_{\mathrm{hat}})\; + \;{\varepsilon}).
  • Defaults: β1=0.9{\beta}_{1} = 0.9, β2=0.999{\beta}_{2} = 0.999, ε=1e-8.
  • Combines momentum and per-param LR — robust default optimizer.
#optimizersPermalink & quiz →

Why does AdamW work better than Adam + L2 weight decay for transformers?

hard
  • In Adam, adding λ*θ inside the gradient means weight decay gets rescaled by the per-parameter learning rate 1/sqrt(vhat)1 / \mathrm{sqrt}(v_{\mathrm{hat}}).
  • Parameters with small gradients get almost no decay; those with large ones get too much.
  • AdamW decouples: apply weight decay directly to parameters (θ ← θ  −  η⋅(mhat/sqrt(vhat)  +  λ⋅θ){\theta}\; - \;{\eta} \cdot (m_{\mathrm{hat}} / \mathrm{sqrt}(v_{\mathrm{hat}})\; + \;{\lambda} \cdot {\theta})) — every parameter shrinks by η*λ uniformly.
  • Critical for transformer training.
#optimizers#regularizationPermalink & quiz →

What is the Lion optimizer and why is it interesting?

hard
  • Lion (Chen et al., 2023, from Google) uses only the sign of the momentum: θ ← θ  −  η  ⋅  sign⁡(β1⋅m  +  (1−β1)⋅g){\theta}\; - \;{\eta}\; \cdot \;\operatorname{sign}({\beta}_{1} \cdot m\; + \;(1 - {\beta}_{1}) \cdot g), then m ← β2⋅m  +  (1−β2)⋅g{\beta}_{2} \cdot m\; + \;(1 - {\beta}_{2}) \cdot g.
  • Half the optimizer state of Adam (no v), often trains 1.5-2x faster in wall-clock at same accuracy on transformers and vision.
  • Requires smaller LR (~10x smaller than Adam) and more aggressive weight decay.
#optimizersPermalink & quiz →

Why does Adafactor use less memory than Adam?

hard
  • Adafactor factorizes the second-moment matrix v: instead of storing a full vtv_{t} per parameter (O(N) memory), it stores row-sum and column-sum statistics for each 2D weight matrix (O(sqrt(N))).
  • Massive memory savings — enabled T5 training.
  • Trade-off: some hyperparameters harder to tune, occasionally less stable than full Adam.
#optimizers#distributedPermalink & quiz →

What is Shampoo and when does second-order optimization pay off?

hard
  • Shampoo (Gupta et al., 2018 / Anil et al., 2020) is a second-order-ish optimizer that maintains a per-parameter matrix preconditioner factorized along tensor axes — computes the inverse Kronecker-factored covariance of gradients.
  • Uses more compute per step than Adam but converges in far fewer steps.
  • Practical for very large-scale training where compute is limited by memory bandwidth rather than FLOPs (T5, PaLM experiments).
#optimizers#optimizationPermalink & quiz →

How does weight decay change the objective and the update?

medium
  • Adds an L2 penalty λ/2 * ||θ||² to the loss.
  • In SGD: θ ← θ - η * (∇L + λ*θ) = (1 - η*λ) * θ - η * ∇L — shrinks weights each step.
  • Effects: prevents overfitting (Occam-style bias toward small weights), reduces effective model capacity, and improves generalization.
  • In AdamW, applied directly to θ (not through g), which is why AdamW works better than 'Adam + L2 in loss'.
#regularization#optimizersPermalink & quiz →

How much extra memory does Adam use versus SGD?

medium
  • SGD with momentum: 1x parameter count in optimizer state (momentum).
  • Adam: 2x parameter count (first and second moment estimates).
  • AdamW: same as Adam.
  • For a 7B model in fp32, this is 28 GB just for m and v — a big reason why large-model training uses fp16 / bf16 optimizer states, sharded optimizers (ZeRO), or memory-efficient optimizers (Adafactor, 8-bit Adam).
#optimizers#distributedPermalink & quiz →

What does SAM (Sharpness-Aware Minimization) do?

hard
  • SAM (Foret et al., 2020) minimizes a surrogate that penalizes sharp minima: for each step, first perturb θ in the direction that maximizes the loss within a small ball (θ  +  ρ  ⋅  ∇L  /  ∣∣∇L∣∣)({\theta}\; + \;{\rho}\; \cdot \; \nabla L\; / \; \mid \mid \nabla L \mid \mid ), then compute the gradient at that perturbed point and update the original θ.
  • Roughly 2x compute per step, but consistently improves generalization on vision and NLP benchmarks.
  • Extended in adaptive SAM variants.
#optimizers#regularizationPermalink & quiz →

Why is weight decay via L2-in-loss different from decoupled weight decay in Adam?

hard
  • L2 in loss: gradient becomes g + λ*θ, then Adam divides by sqrt(vhat)\mathrm{sqrt}(v_{\mathrm{hat}}).
  • Parameters with large gradients get smaller effective weight decay — inverted relative to what you want.
  • Decoupled (AdamW): weight decay applied directly to θ as θ ← θ - η*λ*θ, independent of the adaptive scale.
  • Every parameter shrinks uniformly.
  • Empirically much better generalization on transformers.
#regularization#optimizersPermalink & quiz →

When do you prefer SGD with momentum over Adam?

medium
  • For convolutional vision models trained from scratch to their best possible accuracy, tuned SGD with momentum still tends to generalize slightly better than Adam, which is why many image classification recipes use it.
  • Adam wins where gradient scales differ wildly across parameters: transformers, embeddings with sparse updates, and anything with attention.
  • Adam is also far more forgiving of a poorly chosen learning rate, which makes it the right default when you cannot afford a sweep.
  • Memory matters too, since Adam stores two extra states per parameter, which is significant at scale.
  • The honest summary is that Adam, or AdamW, is the default for language models and SGD remains competitive for vision, and the difference shrinks once both are properly tuned.
#optimizers#optimizationPermalink & quiz →

Why does AdamW handle weight decay differently from Adam, and why does it matter?

hard
  • In Adam, weight decay is added into the gradient, so it passes through the adaptive scaling.
  • Parameters with a large accumulated second moment get their decay divided down, which means the effective regularization differs per parameter and does not match what you asked for.
  • AdamW decouples it: the decay is applied directly to the weights, outside the adaptive step, so every parameter shrinks at the same relative rate.
  • In practice this makes the weight decay hyperparameter behave predictably and transfer between learning rates, and it measurably improves generalization on transformers.
  • It is also why decay is usually excluded from bias and normalization parameters, where shrinking towards zero has no regularizing meaning.
#regularization#optimizersPermalink & quiz →

Practise Deep Learning

All 19 sectionsFlashcardsQuiz