What does SAM (Sharpness-Aware Minimization) do?
hardAnswer
- SAM (Foret et al., 2020) minimizes a surrogate that penalizes sharp minima: for each step, first perturb θ in the direction that maximizes the loss within a small ball , then compute the gradient at that perturbed point and update the original θ.
- Roughly 2x compute per step, but consistently improves generalization on vision and NLP benchmarks.
- Extended in adaptive SAM variants.
Check yourself — multiple choice
- SAM sharpens the minimum
- Compute gradient at a perturbed θ that maximizes local loss — penalizes sharpness → flatter minima
- SAM removes momentum
- SAM has zero overhead
SAM: gradient at a worst-case perturbation → flatter minima, ~2× cost, better generalization.
#optimizers#regularization
Practise Deep Learning
214 interview questions in this topic.
Related questions
- Why does AdamW work better than Adam + L2 weight decay for transformers?
- How does weight decay change the objective and the update?
- Why is weight decay via L2-in-loss different from decoupled weight decay in Adam?
- Why does AdamW handle weight decay differently from Adam, and why does it matter?
- How does dropout work and when is it applied?
- SGD vs Adam vs AdamW — how do you choose?