How does classical momentum modify SGD?

easy

Answer

$v_{t}\; = \;{\beta}\; \cdot \;v_{t} - 1\; + \;g_{t}$
$Typical β = 0.9.$
$Momentum accumulates gradients over recent steps, damping oscillations along high-curvature directions and accelerating along consistent ones.$
$Roughly equivalent to averaging gradients over the last ~1/(1-β) steps — often improves convergence 2-10x on ill-conditioned problems.$

Check yourself — multiple choice

Momentum divides the gradient by β
v ← β·v + g, θ ← θ - η·v — smooths and accelerates convergence on ill-conditioned loss
Momentum doubles learning rate
Momentum only works on CPUs

Momentum accumulates gradient exponentially (β≈0.9) — smoother, faster convergence.

#optimizers

214 interview questions in this topic.