EasyDeepLearn

What is Nesterov momentum and how is it different from classical momentum?

medium

Answer

  • Look-ahead trick: compute the gradient at the parameters shifted by the current velocity, not at the current parameters.
  • Update: vt  =  β  ⋅  vt−1v_{t}\; = \;{\beta}\; \cdot \;v_{t} - 1 + ∇L(θt  −  η  ⋅  β  ⋅  vt−1)L({\theta}_{t}\; - \;{\eta}\; \cdot \;{\beta}\; \cdot \;v_{t} - 1); θt+1  =  θt  −  η  ⋅  vt{\theta}_{t} + 1\; = \;{\theta}_{t}\; - \;{\eta}\; \cdot \;v_{t}.
  • Effectively 'peek' where momentum would take you and correct.
  • Theoretical acceleration in convex settings (O(1/T2)  instead  of  O(1/T))(O(1 / T^{2})\;\mathrm{instead}\;\mathrm{of}\;O(1 / T)), modest improvement in deep learning practice.
Check yourself — multiple choice
  • Same as classical momentum
  • Look-ahead gradient at θ - η·β·v — theoretical acceleration in convex problems
  • Uses second derivatives
  • Removes momentum

Nesterov = gradient evaluated at the look-ahead point → faster convex convergence.

#optimizers

Practise Deep Learning

214 interview questions in this topic.

Read the lessonFlashcardsTake the quiz

Related questions