EasyDeepLearn

How does RMSProp fix AdaGrad?

medium

Answer

  • Replaces the running sum of squared gradients with an exponential moving average: vt  =  β  ⋅  vt−1  +  (1−β)  ⋅  gt2v_{t}\; = \;{\beta}\; \cdot \;v_{t} - 1\; + \;(1 - {\beta})\; \cdot \;g_{t}^{2}.
  • Update: θ ← θ  −  η  ⋅  g  /  (sqrt(vt)  +  ε){\theta}\; - \;{\eta}\; \cdot \;g\; / \;(\mathrm{sqrt}(v_{t})\; + \;{\varepsilon}).
  • Typical β = 0.9-0.99.
  • LR no longer decays to zero — the running variance forgets old gradients.
  • Good default for RNNs before Adam took over.
Check yourself — multiple choice
  • RMSProp uses sum instead of EMA
  • EMA of squared gradients replaces AdaGrad's sum — LR no longer shrinks to zero
  • RMSProp adds momentum only
  • RMSProp has no hyperparameters

RMSProp = AdaGrad but with EMA of g2g^{2} — prevents LR from vanishing.

#optimizers

Practise Deep Learning

214 interview questions in this topic.

Read the lessonFlashcardsTake the quiz

Related questions