EasyDeepLearn

What is the Lion optimizer and why is it interesting?

hard

Answer

  • Lion (Chen et al., 2023, from Google) uses only the sign of the momentum: θ ← θ  −  η  ⋅  sign⁡(β1⋅m  +  (1−β1)⋅g){\theta}\; - \;{\eta}\; \cdot \;\operatorname{sign}({\beta}_{1} \cdot m\; + \;(1 - {\beta}_{1}) \cdot g), then m ← β2⋅m  +  (1−β2)⋅g{\beta}_{2} \cdot m\; + \;(1 - {\beta}_{2}) \cdot g.
  • Half the optimizer state of Adam (no v), often trains 1.5-2x faster in wall-clock at same accuracy on transformers and vision.
  • Requires smaller LR (~10x smaller than Adam) and more aggressive weight decay.
Check yourself — multiple choice
  • Lion is identical to Adam
  • Uses sign of momentum — halves optimizer memory, often faster than Adam at same accuracy
  • Lion needs a bigger LR than Adam
  • Lion tracks only gradients, not momentum

Lion uses sign⁡(momentum)\operatorname{sign}(\mathrm{momentum}): less state, faster in practice; discovered via symbolic search.

#optimizers

Practise Deep Learning

214 interview questions in this topic.

Read the lessonFlashcardsTake the quiz

Related questions