EasyDeepLearn

Explain the four gates of an LSTM.

medium

Answer

  • (1) Forget gate f  =  σ(Wf  ⋅  [ht−1,  xt])f\; = \;{\sigma}(W_{f}\; \cdot \;[h_{t - 1}, \;x_{t}]): what to remove from cell state.
  • (2) Input gate i = σ(...) and candidate ctilde  =  tanh()c_{\mathrm{tilde}}\; = \;\mathrm{tanh}(): what to add.
  • (3) Cell update: ct  =  f  ⋅  ct−1  +  i  ⋅  ctildec_{t}\; = \;f\; \cdot \;c_{t - 1}\; + \;i\; \cdot \;c_{\mathrm{tilde}}.
  • (4) Output gate o = σ(...) and ht  =  o  ⋅  tanh(ct)h_{t}\; = \;o\; \cdot \;\mathrm{tanh}(c_{t}).
  • The additive cell-state update lets gradients flow across many timesteps without vanishing.
  • Introduced by Hochreiter & Schmidhuber 1997.
Check yourself — multiple choice
  • LSTM has one gate
  • Forget/Input/Candidate/Output; additive cell-state update ct  =  fc_{t}\; = \;f·ct−1  +  ic_{t - 1}\; + \;i·ctildec_{\mathrm{tilde}} preserves gradients
  • LSTM has no cell state
  • Only one gate matters

LSTM: 4 gates + additive cell update → gradient highway across time.

#rnn#lstm

Practise Deep Learning

214 interview questions in this topic.

Read the lessonFlashcardsTake the quiz

Related questions