EasyDeepLearn

Write the vanilla RNN update equation and explain why it struggles with long sequences.

easy

Answer

  • ht  =  tanh(Whh  ⋅  ht−1  +  Wxh  ⋅  xt  +  b)h_{t}\; = \;\mathrm{tanh}(W_{\mathrm{hh}}\; \cdot \;h_{t - 1}\; + \;W_{\mathrm{xh}}\; \cdot \;x_{t}\; + \;b).
  • Repeatedly multiplying by WhhW_{\mathrm{hh}} during BPTT causes gradients to vanish (spectral norm < 1) or explode (> 1) over long sequences.
  • LSTMs / GRUs introduce gated additive updates to preserve gradients across time; transformers skip recurrence entirely and use attention.
Check yourself — multiple choice
  • ht  =  Wh_{t}\; = \;W · ht−1h_{t - 1} only
  • ht  =  tanh(Whh  ⋅  ht−1  +  Wxh  ⋅  xt  +  b)h_{t}\; = \;\mathrm{tanh}(W_{\mathrm{hh}}\; \cdot \;h_{t - 1}\; + \;W_{\mathrm{xh}}\; \cdot \;x_{t}\; + \;b); repeated WhhW_{\mathrm{hh}} multiplications cause vanish/explode
  • RNNs have no gradients
  • Vanilla RNNs use attention

Vanilla RNN: repeated WhhW_{\mathrm{hh}} multiplication → gradient problems on long sequences.

#rnn#fundamentals

Practise Deep Learning

214 interview questions in this topic.

Read the lessonFlashcardsTake the quiz

Related questions