EasyDeepLearn
Deep Learning · section 13 of 19

Recurrent networks

8 interview questions on recurrent networks, each answered in full. Free to read, no account needed.

Why did transformers replace RNNs for sequence modeling?

medium
  • RNNs process tokens sequentially, so they don't parallelize across time and struggle with long-range dependencies due to vanishing gradients (LSTMs help but only partly).
  • Transformers process all tokens in parallel with attention, capture arbitrary-range dependencies directly, and scale much better on GPUs/TPUs.
  • RNNs still appear in niche low-resource or online settings and in modern SSMs (Mamba) that revisit sequential modeling.
#transformers#rnnPermalink & quiz →

Explain the four gates of an LSTM.

medium
  • (1) Forget gate f  =  σ(Wf  ⋅  [ht−1,  xt])f\; = \;{\sigma}(W_{f}\; \cdot \;[h_{t - 1}, \;x_{t}]): what to remove from cell state.
  • (2) Input gate i = σ(...) and candidate ctilde  =  tanh()c_{\mathrm{tilde}}\; = \;\mathrm{tanh}(): what to add.
  • (3) Cell update: ct  =  f  ⋅  ct−1  +  i  ⋅  ctildec_{t}\; = \;f\; \cdot \;c_{t - 1}\; + \;i\; \cdot \;c_{\mathrm{tilde}}.
  • (4) Output gate o = σ(...) and ht  =  o  ⋅  tanh(ct)h_{t}\; = \;o\; \cdot \;\mathrm{tanh}(c_{t}).
  • The additive cell-state update lets gradients flow across many timesteps without vanishing.
  • Introduced by Hochreiter & Schmidhuber 1997.
#rnn#lstmPermalink & quiz →

GRU vs LSTM — how are they different in practice?

medium
  • GRU merges forget + input gates into an update gate and drops the separate cell state (uses only h).
  • Fewer parameters (~25% less), often trains faster, similar accuracy on most sequence tasks.
  • LSTM is a bit more expressive on very long sequences with complex temporal dynamics.
  • Rule of thumb: try GRU first for simplicity; use LSTM when GRU underfits.
#rnn#lstm#gruPermalink & quiz →

How does a bidirectional RNN work and when is it appropriate?

easy
  • Run two RNNs — one left-to-right, one right-to-left — and concatenate their hidden states at each timestep.
  • Provides access to both past and future context.
  • Appropriate for tagging tasks (POS, NER), speech recognition, classification.
  • Not usable for online generation (you need the full sequence).
  • Transformers with full attention subsume this by attending both directions in encoder-only setups (BERT).
#rnnPermalink & quiz →

What was Bahdanau attention and why was it a big deal?

hard
  • Bahdanau et al. (2014) introduced attention for seq2seq: at each decoder step, compute alignment scores between the current decoder hidden state and every encoder hidden state, softmax to get weights, take a weighted sum to form a 'context' vector, and use it in the decoder update.
  • Removed the RNN's fixed-length bottleneck between encoder and decoder — enabled machine translation on long sentences.
  • Precursor to full self-attention.
#attention#rnnPermalink & quiz →

How does beam search work and what is its main failure mode?

medium
  • At each decoding step, keep the k highest-probability sequences ('beams') so far, expand each by one token, keep the top k of the resulting k*|V∣  candidatesV \mid \;\mathrm{candidates}.
  • Approximate search — better than greedy, cheaper than exhaustive.
  • Failure mode: length bias (shorter sequences get higher joint probability) — fix with length normalization (divide  by  lengthα)(\mathrm{divide}\;\mathrm{by}\;\mathrm{length}{\alpha}).
  • Also produces 'safe, boring' outputs on open-ended generation — LLM decoders now prefer sampling with temperature / top-p.
#transformers#rnnPermalink & quiz →

Why is length normalization needed in beam search?

medium
  • Log-probability of a sequence is the sum of log P(tokent  ∣  history)P(\mathrm{token}_{t}\; \mid \;\mathrm{history}) — always negative, and longer sequences pile up more negatives.
  • Without normalization, beam search prefers shorter sequences by construction.
  • Length normalization divides by length^α (α ~ 0.6-1.0) or uses a length penalty.
  • Standard in NMT and summarization.
  • Modern LLM sampling uses temperature/top-p/top-k instead of beam search, so length norm is less critical there.
#transformers#rnnPermalink & quiz →

What is CTC loss and where is it used?

hard
  • Connectionist Temporal Classification (Graves 2006): sums the probability over all valid alignments between an input sequence (frames) and an output sequence (labels) that may be shorter.
  • Allows a 'blank' token and repeated labels.
  • Used in speech recognition (DeepSpeech), handwriting recognition — anywhere you have variable-length outputs but no per-frame labels.
  • Modern speech models often replace CTC with attention-based or CTC+Transducer hybrids.
#losses#rnnPermalink & quiz →

Practise Deep Learning

All 19 sectionsFlashcardsQuiz