What was Bahdanau attention and why was it a big deal?
hardAnswer
- Bahdanau et al. (2014) introduced attention for seq2seq: at each decoder step, compute alignment scores between the current decoder hidden state and every encoder hidden state, softmax to get weights, take a weighted sum to form a 'context' vector, and use it in the decoder update.
- Removed the RNN's fixed-length bottleneck between encoder and decoder — enabled machine translation on long sentences.
- Precursor to full self-attention.
Check yourself — multiple choice
- Same as scaled dot-product attention
- Learned alignment between decoder state and encoder states — removed fixed bottleneck in seq2seq
- Attention was invented in 2020
- Only for CNNs
Bahdanau attention: dynamic context from all encoder states → removed bottleneck.
#attention#rnn
Practise Deep Learning
214 interview questions in this topic.
Related questions
- How does self-attention work in a transformer?
- Why did transformers replace RNNs for sequence modeling?
- When is orthogonal weight initialization useful?
- What causes exploding gradients and how do you handle them?
- Why is BatchNorm awkward in RNNs?
- Write the vanilla RNN update equation and explain why it struggles with long sequences.