Why did transformers replace RNNs for sequence modeling?
mediumAnswer
- RNNs process tokens sequentially, so they don't parallelize across time and struggle with long-range dependencies due to vanishing gradients (LSTMs help but only partly).
- Transformers process all tokens in parallel with attention, capture arbitrary-range dependencies directly, and scale much better on GPUs/TPUs.
- RNNs still appear in niche low-resource or online settings and in modern SSMs (Mamba) that revisit sequential modeling.
Check yourself — multiple choice
- Transformers cannot parallelize training
- Transformers parallelize sequence processing and handle long-range context well
- RNNs handle long context better than transformers
- Transformers use only recurrence, no attention
Parallel training and direct long-range attention are the key advantages.
#transformers#rnn
Practise Deep Learning
214 interview questions in this topic.