When do you use truncated BPTT and what's the trade-off?
mediumAnswer
- For very long sequences (language modeling on 1000+ tokens, time-series, audio).
- Unroll for k steps, backward, then carry hidden state forward but detach it from the graph.
- Bounded memory, faster training.
- Trade-off: can't learn dependencies longer than k without further tricks (e.g., overlap windows, memorized state).
- Modern transformers avoid this via full parallel attention over the whole context.
Check yourself — multiple choice
- Truncated BPTT keeps full graph
- Unroll k steps, detach hidden state — bounded memory, can't learn dependencies > k
- Same as full BPTT
- Removes gradients entirely
Truncated BPTT: bounded unroll → bounded memory, but limits long-range learning.
#rnn#training
Practise Deep Learning
214 interview questions in this topic.