EasyDeepLearn
Deep Learning · section 2 of 19

Backpropagation & autograd

2 interview questions on backpropagation & autograd, each answered in full. Free to read, no account needed.

Why does the backward pass require more memory than the forward pass?

medium
  • Reverse-mode autodiff needs the intermediate activations from the forward pass to compute gradients (the chain rule uses them).
  • So all activations are stored until backward finishes.
  • Peak memory ≈ activations + gradients + optimizer state.
  • Fixes for memory pressure: gradient checkpointing (recompute activations during backward instead of storing), mixed precision, model / ZeRO parallelism.
#backprop#gradient-checkpointing#trainingPermalink & quiz →

What is backpropagation through time (BPTT)?

easy
  • Unroll the RNN over the sequence length T, treat it as a very deep feed-forward network with shared weights, and apply standard backprop.
  • Memory scales linearly with T.
  • Truncated BPTT limits the unroll length to k steps to bound memory (each backward pass covers only the most recent k timesteps), at the cost of not learning dependencies longer than k.
#rnn#backpropPermalink & quiz →

Practise Deep Learning

All 19 sectionsFlashcardsQuiz