Why is BatchNorm awkward in RNNs?
mediumAnswer
- RNNs share weights across time; you'd need separate BN statistics for every timestep to be principled, but sequences have variable length.
- Also, small effective batch (batch × time reshape doesn't help because time is not exchangeable), and running stats depend on sequence length.
- LayerNorm has none of these issues — per-example, per-timestep, unconditional on batch.
- That's why LSTMs and transformers use LN, not BN.
Check yourself — multiple choice
- BN works perfectly in RNNs
- Variable sequence length and per-timestep stats make BN awkward — LayerNorm is standard for RNNs / transformers
- RNNs never use normalization
- BN is required for LSTMs
BN needs per-timestep stats + fixed sequence length — LayerNorm avoids both.
#normalization#rnn
Practise Deep Learning
214 interview questions in this topic.
Related questions
- What does Batch Normalization do?
- When do you use LayerNorm instead of BatchNorm?
- Why did transformers replace RNNs for sequence modeling?
- When is orthogonal weight initialization useful?
- Do you need bias terms in every layer of a modern neural network?
- What causes exploding gradients and how do you handle them?