When do you use LayerNorm instead of BatchNorm?
mediumAnswer
- Use LayerNorm when batch size is small or varies (RNNs, transformers, sequence data).
- LayerNorm computes statistics across features within a single example, so it doesn't depend on batch statistics.
- That's why it's standard in transformers and language models where sequence lengths vary and batches can be tiny.
Check yourself — multiple choice
- LayerNorm depends on batch size
- LayerNorm normalizes across features within one example
- BatchNorm is standard in transformers
- LayerNorm has no learnable parameters
LayerNorm is per-example, making it robust to small/variable batches.
#normalization
Practise Deep Learning
214 interview questions in this topic.