Why did transformers switch from post-norm to pre-norm at scale?
hardAnswer
- Original transformer used post-norm: LayerNorm(x + sublayer(x)).
- At depth > 12, the sublayer outputs interfere with LN's running scale, causing training instabilities that require very long warmup or careful initialization.
- Pre-norm — x + sublayer(LayerNorm(x)) — puts the LayerNorm inside the residual branch, leaving a clean identity path for gradients.
- Trains stably for 100+ layers with short warmup.
- All modern LLMs use pre-norm.
Check yourself — multiple choice
- Post-norm scales best
- Pre-norm keeps residual path identity → stable at 100+ layers, short warmup, standard in LLMs
- No difference
- Only pre-norm existed
Pre-norm: identity residual → stable at extreme depth.
#transformers#normalization
Practise Deep Learning
214 interview questions in this topic.