Pre-norm vs post-norm transformers — which is standard and why?
hardAnswer
- Original Transformer (Vaswani 2017) used post-norm: x + Attn(x), then LayerNorm.
- Trains poorly at scale (deep post-norm nets need long warmup).
- Pre-norm applies LayerNorm before the sublayer: x + Attn(LN(x)) — gradients flow through the identity skip unchanged, trains stably with very deep nets and short warmup.
- All modern LLMs use pre-norm.
Check yourself — multiple choice
- Post-norm scales better
- Pre-norm (LayerNorm before sublayer, then residual) — stable at depth, short warmup, standard in modern LLMs
- Both are identical
- Pre-norm removes the residual
Pre-norm: identity residual → stable deep-transformer training → adopted in all modern LLMs.
#normalization#transformers
Practise Deep Learning
214 interview questions in this topic.