How does RMSNorm differ from LayerNorm?
mediumAnswer
- RMSNorm(x) = x / RMS(x) * γ, where .
- Skips the mean-subtraction and β (bias) of LayerNorm.
- About 10-30% faster (fewer ops), similar accuracy on transformers.
- Adopted by LLaMA, T5, and most 2023+ open LLMs.
Check yourself — multiple choice
- RMSNorm subtracts the mean like LayerNorm
- Divide by RMS only (no mean subtraction, no bias) — faster than LN, standard in LLaMA/T5
- RMSNorm requires batch stats
- RMSNorm is a loss function
RMSNorm skips mean and bias — faster, adopted by LLaMA/T5.
#normalization#transformers
Practise Deep Learning
214 interview questions in this topic.
Related questions
- Pre-norm vs post-norm transformers — which is standard and why?
- Why did transformers switch from post-norm to pre-norm at scale?
- What does Batch Normalization do?
- When do you use LayerNorm instead of BatchNorm?
- How does self-attention work in a transformer?
- Why does a transformer need positional encoding?