Why do transformers need learning-rate warmup?
mediumAnswer
- At the start of training, weights are random and LayerNorm statistics are meaningless.
- A high LR causes large gradients that destabilize LayerNorm's running scale and can permanently damage the network (attention heads collapse).
- Warmup ramps LR linearly from 0 to peak over a few thousand steps, giving the network time to stabilize activations before high-LR updates.
Check yourself — multiple choice
- Warmup increases learning rate to infinity
- Stabilizes early training when LayerNorm stats and weights are wild — avoids divergence and dead heads
- Warmup is only for CNNs
- Warmup replaces optimizer
Warmup avoids destabilizing LayerNorm and attention at the very start.
#schedules#learning-rate#transformers
Practise Deep Learning
214 interview questions in this topic.