Why does transformer training usually need a learning-rate warmup?

hard

Answer

$At initialization, Adam's second-moment estimate is based on almost no history, so its normalization is unreliable and the first updates can be enormous relative to the weights.$
$Large early steps push a transformer into a region from which it never fully recovers, often visible as an attention collapse or an immediate loss spike.$
$Warmup ramps the learning rate from near zero over a few hundred to a few thousand steps, giving the optimizer state time to become meaningful before large steps are taken.$
$Post-layer-norm architectures need it most, because gradients through the residual path are badly scaled at initialization; pre-layer-norm and careful residual scaling reduce the dependence but rarely remove it entirely.$

Check yourself — multiple choice

It saves compute
Early Adam second-moment estimates are unreliable, so the first steps can be huge; warmup lets the optimizer state stabilize, which matters most for post-layer-norm transformers
It prevents overfitting
Warmup is purely cosmetic

Warmup protects against oversized early updates while the adaptive state is still uninformative.

#schedules#learning-rate#transformers

214 interview questions in this topic.