Why use a learning-rate schedule (warmup + cosine decay)?
mediumAnswer
- Large models with LayerNorm behave badly at high LR early on — warmup ramps LR from 0 to peak over a few thousand steps to avoid divergence.
- Then cosine (or linear) decay reduces LR smoothly toward zero, which improves final generalization.
- Standard recipe for training transformers and LLMs.
Check yourself — multiple choice
- Warmup increases the batch size gradually
- Warmup ramps LR up; cosine schedule decays LR smoothly to improve convergence
- Constant LR is optimal for transformers
- Cosine decay increases LR at the end
Warmup + cosine decay is the go-to recipe for large-model training.
#schedules#learning-rate#training
Practise Deep Learning
214 interview questions in this topic.