When is step decay still appropriate?
easyAnswer
- Multiply LR by a factor (e.g., 0.1) at fixed milestones (e.g., 60/90 epochs on ImageNet).
- Simple, deterministic, well-understood — classic recipe for training ResNet-style CNNs with SGD+momentum.
- Downsides: sudden LR jumps interact poorly with running momentum, need careful tuning of milestones.
- Cosine is smoother but step decay is fine for well-studied recipes.
Check yourself — multiple choice
- Step decay is smooth
- Multiply LR by a factor at fixed milestones — classic ResNet recipe; simple but has sudden jumps
- Step decay uses cosine internally
- Only for transformers
Step decay: LR × factor at milestones — classic ResNet recipe.
#schedules#learning-rate
Practise Deep Learning
214 interview questions in this topic.