What are cyclical learning rates (CLR)?
mediumAnswer
- Instead of monotonic decay, oscillate LR between a low and a high bound in periodic triangular / sinusoidal / exp cycles.
- Helps escape shallow local minima (high LR periods) and refines fits (low LR periods).
- Underlying insight: SGD is a stochastic process — periodic 'heating' explores basins, 'cooling' exploits them.
- Precursor to SGDR / cosine with restarts.
Check yourself — multiple choice
- LR always decreases
- Oscillate LR between low and high bounds — escape minima on the up, refine on the down
- Cyclical means momentum only
- Only for classification
CLR: periodic LR heating/cooling — good exploration/exploitation trade-off.
#schedules#learning-rate
Practise Deep Learning
214 interview questions in this topic.