Describe the LR-range finder (Smith, 2015).
easyAnswer
- Train the model for a few epochs with the learning rate increased exponentially from very small to very large.
- Plot loss vs LR.
- Pick a LR one order of magnitude below where the loss starts to diverge (or where the negative slope is steepest).
- Fast heuristic — replaces manual grid search over the most important hyperparameter.
- Available in fastai and PyTorch Lightning.
Check yourself — multiple choice
- Random search over LR
- Exponentially ramp LR up during a short run, pick 1 dec below divergence — fast heuristic
- Fixed 1e-3 for every model
- Uses no training data
LR finder: exponential ramp → plot loss vs LR → pick LR just below divergence.
#learning-rate#schedules#training
Practise Deep Learning
214 interview questions in this topic.