EasyDeepLearn
Deep Learning · section 6 of 19

Learning-rate schedules

7 interview questions on learning-rate schedules, each answered in full. Free to read, no account needed.

What are cyclical learning rates (CLR)?

medium
  • Instead of monotonic decay, oscillate LR between a low and a high bound in periodic triangular / sinusoidal / exp cycles.
  • Helps escape shallow local minima (high LR periods) and refines fits (low LR periods).
  • Underlying insight: SGD is a stochastic process — periodic 'heating' explores basins, 'cooling' exploits them.
  • Precursor to SGDR / cosine with restarts.
#schedules#learning-ratePermalink & quiz →

Why do transformers need learning-rate warmup?

medium
  • At the start of training, weights are random and LayerNorm statistics are meaningless.
  • A high LR causes large gradients that destabilize LayerNorm's running scale and can permanently damage the network (attention heads collapse).
  • Warmup ramps LR linearly from 0 to peak over a few thousand steps, giving the network time to stabilize activations before high-LR updates.
#schedules#learning-rate#transformersPermalink & quiz →

What is cosine annealing and why is it preferred over step decay?

medium
  • LR(t) = LR_min + 0.5 * (LR_max - LR_min) * (1 + cos(π * t / T)).
  • Starts high, decays smoothly to LRmin\mathrm{LR}_{\mathrm{min}} over T steps.
  • Smoother than step decay (no sudden LR jumps that upset momentum), typically better final accuracy on transformers and CNNs.
  • Combined with warmup, it's the de-facto schedule for modern large-model training.
#schedules#learning-ratePermalink & quiz →

When is step decay still appropriate?

easy
  • Multiply LR by a factor (e.g., 0.1) at fixed milestones (e.g., 60/90 epochs on ImageNet).
  • Simple, deterministic, well-understood — classic recipe for training ResNet-style CNNs with SGD+momentum.
  • Downsides: sudden LR jumps interact poorly with running momentum, need careful tuning of milestones.
  • Cosine is smoother but step decay is fine for well-studied recipes.
#schedules#learning-ratePermalink & quiz →

How does ReduceLROnPlateau work?

easy
  • Track a validation metric; if it doesn't improve for 'patience' epochs, multiply LR by a factor (e.g., 0.5).
  • Adaptive to the actual loss curve — doesn't require guessing the schedule in advance.
  • Good default for prototyping.
  • Downside: introduces validation feedback into training, and its aggressiveness (patience, factor) matters a lot.
#schedules#learning-ratePermalink & quiz →

What is SGDR (warm restarts) and when does it help?

medium
  • Cosine annealing with periodic restarts back to LRmax\mathrm{LR}_{\mathrm{max}}: each 'cycle' cools LR to LRmin\mathrm{LR}_{\mathrm{min}} then jumps back up.
  • Enables ensembling by saving one model per cycle (Snapshot Ensembles).
  • Helps escape sharp minima and revisits the loss surface.
  • Widely used in vision competitions and image-classification recipes; less common in transformers where a single cosine schedule dominates.
#schedules#learning-ratePermalink & quiz →

Why does transformer training usually need a learning-rate warmup?

hard
  • At initialization, Adam's second-moment estimate is based on almost no history, so its normalization is unreliable and the first updates can be enormous relative to the weights.
  • Large early steps push a transformer into a region from which it never fully recovers, often visible as an attention collapse or an immediate loss spike.
  • Warmup ramps the learning rate from near zero over a few hundred to a few thousand steps, giving the optimizer state time to become meaningful before large steps are taken.
  • Post-layer-norm architectures need it most, because gradients through the residual path are badly scaled at initialization; pre-layer-norm and careful residual scaling reduce the dependence but rarely remove it entirely.
#schedules#learning-rate#transformersPermalink & quiz →

Practise Deep Learning

All 19 sectionsFlashcardsQuiz