What is cosine annealing and why is it preferred over step decay?

medium

Answer

$LR(t) = LR_min + 0.5 * (LR_max - LR_min) * (1 + cos(π * t / T)).$
$\mathrm{LR}_{\mathrm{min}}$
$Smoother than step decay (no sudden LR jumps that upset momentum), typically better final accuracy on transformers and CNNs.$
$Combined with warmup, it's the de-facto schedule for modern large-model training.$

Check yourself — multiple choice

Cosine decay increases LR over time
Smooth cosine-shaped decay from $\mathrm{LR}_{\mathrm{max}}$ to $\mathrm{LR}_{\mathrm{min}}$ — smoother than step decay, standard with warmup
Cosine is identical to exponential
Only used at inference

Cosine decay: smooth LR reduction — no shocks, better final loss than step decay.

#schedules#learning-rate

214 interview questions in this topic.