What is SGDR (warm restarts) and when does it help?

medium

Answer

$\mathrm{LR}_{\mathrm{max}}$
$Enables ensembling by saving one model per cycle (Snapshot Ensembles).$
$Helps escape sharp minima and revisits the loss surface.$
$Widely used in vision competitions and image-classification recipes; less common in transformers where a single cosine schedule dominates.$

Check yourself — multiple choice

SGDR removes cosine annealing
Cosine decay with periodic restarts back to $\mathrm{LR}_{\mathrm{max}}$ — enables snapshot ensembling
SGDR is a data-augmentation method
Only for RNNs

SGDR: cosine decay + restarts → snapshot ensembles + better exploration.

#schedules#learning-rate

214 interview questions in this topic.