How does snapshot ensembling work?
mediumAnswer
- Train with a cyclical LR schedule (SGDR / cosine with restarts).
- Save a checkpoint at each cycle's minimum LR (one snapshot per cycle).
- At inference, ensemble the K snapshots by averaging predictions.
- Same training cost as a single training run, but ensemble diversity comes from the LR restarts landing in different basins.
Check yourself — multiple choice
- Train K separate models
- Save one snapshot per cyclical LR cycle → ensemble at inference at no extra training cost
- Same as bagging
- Only for CNNs
Snapshot ensembling: SGDR + save one snapshot per cycle → free ensemble diversity.
#training#schedules
Practise Deep Learning
214 interview questions in this topic.