What is Stochastic Weight Averaging (SWA)?
mediumAnswer
- During the last part of training (with a constant or cyclical LR), keep a running average of the weights: of across recent steps.
- Use at inference.
- Averaging in the loss landscape moves toward the center of a flat minimum, which generalizes better than any individual point.
- Adds ~0 cost and often gains 0.5-1% accuracy.
- Extended to EMA of weights (used in EMA-teacher / EMA distillation).
Check yourself — multiple choice
- SWA averages predictions
- Average weights over the last training steps → flatter effective minimum, better generalization
- Same as gradient averaging
- Doubles training cost
SWA: weight averaging → moves to the center of a flat minimum → better generalization.
#training#regularization
Practise Deep Learning
214 interview questions in this topic.
Related questions
- How does dropout work and when is it applied?
- What is stochastic depth / DropPath and where is it used?
- What is early stopping and how do you configure it?
- What is adversarial training (PGD, FGSM)?
- What is an EMA of weights and why do modern training recipes use it?
- Your validation loss is lower than your training loss. Is something broken?