EasyDeepLearn

What is an EMA of weights and why do modern training recipes use it?

medium

Answer

  • Maintain a shadow copy of the weights θema{\theta}_{\mathrm{ema}} updated as θema{\theta}_{\mathrm{ema}} ← α  ⋅  θema  +  (1−α)  ⋅  θ{\alpha}\; \cdot \;{\theta}_{\mathrm{ema}}\; + \;(1 - {\alpha})\; \cdot \;{\theta} every step, with α ~ 0.999-0.9999.
  • Evaluate / deploy with θema{\theta}_{\mathrm{ema}}, not the raw training weights.
  • The EMA lives at the center of a flat minimum → ~0.5-1% better accuracy for free, better calibration, more stable predictions.
  • Widely used in diffusion training (essential for sample quality), BYOL / DINO self-supervised methods (EMA teacher), and modern classification recipes.
Check yourself — multiple choice
  • EMA is only a sanity check
  • Exponential moving average of weights → evaluate on EMA copy → better accuracy, calibration, stability; essential in diffusion / SSL teachers
  • Same as gradient accumulation
  • Deprecated technique

Weight EMA: shadow copy at the center of a flat minimum → free accuracy + stability.

#training#regularization#distillation

Practise Deep Learning

214 interview questions in this topic.

Read the lessonFlashcardsTake the quiz

Related questions