What is Synchronized BatchNorm (SyncBN) and when do you need it?
mediumAnswer
- In data-parallel training, each GPU has a mini-batch — regular BN computes stats per GPU.
- If per-GPU batch is small (e.g., 2-4 for detection / segmentation), stats are noisy and accuracy suffers.
- SyncBN aggregates mean and variance across all GPUs each forward pass, giving one global batch statistic.
- Slower (extra collective), but essential for dense-prediction tasks with small per-GPU batch.
Check yourself — multiple choice
- SyncBN is only for single-GPU
- Aggregates BN stats across all GPUs — needed when per-GPU batch is small (detection/segmentation)
- Removes normalization entirely
- SyncBN is faster than regular BN
SyncBN: collective all-reduce of BN stats — critical for small per-GPU batches.
#normalization#distributed
Practise Deep Learning
214 interview questions in this topic.