EasyDeepLearn
Deep Learning · section 7 of 19

Normalization

9 interview questions on normalization, each answered in full. Free to read, no account needed.

When do you use LayerNorm instead of BatchNorm?

medium
  • Use LayerNorm when batch size is small or varies (RNNs, transformers, sequence data).
  • LayerNorm computes statistics across features within a single example, so it doesn't depend on batch statistics.
  • That's why it's standard in transformers and language models where sequence lengths vary and batches can be tiny.
#normalizationPermalink & quiz →

What is GroupNorm and when do you use it?

medium
  • Divide channels into G groups, normalize each group per example.
  • No dependence on batch size — works with batch=1, useful for detection / segmentation where memory forces small batches, or for very high-resolution training.
  • Middle ground between LayerNorm (all channels one group) and InstanceNorm (each channel its own group).
  • Standard in Detectron2, MMDetection, and small-batch computer vision.
#normalizationPermalink & quiz →

What is InstanceNorm and where does it shine?

medium
  • Normalize each channel independently within one example.
  • Removes per-image style / contrast information — hence its use in style transfer and image-to-image translation (CycleGAN, StyleGAN).
  • The scale/shift then allow the network to inject any desired 'style'.
  • Not usually used for classification because it destroys instance-specific information that helps discrimination.
#normalization#generativePermalink & quiz →

What is Weight Normalization and its trade-off?

medium
  • Reparameterize weights as w = g * v / ||v||, decoupling magnitude (g, scalar per neuron) from direction (v, vector).
  • Data-independent normalization — no batch statistics.
  • Cheaper than BN, useful in RNNs and RL where BN struggles.
  • Trade-off: less regularization than BN and worse convergence in most modern CNNs and transformers, so it's rarely used today.
#normalizationPermalink & quiz →

How does RMSNorm differ from LayerNorm?

medium
  • RMSNorm(x) = x / RMS(x) * γ, where RMS(x)  =  sqrt(mean(x2)  +  ε)\mathrm{RMS}(x)\; = \;\mathrm{sqrt}(\mathrm{mean}(x^{2})\; + \;{\varepsilon}).
  • Skips the mean-subtraction and β (bias) of LayerNorm.
  • About 10-30% faster (fewer ops), similar accuracy on transformers.
  • Adopted by LLaMA, T5, and most 2023+ open LLMs.
#normalization#transformersPermalink & quiz →

Pre-norm vs post-norm transformers — which is standard and why?

hard
  • Original Transformer (Vaswani 2017) used post-norm: x + Attn(x), then LayerNorm.
  • Trains poorly at scale (deep post-norm nets need long warmup).
  • Pre-norm applies LayerNorm before the sublayer: x + Attn(LN(x)) — gradients flow through the identity skip unchanged, trains stably with very deep nets and short warmup.
  • All modern LLMs use pre-norm.
#normalization#transformersPermalink & quiz →

What is Synchronized BatchNorm (SyncBN) and when do you need it?

medium
  • In data-parallel training, each GPU has a mini-batch — regular BN computes stats per GPU.
  • If per-GPU batch is small (e.g., 2-4 for detection / segmentation), stats are noisy and accuracy suffers.
  • SyncBN aggregates mean and variance across all GPUs each forward pass, giving one global batch statistic.
  • Slower (extra collective), but essential for dense-prediction tasks with small per-GPU batch.
#normalization#distributedPermalink & quiz →

Why is BatchNorm awkward in RNNs?

medium
  • RNNs share weights across time; you'd need separate BN statistics for every timestep to be principled, but sequences have variable length.
  • Also, small effective batch (batch × time reshape doesn't help because time is not exchangeable), and running stats depend on sequence length.
  • LayerNorm has none of these issues — per-example, per-timestep, unconditional on batch.
  • That's why LSTMs and transformers use LN, not BN.
#normalization#rnnPermalink & quiz →

Why did transformers switch from post-norm to pre-norm at scale?

hard
  • Original transformer used post-norm: LayerNorm(x + sublayer(x)).
  • At depth > 12, the sublayer outputs interfere with LN's running scale, causing training instabilities that require very long warmup or careful initialization.
  • Pre-norm — x + sublayer(LayerNorm(x)) — puts the LayerNorm inside the residual branch, leaving a clean identity path for gradients.
  • Trains stably for 100+ layers with short warmup.
  • All modern LLMs use pre-norm.
#transformers#normalizationPermalink & quiz →

Practise Deep Learning

All 19 sectionsFlashcardsQuiz