What is Weight Normalization and its trade-off?
mediumAnswer
- Reparameterize weights as w = g * v / ||v||, decoupling magnitude (g, scalar per neuron) from direction (v, vector).
- Data-independent normalization — no batch statistics.
- Cheaper than BN, useful in RNNs and RL where BN struggles.
- Trade-off: less regularization than BN and worse convergence in most modern CNNs and transformers, so it's rarely used today.
Check yourself — multiple choice
- WN normalizes activations
- Reparam w = g·v/||v|| — decouples norm from direction; batch-independent but weaker than BN today
- WN depends on batch size
- WN is deprecated
Weight Norm: w = g · v/||v|| — data-independent but rarely beats BN/LN in modern nets.
#normalization
Practise Deep Learning
214 interview questions in this topic.