Why can't you initialize a neural net with all zeros?
easyAnswer
- Symmetric weights mean every neuron in a layer computes the same output and receives the same gradient — the layer effectively has one neuron.
- Training never breaks the symmetry.
- Break it with any random init (Xavier / He / uniform).
- Biases can be initialized to zero because they aren't multiplied together; only the weight matrices need randomness.
Check yourself — multiple choice
- It's fine as long as you use momentum
- Zero-init makes neurons in a layer symmetric — they never differentiate
- Only biases need randomness
- Zero-init makes training faster
Zero weights ⇒ identical gradients ⇒ symmetry never broken.
#initialization#fundamentals
Practise Deep Learning
214 interview questions in this topic.
Related questions
- ReLU vs sigmoid vs GELU — when do you use each?
- In one sentence, what is backpropagation?
- State the universal approximation theorem in one sentence and its practical caveat.
- Why does stacking linear layers without nonlinearity collapse to a single linear layer?
- Depth vs width — which do you scale first?
- What is Xavier (Glorot) initialization and why?