When is orthogonal weight initialization useful?
hardAnswer
- Orthogonal init sets weight matrices to random orthogonal matrices (singular values all 1), which preserves the norm of the input under linear map.
- Especially useful for very deep networks and RNNs where repeated multiplication amplifies or shrinks signals — orthogonal keeps things unit-norm.
- Slightly better than He on very deep nets without normalization.
Check yourself — multiple choice
- Orthogonal init has random singular values
- Sets weight matrices to orthogonal — preserves signal norm through very deep layers and RNNs
- Same as Xavier
- Not useful in modern networks
Orthogonal matrices preserve norms — great for very deep nets and RNNs.
#initialization#rnn
Practise Deep Learning
214 interview questions in this topic.
Related questions
- Why did transformers replace RNNs for sequence modeling?
- What is Xavier (Glorot) initialization and why?
- Why does He initialization use Var = 2 / fan_in instead of Xavier's 2 / (fan_in + fan_out)?
- Why can't you initialize a neural net with all zeros?
- What causes exploding gradients and how do you handle them?
- Why is BatchNorm awkward in RNNs?