Why is attention scaled by ?
hardAnswer
- Assume q, k have entries with mean 0 and variance 1 (post-init or post-normalization).
- Their dot product has variance .
- As grows (say, 64+), un-scaled dot products become large in magnitude, pushing softmax into the saturated regime where one weight is ~1 and the rest ~0 → tiny gradients w.r.t. the losing keys.
- Dividing by rescales the variance back to 1 and keeps softmax gradients healthy.
Check yourself — multiple choice
- Scaling reduces expressiveness
- Dot product has variance under standard init → 1/√ rescales to variance 1 → healthy softmax gradients
- Purely for numerical reasons
- Any scale works
1/√ keeps dot-product variance ~1 → softmax stays in the linear regime.
#attention#transformers
Practise Deep Learning
214 interview questions in this topic.
Related questions
- How does self-attention work in a transformer?
- Describe a single transformer encoder block in detail.
- How does a decoder block differ from an encoder block?
- Why multi-head attention instead of a single big head?
- How do you pick the head dimension d_head?
- Why is standard self-attention O(n²) in sequence length?