How does self-attention work in a transformer?

hard

Answer

$For each token, produce a Query, Key, and Value vector.$
$\operatorname{softmax}(Q_{i}\; \cdot \;K_{j}\; / \;\mathrm{sqrt}(d_{k}))$
$Multi-head attention runs this in parallel with different projections and concatenates the results.$
$O(n^{2})$

Check yourself — multiple choice

Attention weights are computed as Q · V
Weights are $\operatorname{softmax}(Q\; \cdot \;K\; / \;\mathrm{sqrt}(d_{k}))$ applied to V
Attention only looks at neighboring tokens
Multi-head means multiple layers stacked in series

That's the scaled dot-product attention formula.

#attention#transformers

214 interview questions in this topic.