How do you pick the head dimension $d_{\mathrm{head}}$ ?

hard

Answer

$d_{\mathrm{head}}\; = \;d\; / \;n_{\mathrm{heads}}$
$d_{\mathrm{head}}\; = \;64 - 128$
$Grouped-query attention (GQA, MQA) share K/V across groups of heads to reduce KV cache without hurting quality much.$

Check yourself — multiple choice

$d_{\mathrm{head}}$ is fixed at 1
$d_{\mathrm{head}}\; = \;d / n_{\mathrm{heads}}$ typically; too small underfits, too big wastes params; GQA/MQA share K/V to save KV cache
$d_{\mathrm{head}}$ can only be 512
Doesn't matter

$d_{\mathrm{head}}$ ≈ 64-128 typical; GQA/MQA share K/V to cut inference memory.

#attention#transformers

214 interview questions in this topic.