Why multi-head attention instead of a single big head?

medium

Answer

$Different heads can attend to different types of relationships (syntactic, semantic, positional) in different subspaces.$
$Multi-head with H heads of dim d/H has similar compute to one head of dim d, but the diversity of learned attention patterns empirically helps a lot.$
$Also enables interpretation via per-head attention maps.$
$Typical: 8-32 heads for d ~ 512-4096.$

Check yourself — multiple choice

Multi-head is slower and worse
Different heads attend to different relationship types in different subspaces — richer patterns at similar compute
Multi-head has no purpose
Only one head is ever used

Multi-head: parallel attention in different subspaces → diverse relationships captured.

#attention#transformers

214 interview questions in this topic.