What is Grouped-Query Attention (GQA) and why does it matter?
hardAnswer
- Standard multi-head attention: each head has its own K and V — the KV cache is heads × × seq × batch.
- MQA (Shazeer 2019): all heads share one K/V — much smaller KV cache, ~1% quality loss.
- GQA (Ainslie 2023): heads are grouped (e.g., 8 groups of 4 heads), each group shares K/V — 4-8x smaller KV cache, negligible quality loss.
- Used by LLaMA-2 70B, LLaMA-3, Mistral, Qwen.
- Critical for long-context inference where KV cache dominates VRAM.
Check yourself — multiple choice
- GQA increases KV cache
- Heads share K/V within groups → 4-8× smaller KV cache, negligible quality loss (LLaMA-2/3, Mistral)
- GQA is the same as MHA
- Only for training
GQA: grouped heads share K/V → dramatic KV cache reduction for long-context inference.
#architecture#attention#kv-cache#inference
Practise LLMs & GenAI
214 interview questions in this topic.
Related questions
- How does Flash Attention 2 speed up training?
- What is the context window and what tricks extend it?
- What is the KV cache and why does it matter for LLM inference?
- What is prompt caching and when does it help?
- What's the difference between prefill and decode in LLM inference?
- How does Paged Attention work?