What is the KV cache and why does it matter for LLM inference?
hardAnswer
- During autoregressive generation, attention needs the Keys and Values of all previous tokens.
- Recomputing them at every step is wasteful, so we cache them and only compute K/V for the new token.
- Effect: per-token generation becomes O(n) instead of .
- Downside: memory grows with context length — the KV cache often dominates VRAM usage in long-context inference.
Check yourself — multiple choice
- The KV cache stores gradients for backprop
- The KV cache stores past Keys and Values so we don't recompute them
- The KV cache is only used during training
- The KV cache is independent of context length
Caching K/V of past tokens makes decoding much faster.
#kv-cache#inference
Practise LLMs & GenAI
214 interview questions in this topic.