EasyDeepLearn

What is the KV cache in transformer inference?

hard

Answer

  • During autoregressive generation, tokens are produced one at a time.
  • For each layer, the K and V projections of all previously generated tokens are cached; only the new token's Q, K, V are computed each step.
  • Time per new token drops from O(n2)O(n^{2}) to O(n).
  • Memory cost: 2  ⋅  nlayers  ⋅  nheads  ⋅  dhead  ⋅  seqlen2\; \cdot \;n_{\mathrm{layers}}\; \cdot \;n_{\mathrm{heads}}\; \cdot \;d_{\mathrm{head}}\; \cdot \;\mathrm{seq}_{\mathrm{len}} per sequence — often dominates GPU memory in long-context serving.
  • Techniques: MQA/GQA (share K/V across heads), Paged attention (vLLM), sliding-window caches.
Check yourself — multiple choice
  • Cache is only for training
  • Cache K, V of prior tokens during autoregressive generation → O(n) per new token; memory dominant → MQA/GQA/Paged
  • Same as gradient checkpointing
  • KV cache removes attention

KV cache: reuse K/V of past tokens → O(n) per step; memory optimizations = MQA/GQA/Paged.

#attention#transformers#efficient-attention

Practise Deep Learning

214 interview questions in this topic.

Read the lessonFlashcardsTake the quiz

Related questions