How do you quantize the KV cache and why?

hard

Answer

$KV cache typically dominates VRAM at long context.$
$Quantize K and V from FP16 \to INT8 or FP8 per token.$
$Per-head scaling to preserve accuracy.$
$Halves or quarters KV memory \to allows longer context or bigger batch on the same GPU.$
$Small quality impact (< 1% perplexity) with careful implementation.$
$Supported in vLLM, TensorRT-LLM.$
$Combined with GQA / MQA (share KV across heads), you get 4-16\times KV cache reduction \to serving 128k context on a single H100.$

Check yourself — multiple choice

Never quantize KV
Per-head INT8/FP8 quantization of K and V → 2-4× smaller KV cache with <1% quality loss; combines with GQA
Only quantize weights
Impossible

KV cache quant: INT8/FP8 with per-head scale → longer context, bigger batches.

#quantization#kv-cache#inference

214 interview questions in this topic.