vLLM — why is it fast for LLM serving?
hardAnswer
- (1) PagedAttention: virtual memory for KV cache (page-based), eliminates fragmentation → 2-4x throughput.
- (2) Continuous batching: at each step, start new sequences + finish completed ones (no waiting for batch to complete).
- (3) Optimized CUDA kernels.
- (4) Speculative decoding integration.
- (5) Quantization (AWQ, GPTQ, FP8).
- (6) Multi-GPU tensor parallelism.
- UC Berkeley project, now industry standard.
- Alternatives: TGI (HF), TensorRT-LLM (NVIDIA), SGLang, LMDeploy.
Check yourself — multiple choice
- Random
- PagedAttention (paged KV cache eliminates fragmentation) + continuous batching (start new / finish complete each step) + CUDA kernels + speculative + AWQ/GPTQ/FP8 quant + TP; alternatives TGI/TRT-LLM/SGLang/LMDeploy
- Just fast
- Not real
vLLM: PagedAttention + continuous batching + quant + TP.
#serving#llmops
Practise MLOps & Data Quality
215 interview questions in this topic.