26 interview questions on inference: caching, batching, serving, each answered in full. Free to read, no account needed.
What is quantization and what tradeoffs come with it?
medium- Quantization stores weights (and sometimes activations) in lower precision — INT8, INT4, or even lower — reducing memory and speeding up inference.
- Post-training quantization is cheap but can hurt quality; quantization-aware training keeps accuracy closer to FP16. 4-bit weight-only quantization (e.g., GPTQ, AWQ, bitsandbytes NF4) is the sweet spot for running big LLMs on smaller GPUs.
What is the KV cache and why does it matter for LLM inference?
hard- During autoregressive generation, attention needs the Keys and Values of all previous tokens.
- Recomputing them at every step is wasteful, so we cache them and only compute K/V for the new token.
- Effect: per-token generation becomes O(n) instead of O(n2).
- Downside: memory grows with context length — the KV cache often dominates VRAM usage in long-context inference.
What is prompt caching and when does it help?
hard- Cache the KV attention state of a large prefix (system prompt, few-shot examples, retrieved context) and reuse it across many requests that share that prefix.
- On subsequent calls, prefill only the new suffix — huge savings on TTFT (10-100x) and cost (50-90%).
- Supported by Anthropic API (explicitcachecontrol), Google (implicit), OpenAI (implicit prefix cache), and open-source (vLLM automatic prefix caching).
- Essential for RAG apps and agent loops with stable system prompts.
What's the difference between prefill and decode in LLM inference?
medium- Prefill: process the entire input prompt in one parallel forward pass — compute-bound (GPU tensor cores at 90%+ utilization), fast (thousands of tokens/sec/GPU).
- Produces the KV cache.
- Decode: generate output tokens one at a time — memory-bandwidth bound (reload weights + KV cache per step), slow (30-200 tokens/sec).
- This split matters for cost / SLA: TTFT (time-to-first-token) depends on prefill; throughput on decode.
- Different optimizations target each phase (chunked prefill, continuous batching for decode).
What is continuous / dynamic batching in LLM serving?
hard- Naive static batching: batch of N requests, wait for the slowest to finish, then start the next batch → many GPU cycles wasted.
- Continuous batching (Yu 2022, vLLM): as soon as any request in the batch finishes, remove it and slot in a new request at that free position — GPU stays saturated.
- Combined with paged attention (KV cache in fixed pages), enables mixing requests of different lengths seamlessly.
- Typical throughput gain: 2-10x over static batching.
- Standard in vLLM, TGI, TensorRT-LLM.
How does Paged Attention work?
hard- vLLM (Kwon 2023) allocates the KV cache in fixed-size 'pages' (e.g., 16 tokens each) via a page table — like OS virtual memory.
- Benefits: (1) no fragmentation as sequences grow / shrink; (2) memory sharing across sequences with a common prefix (parallel sampling, beam search); (3) enables continuous batching with variable-length sequences.
- Reduces KV-cache memory waste from ~60% (naive contiguous allocation) to ~4%.
- Foundation of vLLM's throughput advantage.
What is chunked prefill and why is it useful?
hard- Split a long prompt's prefill into chunks of a few hundred tokens and interleave them with decode steps of other requests in the same batch.
- Prevents a single long prompt from starving decode users (a 32k-token prefill can take seconds → other users see TTL spikes).
- Reduces P99 TTFT dramatically without hurting throughput.
- Enabled by continuous batching + Paged Attention.
- Standard in modern serving stacks (vLLM, SGLang, TensorRT-LLM).
How do you trade off throughput vs latency in LLM serving?
medium- Larger batch size → higher throughput (better GPU utilization) but higher latency per request (waiting for batchmates).
- Smaller batch → lower latency, worse throughput.
- Latency-oriented deployments (chat) use small batches + speculative decoding + FP8.
- Throughput-oriented (offline batch processing, embeddings) use huge batches + higher context length.
- Modern serving frameworks (vLLM, TGI) let you set maxbatchsize and maxbatchtokens knobs.
- Measure both P50 and P99 latency + tokens/sec throughput.
How does INT8 weight quantization work in practice?
medium- For each weight tensor W (fp16, shape [out, in]), compute a per-channel scale=max∣Wi| / 127 and quantized weight Wq=round(W/scale) as int8.
- At inference, compute Wq @ xfp16 → dequantize output with scale.
- Modern kernels (bitsandbytes, torch quant, TensorRT) fuse quantize + matmul + dequantize into a single INT8 tensor core op.
- Memory: 2× less; latency: 1.5-2× faster on GPUs with INT8 tensor cores.
- Accuracy loss: usually < 0.5% on quality benchmarks.
Compare GPTQ, AWQ, and NF4 quantization.
hard- GPTQ (Frantar 2022): post-training layer-by-layer 4-bit weight quantization that minimizes reconstruction error via a Hessian-aware update.
- Precise, slow to compute (hours for 70B).
- AWQ (Lin 2023): 'Activation-aware Weight Quantization' — preserves the small fraction of weights corresponding to high-magnitude activations at higher precision.
- Faster than GPTQ, similar quality.
- NF4 (bitsandbytes / QLoRA): 4-bit 'NormalFloat' data type designed for normally-distributed weights → simple, works well as base for QLoRA fine-tuning.
- Practical: GPTQ/AWQ for pure serving, NF4 for QLoRA training.
What is FP8 inference and where does it help?
hard- 8-bit floating-point (typically E4M3 or E5M2 formats) available on H100 / H200 / MI300 GPUs.
- Better dynamic range than INT8 (activations don't need per-channel scaling calibration), 2x faster than FP16 on tensor cores.
- Used for both weights and activations in modern serving stacks (TensorRT-LLM FP8, vLLM FP8-KV cache).
- Quality: near-zero loss vs FP16.
- FP8-KV cache alone halves KV memory → longer context or larger batch.
How do you quantize the KV cache and why?
hard- KV cache typically dominates VRAM at long context.
- Quantize K and V from FP16 → INT8 or FP8 per token.
- Per-head scaling to preserve accuracy.
- Halves or quarters KV memory → allows longer context or bigger batch on the same GPU.
- Small quality impact (< 1% perplexity) with careful implementation.
- Supported in vLLM, TensorRT-LLM.
- Combined with GQA / MQA (share KV across heads), you get 4-16× KV cache reduction → serving 128k context on a single H100.
When do you distill an LLM for serving?
medium- When latency / cost constraints require a smaller model than the frontier.
- Train a smaller student on outputs from a larger teacher, either via: (1) SFT on teacher's greedy outputs (imitation); (2) matching teacher's logit distribution at high temperature (KD loss); (3) mining teacher's chain-of-thought traces.
- Typical setup: distill a 70B teacher into a 7B / 13B student for 90-95% of the quality at 10× cheaper serving.
- DistilBERT (2019) was the first big example; today used everywhere (Zephyr, Qwen, Phi).
vLLM vs TensorRT-LLM vs TGI vs SGLang — how do you pick?
medium- vLLM: open-source, easy to install, wide model support, strong throughput.
- TensorRT-LLM: NVIDIA's optimized engine (compiled kernels), best raw performance on H100 but complex to build.
- TGI (HuggingFace): production-friendly, good streaming, decent perf.
- SGLang: high-perf, native support for structured / constrained decoding, growing ecosystem.
- For most teams: vLLM by default.
- Move to TensorRT-LLM if squeezing every % matters at scale.
- TGI for HF ecosystem integration.
- SGLang for heavy structured output workloads.
What does 'automatic prefix caching' do in vLLM?
hard- vLLM hashes the token prefix of each request.
- When two requests share a prefix (same system prompt + few-shot examples), the second one reuses the first's KV cache for that prefix — only its unique suffix needs prefill.
- Massive speedup for RAG / chat where system prompts + retrieved context are shared.
- Works transparently; users don't need to declare a cache key.
- Hit rate depends on prefix stability; agent loops with stable system prompts see 60-90% cache hit.
What role does NVIDIA Triton play in LLM serving?
medium- Triton Inference Server is a generic model server: HTTP / gRPC front-end, multi-model support, dynamic batching, model versioning, ensemble models (chain inputs across models).
- Often wraps a TensorRT-LLM or ONNX runtime backend.
- Handles the productization concerns (rate limiting, health checks, GPU sharing between models) that vLLM / TGI don't focus on.
- Combined with TensorRT-LLM, common enterprise stack.
- Cloud alternatives: SageMaker LMI, Vertex AI, Bedrock — abstract this away.
How do you pick a GPU for serving a 70B LLM?
hard- 70B in FP16 = 140 GB VRAM for weights alone → doesn't fit on a single A100 80GB or H100 80GB.
- Options: (1) tensor-parallel across 2 GPUs (2× H100/A100); (2) 4-bit quantize to ~35GB → fits on 1× H100 80GB with room for KV cache.
- Per-token throughput: A100 ~30-40 tokens/sec, H100 ~50-70 tokens/sec (dense), H200 ~100+.
- For latency-sensitive apps: H100/H200 (higher memory bandwidth).
- For throughput: batch on H100 or A100.
- Consumer options (RTX 4090 24GB) can run 7B-13B quantized but not 70B.
What are typical tokens/sec numbers for popular LLMs on H100?
medium- Rough per-GPU decode throughput on H100 (batch 1, FP8, vLLM): 7B ~200-300 tok/s, 13B ~120-180, 70B (tensor-parallel 2×H100) ~50-80.
- With continuous batching, aggregate throughput scales 5-10× (many concurrent requests).
- Numbers vary by prompt length, quantization, framework.
- Reference numbers help capacity planning: 1 M2 Ultra 60GB gets ~30 tok/s on 70B Q4; RTX 4090 ~50 tok/s on 13B Q4.
How do you autoscale LLM serving?
hard- Metrics: (1) queue depth / concurrent requests; (2) P99 latency SLA breach; (3) GPU utilization > threshold.
- Scale-up: launch new pod / container with warm model (cold-start on 70B can take 60+ seconds due to weight loading).
- Scale-down: only when queue is empty and utilization is low for N minutes — protects against oscillation.
- Warmup: pre-load a shard of common prompts into KV cache.
- Kubernetes with GPU node pools + HPA is the standard stack; managed services (SageMaker, Vertex) abstract this.
What SLIs / SLOs are typical for LLM serving?
medium- SLIs: TTFT P50/P95 (time-to-first-token), P95 tokens/second during decode, request success rate, tool-call success rate.
- SLOs (typical for chat): TTFT P95 < 1s, decode > 30 tok/s, availability 99.9%.
- Track cost/request too.
- Monitoring stack: Datadog / Grafana for latency, Prometheus for GPU metrics, W&B / Langfuse for LLM-specific traces (prompts, retrievals, tool calls, hallucination flags).
- Trace per-request end-to-end for debugging.
Why is cold-starting an LLM slow and how do you fix it?
hard- Weight loading: 70B in FP16 = 140GB over PCIe / network → 30-90 seconds.
- Kernel compilation: TensorRT-LLM / Triton compile kernels on first request.
- Fixes: (1) warm pools — keep min-replicas > 0; (2) tensor-parallel across faster memory tiers; (3) fp8 / int4 weights (smaller = faster load); (4) shard weights on local NVMe with mmap; (5) pre-warmed base image with weights baked in; (6) serverless with per-request cold-start needs weight-in-cache.
- LLM cold-start is significantly worse than typical microservices.
What LLM observability tools do you deploy?
medium- (1) LLM-specific tracing: Langfuse, LangSmith, Helicone, Braintrust — capture prompts, retrieved contexts, tool calls, outputs, latency, cost per trace.
- (2) Feedback: thumbs-up/down + free-text feedback tied to traces.
- (3) Prompt / model version tracking.
- (4) Automatic eval on production samples (LLM-as-judge).
- (5) Regression detection: alert when metrics drift after model or prompt change.
- (6) PII redaction before logging.
- Combine with metric stacks (Prometheus, Datadog) for GPU / latency.
How do you canary-deploy a new LLM version?
medium- (1) Route 1-5% of traffic to the new model, 95-99% to the current one.
- (2) A/B compare: quality metrics (LLM-judge, human eval on sampled outputs), latency, cost, error rate.
- (3) Include a fallback path if the canary throws or misbehaves.
- (4) Compare per-cohort (locale, tenant, use case) — regressions often show in subsets.
- (5) Roll forward gradually (5% → 25% → 50% → 100%) over hours / days with automated rollback triggers.
- Never a big-bang cutover on production LLM changes.
What is a 'model router' and when is it worth it?
hard- Cheap classifier that decides which downstream LLM to send each request to based on difficulty / cost / latency.
- E.g., simple factual questions → 8B cheap model; complex reasoning → 70B or GPT-4; multimodal → a vision-language model.
- Router can be a small transformer, a linear probe on embeddings, or an LLM-as-router.
- Rationale: 60-80% of production traffic doesn't need frontier quality — routing saves 3-10× on average cost.
- Trade-off: routing errors send hard queries to weak models.
- Route on a small % and A/B.
Your LLM feature costs too much per request. What levers do you pull, in order?
medium- Start with the cheapest wins.
- Cache: identical and near-identical requests are common, and a semantic cache can remove a large share of traffic.
- Shorten the prompt, since few-shot examples and boilerplate often dominate the token count, and prompt caching makes a stable prefix nearly free.
- Route by difficulty: send easy requests to a small model and escalate only when a cheap check says the answer is uncertain.
- Then trim output length, which usually costs more per token than input.
- Only after that consider distilling a small model on your own traffic, or self-hosting a quantized model, since both add real operational burden.
How do you make an LLM feature feel fast when generation is inherently slow?
medium- Optimize the metric the user feels, which is time to first token rather than total time.
- Stream tokens so reading begins immediately, and the perceived wait collapses to the prefill time.
- Do retrieval and other setup in parallel rather than in series before the call.
- Keep the prompt prefix stable so provider prefix caching cuts prefill.
- Show intermediate progress for agents, since a visible tool step reads as work rather than a hang.
- Where the output is long and structured, render sections as they arrive.
- Speculative decoding helps at the model level, but streaming and parallelism give the larger perceived win.