How do you autoscale LLM serving?
hardAnswer
- Metrics: (1) queue depth / concurrent requests; (2) P99 latency SLA breach; (3) GPU utilization > threshold.
- Scale-up: launch new pod / container with warm model (cold-start on 70B can take 60+ seconds due to weight loading).
- Scale-down: only when queue is empty and utilization is low for N minutes — protects against oscillation.
- Warmup: pre-load a shard of common prompts into KV cache.
- Kubernetes with GPU node pools + HPA is the standard stack; managed services (SageMaker, Vertex) abstract this.
Check yourself — multiple choice
- Scale on CPU only
- Queue depth + P99 SLA + GPU util as signals; warm-model pre-load; slow scale-down to prevent oscillation; cold-start on 70B is 60+s
- No autoscaling possible
- Instant scale-up
LLM autoscale: queue / SLA / util metrics + warm loads + slow scale-down.
#serving#production#latency
Practise LLMs & GenAI
214 interview questions in this topic.
Related questions
- What SLIs / SLOs are typical for LLM serving?
- Why is streaming output important in production LLM apps?
- How do you trade off throughput vs latency in LLM serving?
- vLLM vs TensorRT-LLM vs TGI vs SGLang — how do you pick?
- What role does NVIDIA Triton play in LLM serving?
- How do you pick a GPU for serving a 70B LLM?