Why is cold-starting an LLM slow and how do you fix it?
hardAnswer
- Weight loading: 70B in FP16 = 140GB over PCIe / network → 30-90 seconds.
- Kernel compilation: TensorRT-LLM / Triton compile kernels on first request.
- Fixes: (1) warm pools — keep min-replicas > 0; (2) tensor-parallel across faster memory tiers; (3) fp8 / int4 weights (smaller = faster load); (4) shard weights on local NVMe with mmap; (5) pre-warmed base image with weights baked in; (6) serverless with per-request cold-start needs weight-in-cache.
- LLM cold-start is significantly worse than typical microservices.
Check yourself — multiple choice
- Instant cold start
- 70B FP16 = 140GB → 30-90s to load; fix via warm pools, quantization, NVMe mmap, pre-warmed images
- Same as HTTP servers
- Not solvable
LLM cold-start: slow (huge weights); fix via warm pools + quantization + NVMe.
#serving#production#cost
Practise LLMs & GenAI
214 interview questions in this topic.
Related questions
- How do you pick a GPU for serving a 70B LLM?
- What is a 'model router' and when is it worth it?
- How much does it cost to pretrain a modern LLM?
- How should max_tokens be set in production?
- What is prompt caching and when does it help?
- How do you use one prompt to extract multiple fields at once vs many separate prompts?