How do you serve multiple models efficiently on shared infrastructure?
hardAnswer
- (1) Multi-model server (TF Serving / Triton): load N models per container; route by request.
- (2) Model swap on GPU: pre-load, unload cold.
- (3) Weight sharing: distill many task-specific into one MoE / adapter model.
- (4) A/B / champion-challenger co-deploy for comparison.
- (5) Autoscale per model based on QPS.
- Tradeoffs: cold-start latency, memory pressure, tail latency variance.
- Modern: LoRA adapters share base weights across tasks (LLM serving).
Check yourself — multiple choice
- Random
- Multi-model server (TF/Triton) + GPU swap + weight sharing (MoE/adapter) + co-deploy for compare + per-model autoscale; watch cold-start + memory + tail; LoRA adapters share base
- One at a time
- Not real
Multi-model: Triton + swap + share weights + LoRA + autoscale.
#deployment#infrastructure
Practise MLOps & Data Quality
215 interview questions in this topic.