How do you configure autoscaling for ML serving?
hardAnswer
- (1) Metric: QPS + GPU util + p95 latency (not just CPU).
- (2) HPA (K8s Horizontal Pod Autoscaler) with custom metrics via Prometheus adapter.
- (3) Scale out threshold + cooldown period (avoid flapping).
- (4) Min instances > 0 for warm capacity (avoid cold starts).
- (5) Max instances = budget ceiling.
- (6) Provisioned min + burst on demand (Cloud Run / Lambda).
- (7) Predictive scaling for known patterns (e.g., 9am weekday).
- Alternative: KEDA event-driven scaling.
Check yourself — multiple choice
- Just CPU
- Metric = QPS/GPU util/p95 latency (not just CPU); HPA + Prometheus adapter + threshold + cooldown + min>0 warm + max budget cap + provisioned min + predictive scale; KEDA event-driven alternative
- Random
- Not real
Autoscale: custom metrics + threshold + cooldown + min/max + KEDA.
#infrastructure#deployment
Practise MLOps & Data Quality
215 interview questions in this topic.
Related questions
- How do you meet latency budgets in production ML?
- How do you control ML serving cost?
- When is batch inference dramatically cheaper than online?
- How do you serve multiple models efficiently on shared infrastructure?
- Why make inference requests idempotent?
- How do you serve ML models across multiple regions?