How do you control ML serving cost?
mediumAnswer
- (1) Right-size infrastructure: autoscale on utilization.
- (2) Batch requests where latency allows.
- (3) Distill to smaller model.
- (4) Cache popular predictions.
- (5) Two-stage: cheap filter + expensive rerank only on top-K.
- (6) Spot / preemptible instances for batch.
- (7) Compress model (quant / prune) to fit on cheaper HW.
- (8) Move batch scoring off critical path.
- (9) Serverless (Lambda / Cloud Run) for spiky traffic.
- (10) Multi-tenant serving: co-locate models on same GPU.
- Monitor $/prediction.
Check yourself — multiple choice
- Random
- Autoscale + batch + distill + cache + two-stage + spot instances + compress + move batch off critical + serverless spiky + multi-tenant GPU; monitor $/prediction
- Just scale up
- Not real
Cost: autoscale + batch + distill + cache + two-stage + spot + compress.
#deployment#infrastructure
Practise MLOps & Data Quality
215 interview questions in this topic.
Related questions
- How do you meet latency budgets in production ML?
- When is batch inference dramatically cheaper than online?
- How do you serve multiple models efficiently on shared infrastructure?
- Why make inference requests idempotent?
- How do you configure autoscaling for ML serving?
- How do you serve ML models across multiple regions?