How do you meet latency budgets in production ML?
hardAnswer
- Budget = p50 or p99 target (e.g., 50ms).
- Techniques: (1) Model compression: quantization (int8/int4), pruning, distillation.
- (2) ONNX / TensorRT / Triton for optimized runtime.
- (3) Batching: dynamic batching increases throughput at slight latency cost.
- (4) Caching: memoize predictions for repeated inputs.
- (5) Two-stage: fast filter + slow rerank.
- (6) Async / precompute in shadow.
- (7) Hardware: GPU / TPU for large, CPU for small.
- Profile end-to-end; often preprocessing / network dominates, not model.
Check yourself — multiple choice
- Random
- Compression (quant/prune/distill) + optimized runtime (ONNX/TRT/Triton) + batching + cache + two-stage + async + right hardware; profile end-to-end (network/preprocess often dominates)
- Just GPU
- Not real
Latency: compress + runtime + batch + cache + two-stage + async + hardware.
#deployment#infrastructure
Practise MLOps & Data Quality
215 interview questions in this topic.
Related questions
- How do you control ML serving cost?
- When is batch inference dramatically cheaper than online?
- How do you serve multiple models efficiently on shared infrastructure?
- Why make inference requests idempotent?
- How do you configure autoscaling for ML serving?
- How do you serve ML models across multiple regions?