Your model must answer in 50 milliseconds. How do you allocate the budget?
mediumAnswer
- Measure the whole path first, because inference is rarely the dominant cost.
- Feature retrieval usually is, especially if it involves several network calls, so count them and fetch in parallel or precompute.
- Reserve headroom for the network and serialization, and budget against a high percentile rather than the mean, since the ninety-ninth percentile is what users experience and what times out.
- Then choose the model to fit what remains: a gradient boosted tree with a few hundred shallow trees answers in single-digit milliseconds, while a transformer generally does not without batching, quantization or a GPU.
- Cache aggressively where inputs repeat, and design a degraded path, such as a cached or default prediction, for when the budget is exceeded rather than letting the request hang.
Check yourself — multiple choice
- Just use a smaller model
- Budget the whole path at a high percentile: feature retrieval calls, network and serialization first, then pick a model that fits the remainder, with caching and a degraded fallback path
- Optimize mean latency
- Latency is a hardware problem only
Feature retrieval and tail latency usually dominate, so the model choice comes after budgeting the path.
#serving#infrastructure
Practise MLOps & Data Quality
215 interview questions in this topic.