How to trade model quality vs latency in production?
mediumAnswer
- (1) Distill large model into smaller.
- (2) Ensemble → single model.
- (3) Quantize (fp32 → int8) — usually <1% quality loss.
- (4) Prune (remove low-magnitude weights).
- (5) Two-stage: cheap first, expensive rerank on top-K.
- (6) Adaptive compute: run cheap model, escalate uncertain cases to expensive.
- (7) Cache results for repeated queries.
- (8) Speculative decoding for autoregressive.
- Measure Pareto frontier: for each latency budget, what's best quality?
- Choose based on business SLA.
Check yourself — multiple choice
- Random
- Distill + quantize + prune + two-stage + adaptive compute + cache + speculative decoding; measure Pareto frontier (quality vs latency); pick per business SLA
- Just faster
- Not real
Quality/latency: distill + quant + prune + two-stage + Pareto frontier.
#deployment
Practise MLOps & Data Quality
215 interview questions in this topic.