Batch vs online inference — how do you choose?
easyAnswer
- Batch inference precomputes predictions on a schedule (nightly scoring) — cheap, high-throughput, tolerates seconds/minutes of latency.
- Online inference serves predictions in real time via HTTP/gRPC — needed for user-facing systems (recommendations, fraud checks).
- Choose based on latency budget and freshness need.
- Hybrid: online model with cached batch features from a feature store.
Check yourself — multiple choice
- Online inference is always cheaper than batch
- Batch is fine when latency budget is minutes; online for real-time decisions
- Batch inference requires GPUs
- You cannot mix batch and online
Latency and freshness needs drive the choice.
#deployment
Practise MLOps & Data Quality
215 interview questions in this topic.