When is batch inference dramatically cheaper than online?

medium

Answer

$Batch: (1) high GPU utilization via full batches (100-1000).$
$(2) commodity hardware fine.$
$(3) preemptible / spot instances OK.$
$(4) no idle capacity.$
$Online: (1) low batch (often 1) \to low GPU utilization.$
$(2) always-warm capacity for spikes.$
$(3) latency-optimized (more expensive) hardware.$
$Ratio: batch can be 10-100x cheaper per prediction.$
$Use batch when predictions can be precomputed (nightly product recs, credit scores).$
$Push to online store or DB.$

Check yourself — multiple choice

Same
Batch: high GPU util (big batch) + commodity + spot; online: low batch + always-warm + latency HW; batch 10-100x cheaper; use batch when precomputable + push to online store
Random
Not real

Batch 10-100x cheaper; use when precomputable.

#deployment#infrastructure

215 interview questions in this topic.