How do you use spot / preemptible instances safely for ML?

medium

Answer

$50-90% discount but can be reclaimed 2 min notice.$
$Safe for: (1) training (checkpoint frequently, resume).$
$(2) batch inference (retry).$
$(3) experiment sweeps.$
$Unsafe for online serving unless mixed with on-demand.$
$Patterns: (1) checkpoint every N steps \to resume on interruption.$
$(2) mixed instance groups: on-demand baseline + spot burst.$
$(3) spot fleet across zones + instance types \to diversity reduces simultaneous reclaim.$
$(4) SIGTERM handler for graceful shutdown.$
$Save 50-80% compute.$

Check yourself — multiple choice

Never
50-90% discount but 2-min reclaim; safe for training (checkpoint+resume) + batch + sweeps; unsafe online alone; patterns: checkpoint + mixed on-demand baseline + fleet diversity + SIGTERM graceful; save 50-80%
Random
Not real

Spot: checkpoint + mixed baseline + fleet diversity + SIGTERM; save 50-80%.

#infrastructure

215 interview questions in this topic.