Why make inference requests idempotent?
mediumAnswer
- Same request produces same result (or same side effects).
- Enables: (1) safe retry on network / server errors.
- (2) deduplication of accidentally duplicated requests.
- (3) caching by request hash.
- Implementation: (1) client passes idempotency-key header.
- (2) server logs (key, result) → returns cached on repeat.
- (3) side effects (write to DB) also keyed.
- Standard for high-value predictions (payments, credit approvals).
- Stripe pattern.
Check yourself — multiple choice
- Random
- Same request → same result; safe retry + dedup + hash cache; impl: client idempotency-key + server (key→result) log + side-effect keyed; Stripe pattern for high-value predictions
- Not needed
- Just retry
Idempotent inference: idempotency-key + logged results; safe retry.
#deployment#infrastructure
Practise MLOps & Data Quality
215 interview questions in this topic.
Related questions
- How do you meet latency budgets in production ML?
- How do you control ML serving cost?
- When is batch inference dramatically cheaper than online?
- How do you serve multiple models efficiently on shared infrastructure?
- How do you configure autoscaling for ML serving?
- How do you serve ML models across multiple regions?