Why must ML pipelines be idempotent?
mediumAnswer
- Re-running same pipeline on same input must produce same output.
- Enables: (1) safe retries after failures (no double-writes).
- (2) backfills (recompute for historical windows).
- (3) reproducibility (same code + data = same result).
- Requires: (1) deterministic ops (fix seeds).
- (2) content-addressed outputs (hash inputs).
- (3) transactional writes (atomic replace, not append).
- (4) explicit in output partition.
- Anti-pattern: appending to output on rerun → duplicate rows.
Check yourself — multiple choice
- Random
- Same input → same output; enables safe retries + backfills + reproducibility; requires deterministic ops + content-addressed outputs + transactional writes + explicit
- Not needed
- Just log
Idempotent: safe retry / backfill / reproducibility; determinism + atomic writes.
#pipeline#reproducibility
Practise MLOps & Data Quality
215 interview questions in this topic.
Related questions
- How does Hydra help ML config management?
- What are common sources of data leakage in an ML pipeline?
- When and how do you retrain a production model?
- How does imputation cause data leakage?
- How do you handle high-cardinality categorical features?
- How does time-based leakage happen and how do you prevent it?