A model caused a costly wrong decision. What does a useful post-mortem produce?
mediumAnswer
- A timeline anchored in evidence, which is only possible if predictions, inputs and versions were logged, and a first finding of a missing log is itself the most valuable output.
- Then the actual causal chain, which for model incidents is usually upstream: a schema change, a feature pipeline default, or drift nobody was alerted on, rather than the model being wrong in an interesting way.
- Identify the detection gap, meaning how long the problem ran before anyone noticed, since reducing time to detection is generally worth more than preventing that specific cause.
- Produce concrete actions with owners, weighted towards guardrails and validation rather than promises of vigilance.
- And keep it blameless, because engineers who fear the process stop reporting the near misses that are your cheapest information.
Check yourself — multiple choice
- An explanation of which engineer erred
- An evidence-based timeline, the real upstream causal chain, the detection gap and how to shorten it, owned actions favouring guardrails, and a blameless tone that keeps near misses reported
- A decision to change algorithms
- A commitment to be more careful
Timeline, upstream cause, detection gap and owned guardrail actions are what make a post-mortem useful.
#mlops#observability
Practise MLOps & Data Quality
215 interview questions in this topic.