You inherit a model in production with no documentation. What do you check in your first week?
mediumAnswer
- Establish what it does before touching anything.
- Find the inference path and confirm which artefact is actually being served, since the deployed version is often not the one in the repository.
- Check whether the training data can be reconstructed, because a model you cannot retrain is a liability regardless of its accuracy.
- Compare the current input distribution against whatever the model was trained on, which usually reveals drift nobody was watching.
- Verify that predictions are logged with their inputs and a model version, since without that you cannot debug anything.
- Find out how outcomes are eventually observed, as that determines whether you can measure real performance at all.
- Only then look at the model itself.
Check yourself — multiple choice
- Retrain immediately with a better algorithm
- Identify the artefact actually served, whether training data is reconstructible, current input drift, whether predictions are logged with versions, and how ground truth arrives
- Rewrite the training code first
- Add more features
Reproducibility, logging and the ground-truth path determine what is even possible before model work begins.
#mlops#monitoring
Practise MLOps & Data Quality
215 interview questions in this topic.