What do you actually test in a CI pipeline for a model?
hardAnswer
- Split the tests by what they protect.
- Data tests validate schema, ranges, null rates and cardinality on the incoming data, and they should fail the pipeline, because training on broken data is worse than not training.
- Pipeline tests run the whole path on a tiny fixture to catch shape and type errors in seconds.
- Behavioural tests assert properties rather than accuracy: that a known-obvious example is classified correctly, that a monotonic relationship holds, that a perturbation which should not matter does not change the prediction.
- A performance gate compares the candidate against the current production model on a frozen evaluation set, with a tolerance rather than an exact number.
- And a serving test loads the artefact and scores a request, which catches the dependency mismatch that breaks deployments.
Check yourself — multiple choice
- Only unit tests on utility functions
- Data validation, a fast end-to-end pipeline run on a fixture, behavioural property tests, a performance gate against production on a frozen set, and an artefact-loading serving test
- Only final accuracy
- Models cannot be tested in CI
Data, pipeline, behaviour, performance gating and serving each catch a distinct class of failure.
#mlops#reproducibility
Practise MLOps & Data Quality
215 interview questions in this topic.