Shadow deployment, canary, or A/B test — which do you use when?
mediumAnswer
- They answer different questions, so the sequence matters more than the choice.
- Shadow mode sends real traffic to the new model without using its output, which validates that it runs, that latency is acceptable and that predictions are sane, with zero user risk but no information about business impact.
- Canary sends a small fraction of real traffic to the new model to catch operational failures under genuine load and to limit blast radius, and it is about safety rather than measurement.
- An A/B test with proper randomization and enough exposure is the only one that measures whether the new model is actually better on the metric you care about.
- The mature pattern is shadow, then canary, then A/B, and only then a full rollout.
Check yourself — multiple choice
- They are three names for the same thing
- Shadow validates that it runs safely with no user impact, canary limits blast radius under real load, and only a randomized A/B test measures whether it is better — use them in that order
- Only A/B testing is needed
- Canary measures business impact
Shadow checks operability, canary limits risk, A/B measures impact; they are sequential, not alternatives.
#deployment#experimentation
Practise MLOps & Data Quality
215 interview questions in this topic.