How do you A/B test a model rigorously?
hardAnswer
- Randomly assign users to control (old model) and treatment (new model).
- Define the primary business metric before starting.
- Compute sample size needed to detect the expected effect with acceptable power.
- Run for a full business cycle (7+ days).
- Analyze with proper significance tests, account for multiple comparisons, and check guardrail metrics (latency, revenue, complaints).
- Stop early only via pre-specified sequential rules.
Check yourself — multiple choice
- You should decide the primary metric after the test finishes
- Randomization, pre-specified metrics, and adequate power are essential
- Peeking at results and stopping early is fine
- Guardrail metrics are optional
Randomize, pre-register the metric, size for power, watch guardrails.
#experimentation
Practise MLOps & Data Quality
215 interview questions in this topic.