Real interview: your model performs great in A/B but flops post-launch. Why?

hard

Answer

$Common reasons: (1) Selection bias — the A/B population is not the launch population (early adopters, engaged users).$
$(2) SUTVA violation — 50% traffic doesn't scale to 100% (marketplace saturation, ad auction dynamics).$
$(3) Novelty effect not de-biased.$
$(4) Winner's curse — effect regressed to a smaller true value.$
$(5) Metric divergence — A/B primary metric doesn't align with long-term OKR.$
$(6) Reflex reaction from competitors / operations.$
$Debug by: revisiting SUTVA, holdout at 1%, long-term surrogate, and re-measuring at launch.$

Check yourself — multiple choice

Random
Selection bias / SUTVA at 100% / novelty / winner's curse / metric-OKR misalignment / competitor reaction; debug with long holdout and re-measurement
Model was broken
Not real

A/B → launch gap: SUTVA / selection / novelty / winner's curse / metric drift.

#ab-testing#causal-inference

215 interview questions in this topic.