How does Simpson's paradox strike A/B tests?

hard

Answer

$Test wins overall but loses in every user segment (or vice versa) \to aggregate confounded by mid-experiment composition change.$
$Common causes: (1) SRM by segment (imbalanced assignment across segments over time), (2) traffic mix drift while experiment runs, (3) new-vs-returning composition shift.$
$Fix: (1) fix SRM; (2) analyze weighted by pre-experiment segment shares (post-stratification); (3) run segment-level analyses; (4) if segment-level results all point the same way but opposite to aggregate \to trust the segments.$

Check yourself — multiple choice

Never happens
Aggregate result flips within-segment result due to composition change / SRM by segment / mix drift → post-stratify or trust segment results
Random
Same as p-hacking

Simpson in A/B: composition shift flips aggregate; post-stratify.

#ab-testing#causal-inference

215 interview questions in this topic.