What are the pillars of a solid A/B test design?
mediumAnswer
- (1) Clear primary metric + guardrail metrics chosen in advance.
- (2) Sample size / MDE via power analysis.
- (3) Randomization unit chosen at correct level (user, session, cluster).
- (4) Sample ratio check (SRC): assignment counts should match target ratio.
- (5) A/A test as sanity check on randomizer.
- (6) Pre-registered analysis plan.
- (7) Fixed test duration + no-peeking rule (or sequential test).
- (8) Multiple-testing correction if many metrics.
Check yourself — multiple choice
- Just click
- Primary + guardrail metrics, power/MDE, right randomization unit, SRC + A/A, pre-registered plan, no-peek or sequential, multiple-testing correction
- Random
- Same as EDA
A/B design: primary+guardrail, power, right unit, SRC/A-A, no-peek, correction.
#ab-testing#design
Practise Statistics Fundamentals
215 interview questions in this topic.