What do you need to know to compute a sample size for an experiment?

medium

Answer

$Four things, and the hard one is not statistical.$
$The baseline rate of the metric, its variance, which for a proportion follows from the baseline but for a revenue metric must be estimated and is usually much larger than people expect.$
$The minimum detectable effect, meaning the smallest improvement that would actually change a decision, which is a business judgement and the input most often chosen to make the number look acceptable.$
$Then the significance level and power, conventionally 5% and 80%, where power is the probability of detecting the effect if it is real.$
$Sample size scales inversely with the square of the effect size, so halving the detectable effect quadruples the sample, which is why heavy-tailed revenue metrics often cannot be tested directly and are replaced by a capped or binary proxy.$

Check yourself — multiple choice

Only the desired p-value
Baseline rate, metric variance, minimum detectable effect, and significance plus power — with sample size scaling as the inverse square of the effect
Only the traffic volume
Sample size cannot be planned

Baseline, variance, MDE, alpha and power determine sample size, which grows as the inverse square of the effect.

#power#design

215 interview questions in this topic.