Your experiment tracks 20 metrics and one is significant. What now?

medium

Answer

$Treat it as expected noise until proven otherwise, because with 20 independent tests at the 5% level you expect about one false positive even when nothing changed.$
$The structural fix is to declare a single primary metric before the experiment, with the rest labelled as secondary or guardrail, which turns a fishing expedition into a hypothesis test.$
$If several metrics genuinely need decision authority, control the error rate explicitly: Bonferroni is conservative and fine for a handful, and the Benjamini-Hochberg procedure controls the false discovery rate and is the better choice when you have many.$
$For the metric that fired, ask whether it moved in a direction consistent with a plausible mechanism and with the other metrics, and if it is isolated and surprising, the correct action is to replicate rather than to launch.$

Check yourself — multiple choice

Launch, one significant result is enough
Expect about one false positive from 20 tests: declare a primary metric in advance, control error with Bonferroni or Benjamini-Hochberg, and replicate an isolated surprising result
Test even more metrics
Report only the significant one

Twenty tests at 5% produce roughly one spurious hit; pre-registration and error control are the answer.

#multiple-testing#ab-testing

215 interview questions in this topic.