A product manager asks what a p-value of 0.03 means. What do you say?

easy

Answer

$Say that if the change truly had no effect, you would see a difference this large or larger about 3% of the time by chance alone.$
$Then say what it does not mean, because that is where decisions go wrong: it is not the probability that the change works, and it is not the probability that the null hypothesis is true.$
$Add the part that actually matters for the decision, which is the effect size and its confidence interval, since a significant result whose interval spans from trivially small to large does not justify a launch.$
$Frame the conclusion as a decision under uncertainty, weighing the cost of shipping a neutral change against the cost of missing a real one, rather than as a verdict delivered by the threshold.$

Check yourself — multiple choice

There is a 3% chance the change does not work
If the change truly had no effect, a difference this large or larger would occur about 3% of the time — it is not the probability the hypothesis is true, so report the effect size and interval
The change works 97% of the time
The result is 97% accurate

A p-value is computed assuming the null is true; it is not a probability about the hypothesis.

#hypothesis-testing

215 interview questions in this topic.