Your revenue-per-user metric is extremely right-skewed. Can you still use a t-test?

hard

Answer

$Usually yes, but for a reason worth stating precisely: the t-test requires the sampling distribution of the mean to be approximately normal, not the data, and the central limit theorem delivers that at large sample sizes even for skewed data.$
$The practical caveat is that convergence is slower the heavier the tail, so with a few hundred observations and extreme outliers the test can still be unreliable.$
$Better options are to cap or winsorize the metric at a high percentile, which reduces variance dramatically at the cost of a slightly different estimand, or to bootstrap the difference in means, which makes no distributional assumption.$
$Do not switch to a rank test without thinking, because the Mann-Whitney test answers a question about stochastic ordering rather than about mean revenue, and mean revenue is what the business cares about.$

Check yourself — multiple choice

No, the data must be normal
Yes, since normality is required of the sampling distribution of the mean, not the data — but with heavy tails prefer winsorizing or bootstrapping, and note that rank tests change the estimand
Always use Mann-Whitney instead
Skew makes testing impossible

The CLT covers the mean's distribution; heavy tails slow convergence, so winsorizing or bootstrapping is safer than switching estimands.

#parametric-tests#non-parametric#clt

215 interview questions in this topic.