33 interview questions on hypothesis testing, each answered in full. Free to read, no account needed.
What is a p-value, precisely? easy The probability, assuming the null hypothesis is true, of observing a test statistic at least as extreme as the one obtained. It is NOT the probability that the null is true, nor the probability of a mistake. A small p-value means the data are unlikely under H0. Combine with effect size and confidence intervals — never rely on p-values alone.
Type I vs Type II error — what's the difference? easy Type I error (false positive): rejecting H0 when it's actually true. Its probability is alpha (significance level, often 0.05). Type II error (false negative): failing to reject H0 when H1 is true. Its probability is beta; p o w e r = 1 − β \mathrm{power}\; = \;1\; - \;\beta power = 1 − β There's a tradeoff — lowering alpha raises beta unless you increase sample size or effect size.
Why is multiple testing a problem and how do you correct for it? medium Running many independent tests at α = 0.05 \alpha\; = \;0.05 α = 0.05 Corrections: Bonferroni (divide alpha by number of tests — very conservative), Holm-Bonferroni (stepwise, better power), or control the False Discovery Rate with Benjamini-Hochberg (best for discovery-style work with many tests).
How do you formulate null and alternative hypotheses? easy Null H0: the 'no effect' / 'no difference' hypothesis ( μ A = μ B , β = 0 , t r e a t m e n t h a s n o i m p a c t ) ({\mu}_{A}\; = \;{\mu}_{B}, \;{\beta}\; = \;0, \;\mathrm{treatment}\;\mathrm{has}\;\mathrm{no}\;\mathrm{impact}) ( μ A = μ B , β = 0 , treatment has no impact ) Alternative H1: what we suspect ( μ A ≠ μ B t w o − s i d e d , o r μ A > μ B o n e − s i d e d ) ({\mu}_{A}\; \ne \;{\mu}_{B}\;\mathrm{two} - \mathrm{sided}, \;\mathrm{or}\;{\mu}_{A}\; > \;{\mu}_{B}\;\mathrm{one} - \mathrm{sided}) ( μ A = μ B two − sided , or μ A > μ B one − sided ) One-sided tests are more powerful but only appropriate when you commit ex ante to the direction. Rule: define H0 / H1 before seeing the data. Post-hoc directional testing inflates type I error.
One-sample t-test: setup and assumptions. easy H0: μ = μ 0 {\mu}\; = \;{\mu}_{0} μ = μ 0 Statistic: t = ( X − μ 0 ) / ( s / n ) t\; = \;(X\; - \;{\mu}_{0})\; / \;(s\; / \; \sqrt n) t = ( X − μ 0 ) / ( s / n ) Under H0 and normality, t ~ t n − 1 t_{n - 1} t n − 1 Assumptions: (1) i.i.d. samples; (2) approximately normal (robust for n > 30 due to CLT). Compare t to t-distribution quantiles or compute a p-value. Use case: is the mean of a metric different from a target?
Two-sample t-test: pooled vs Welch — when to use each? medium Compare two group means. Pooled (Student's): assumes equal variances → uses pooled SD, d f = n 1 + n 2 − 2 \mathrm{df}\; = \;n_{1}\; + \;n_{2}\; - \;2 df = n 1 + n 2 − 2 Welch's: doesn't assume equal variances → uses Satterthwaite df correction. In practice, Welch is the safer default (behavior similar to Student when variances are equal, better when they aren't). Both assume i.i.d. within groups and approximate normality (CLT-robust for larger n).
When do you use a paired t-test? easy Same subjects measured under two conditions (before / after treatment), or naturally paired observations (twins, matched pairs). Compute differences d i = X A i − X B i d_{i}\; = \;X_{A}i\; - \;X_{B}i d i = X A i − X B i Much more powerful than an unpaired test when the within-pair correlation is high — cancels out subject-level variation. Assumption: differences approximately normal (CLT-robust for n > 30).
z-test vs t-test — when to pick each? easy z-test: known population variance (rare in practice), or very large n (>~100) where t-distribution ≈ normal. t-test: unknown population variance (usual case). For proportions with large n: z-test for a proportion or two-proportion z-test. The 'z-test for a proportion' is common in A/B testing. Modern practice: default to t-test unless you specifically know σ.
Chi-square test of independence — what does it do? medium Tests whether two categorical variables are independent, given an r × c contingency table. Statistic: χ 2 = Σ {\chi}^{2}\; = \;{\Sigma} χ 2 = Σ ( O i j − E i j ) 2 / E i j (O_{\mathrm{ij}}\; - \;E_{\mathrm{ij}})^{2}\; / \;E_{\mathrm{ij}} ( O ij − E ij ) 2 / E ij E i j = r o w i ⋅ c o l j / N E_{\mathrm{ij}}\; = \;\mathrm{row}_{i}\; \cdot \;\mathrm{col}_{j}\; / \;N E ij = row i ⋅ col j / N Requires E i j E_{\mathrm{ij}} E ij Use case: is conversion rate independent of region? For small counts, use Fisher's exact test instead.
Chi-square goodness-of-fit test — setup. medium Tests whether observed frequencies match an expected distribution: χ 2 = Σ {\chi}^{2}\; = \;{\Sigma} χ 2 = Σ ( O i − E i ) 2 / E i (O_{i}\; - \;E_{i})^{2}\; / \;E_{i} ( O i − E i ) 2 / E i Uses: is a die fair? Does a dataset follow Poisson? Requires E i E_{i} E i Modern alternatives: G-test (likelihood ratio), Kolmogorov-Smirnov (continuous), Anderson-Darling (heavier weight on tails).
What does one-way ANOVA test? medium H0: μ 1 = μ 2 {\mu}_{1}\; = \;{\mu}_{2} μ 1 = μ 2 μ k {\mu}_{k} μ k Statistic: F = between-group variance / within-group variance ~ F k − 1 , N − k F_{k - 1, \;N - k} F k − 1 , N − k Assumptions: (1) i.i.d. within groups; (2) normally distributed; (3) equal variances (Welch's ANOVA relaxes this). If F is significant, follow-up with post-hoc tests (Tukey HSD, Bonferroni) to identify which groups differ. Non-parametric alternative: Kruskal-Wallis.
Two-way ANOVA and what interactions mean. hard Tests three effects at once: main effect of factor A, main effect of factor B, and A×B interaction. Interaction: the effect of A depends on the level of B. Significant interaction means main effects are hard to interpret in isolation (plot cell means to visualize). Assumptions: normality + equal variances + independence within cells. Foundation of factorial experimental design.
Mann-Whitney U (Wilcoxon rank-sum) — when and why? medium Non-parametric test comparing two independent groups. Combines all data, ranks it, compares rank sums between groups. Doesn't assume normality — good for skewed, ordinal, or small-sample data. Effectively tests whether one distribution is shifted vs the other. Less powerful than t-test when normality holds; more robust when it doesn't. Not exactly a 'test of medians' — it's a stochastic-dominance test.
Wilcoxon signed-rank test — setup. medium Non-parametric alternative to the paired t-test. Compute differences d i d_{i} d i rank ∣ d i \operatorname{rank}\; \mid d_{i} rank ∣ d i Null: distribution of differences is symmetric around zero. Robust to non-normal differences, sensitive to outliers still (unlike sign test). Uses ranks so drops information but keeps power when normality is doubtful.
Kruskal-Wallis — non-parametric ANOVA analog. medium Extends Mann-Whitney to k > 2 groups. Rank all data across groups, compute H = 12/(N(N+1)) * Σ ( R i 2 / n i ) − 3 ( N + 1 ) (R_{i}^{2}\; / \;n_{i})\; - \;3(N + 1) ( R i 2 / n i ) − 3 ( N + 1 ) χ 2 k − 1 {\chi}^{2}k - 1 χ 2 k − 1 Null: all groups drawn from same distribution. If significant, follow up with pairwise Mann-Whitney with Bonferroni. Use for skewed / ordinal / non-normal data with 3+ groups.
Friedman test and its use case. hard Non-parametric alternative to repeated-measures ANOVA. Same subjects rated on k treatments; rank within each subject, sum ranks per treatment. Null: no treatment effect. Useful for within-subject designs where normality is violated. Follow-up: pairwise Wilcoxon with correction. Common in ML benchmarking (rank algorithms across datasets).
Kolmogorov-Smirnov test — one- and two-sample. medium Statistic: D = max x ∣ F 1 ( x ) − F 2 ( x ) D\; = \;\operatorname{max}_{x}\; \mid F_{1}(x)\; - \;F_{2}(x) D = max x ∣ F 1 ( x ) − F 2 ( x ) Distribution-free, works for continuous data, sensitive to shifts in location, scale, or shape. Weakness: less sensitive in the tails. Use in drift detection (compare distribution today vs baseline).
Shapiro-Wilk normality test — when useful? medium Tests whether data comes from a normal distribution. Very sensitive for n < 50; over-powered for large n (reports significant non-normality for trivial deviations). Practical rule: for large samples, rely more on Q-Q plots than on the p-value. Alternatives: Anderson-Darling (weighted toward tails), Kolmogorov-Smirnov (less sensitive), Lilliefors (KS variant with estimated params).
How do you test equality of variances? medium Bartlett: assumes normality; sensitive to non-normality. Levene: uses absolute deviations from mean or median (Brown-Forsythe uses median); more robust. F-test of variances: strictly two-sample under normality. Use Levene (with median) as the safe default. Purpose: check equal-variance assumption for ANOVA / pooled t-test; if violated, use Welch's variants.
Why report effect size alongside p-values? easy P-values conflate signal size with sample size — huge n makes trivial effects significant. Effect size quantifies practical importance regardless of n. Common measures: Cohen's d (standardized mean difference: 0.2 small, 0.5 medium, 0.8 large); odds ratio / relative risk (binary outcomes); Pearson r / R 2 r\; / \;R^{2} r / R 2 Always report effect size with CIs and p-values.
What is power analysis and how do you use it? medium P o w e r = P ( r e j e c t H 0 ∣ H 1 t r u e ) = 1 − β \mathrm{Power}\; = \;P(\mathrm{reject}\;\mathrm{H0}\; \mid \;\mathrm{H1}\;\mathrm{true})\; = \;1\; - \;{\beta} Power = P ( reject H0 ∣ H1 true ) = 1 − β Power analysis: given effect size, α, and power (typically 0.8), solve for required sample size. Or: given n, α, effect size, solve for achieved power. Uses: (1) plan A/B tests (how many users do we need?); (2) diagnose 'no significant effect' — was the study under-powered? Free tools: G*Power, statsmodels.power. Always do this before running an experiment.
What is MDE (minimum detectable effect)? medium Smallest effect size the experiment can detect with a target power (usually 0.8) given the sample size and α. Computed via power formula. Practical use: 'with N=10000 users per arm at α=0.05 and power 0.8, the MDE is a 2% lift'. If your product intuition says the effect might be 1%, you need to grow n. Cornerstone of A/B testing capacity planning.
Bonferroni correction — how does it work? medium For k independent tests at family-wise error rate α: use α a d j u s t e d = α / k {\alpha}_{\mathrm{adjusted}}\; = \;{\alpha}\; / \;k α adjusted = α / k Guarantees P(any false rejection) ≤ α under independence, ≤ α always (union bound). Very conservative — loses power quickly as k grows. Use for a small number of confirmatory tests (< 20). Holm's step-down variant is uniformly more powerful with the same guarantee.
How does the Benjamini-Hochberg procedure work? hard Controls the FDR (expected fraction of false positives among rejections) at level q. Steps: (1) Sort p-values p_(1) ≤ ... ≤ p_(m). (2) Find largest k such that p_(k) ≤ (k/m) * q. (3) Reject all H_(i) with i ≤ k. Much more powerful than Bonferroni for large m — accepts a small fraction of false positives to detect many true effects. Standard in genomics, feature selection, high-dim testing.
What is a permutation test? medium Non-parametric test based on the exchangeability principle: shuffle group labels, recompute the test statistic, build a null distribution empirically. Compute p-value = fraction of permutations with statistic ≥ observed. Works for any statistic ( m e a n , m e d i a n , d i f f e r e n c e i n R 2 ) (\mathrm{mean}, \;\mathrm{median}, \;\mathrm{difference}\;\mathrm{in}\;R^{2}) ( mean , median , difference in R 2 ) Distribution-free, only assumes exchangeability under H0. Slow (usually 10k+ shuffles) but very general. Modern default when you want strong assumption independence.
What does McNemar's test do? hard Paired categorical (usually binary) test. Compares proportions in matched pairs. From a 2×2 table of concordant / discordant pairs, χ 2 = ( b − c ) 2 / ( b + c ) {\chi}^{2}\; = \;(b\; - \;c)^{2}\; / \;(b\; + \;c) χ 2 = ( b − c ) 2 / ( b + c ) Uses: classifier comparison on the same test set (does model A get more test items right than model B?), before / after intervention on the same subjects.
When to use Fisher's exact test? medium 2×2 (or r×c) contingency table with small expected counts (E < 5) where chi-square approximation fails. Computes an exact p-value from the hypergeometric distribution. Slow for large tables. Standard for small clinical trials, rare-event categorical data. Modern alternative for larger tables: Monte Carlo χ 2 {\chi}^{2} χ 2
What is p-hacking and how do you prevent it? medium Running many analyses (test variants, subgroups, transformations) until you find p < 0.05, then reporting only the significant results. Inflates false-positive rate dramatically. Prevention: (1) pre-register hypotheses + analysis plan; (2) Bonferroni / FDR on all tested variants; (3) separate exploration (open) from confirmation (locked test set + pre-registered stat plan); (4) report all comparisons, not just significant ones. Foundation of the replication crisis.
FWER vs FDR — which do you control when? hard FWER (Family-Wise Error Rate): P(any false rejection). Controlled by Bonferroni / Holm — safe when even a single false positive is costly (safety, regulatory). FDR (False Discovery Rate): E[false / total rejections]. Controlled by Benjamini-Hochberg — accepts some false positives to keep power in discovery / exploratory settings. Rule: FWER for confirmatory / high-stakes; FDR for exploratory / genomics / feature selection.
How is a CI related to a hypothesis test? medium A 95% CI for a parameter includes all null values that would NOT be rejected at α = 0.05 in a two-sided test. Equivalently: if 0 is outside the 95% CI of the difference, then reject H0: no difference at α = 0.05. CIs are more informative — they give you the range of plausible values, not just accept / reject. Always report CIs.
Why is peeking at running A/B tests dangerous? hard Peeking = repeatedly running a test and stopping when p < 0.05 → hugely inflates false-positive rate (up to ~40% for one-sided at α=0.05). Fixes: (1) commit to sample size ex ante and don't peek. (2) Sequential testing: mSPRT, α-spending (Pocock, O'Brien-Fleming), Bayesian sequential decisions. (3) Always-valid inference (Howard & Ramdas): give up 10-20% efficiency for the freedom to check any time. Optimizely / Statsig / Eppo implement always-valid or sequential correctly.
What is alpha-spending in sequential testing? hard Divide the total α (0.05) across scheduled interim looks so cumulative false-positive rate stays capped. Pocock: uniform per look (conservative). O'Brien-Fleming: tiny α early, most saved for the final look (encourages waiting). Haybittle-Peto: small α at interims, α - 0.001 at final. All maintain family-wise error at 0.05 across multiple looks — enables ethical stopping in clinical trials.
A product manager asks what a p-value of 0.03 means. What do you say? easy Say that if the change truly had no effect, you would see a difference this large or larger about 3% of the time by chance alone. Then say what it does not mean, because that is where decisions go wrong: it is not the probability that the change works, and it is not the probability that the null hypothesis is true. Add the part that actually matters for the decision, which is the effect size and its confidence interval, since a significant result whose interval spans from trivially small to large does not justify a launch. Frame the conclusion as a decision under uncertainty, weighing the cost of shipping a neutral change against the cost of missing a real one, rather than as a verdict delivered by the threshold.