EasyDeepLearn
All categoriesStudy
Lesson

Statistics Fundamentals

215 concepts~323 min read15 sections

Probability, hypothesis testing, confidence intervals, and the stats DS interviewers ask about.

Interview flashcardsTake the quiz
Filter by difficulty
Filter by concept tag

Introduction

Statistics is the foundation that keeps ML honest. Data science interviews probe it aggressively because a good model built on a shaky inference is worse than no model at all.

The essentials are: understanding what a p-value actually means (not what most people think), knowing when to bootstrap vs rely on a closed-form CI, reasoning about correlation vs causation, and being able to spot the classic gotchas — Simpson's paradox, multiple testing, sampling bias. This chapter goes through them one by one, interview-style.

01

Probability foundations

41 concepts

State the Central Limit Theorem in one sentence.

easy
  • For independent, identically distributed samples with finite mean mu and variance σ2\sigma^{2}, the sampling distribution of the mean approaches a normal distribution N(μ,  σ2  /  n)N(\mu, \;\sigma^{2}\; / \;n) as n grows, regardless of the underlying distribution.
Permalink

Bayesian vs frequentist — what's the core difference?

medium
  • Frequentists treat parameters as fixed unknowns and estimate them via long-run frequency properties (unbiased estimators, confidence intervals, p-values).
  • Bayesians treat parameters as random variables with a prior distribution, update with data via Bayes' rule to get a posterior, and summarize with credible intervals.
  • Bayesian methods handle prior knowledge naturally and give posterior probability statements directly.
Permalink

State the Law of Large Numbers.

easy
  • As the sample size grows, the sample mean converges to the true expected value.
  • Weak LLN: convergence in probability.
  • Strong LLN: convergence almost surely.
  • This justifies estimating expectations by averaging samples — the foundation of Monte Carlo estimation.
Permalink

Independence vs uncorrelatedness — what's the difference?

medium
  • Independent means P(A, B) = P(A) P(B): no relationship at all.
  • Uncorrelated means covariance = 0: no linear relationship.
  • Independence implies uncorrelated, but the reverse is false in general — X and X2X^{2} can have zero correlation while being fully dependent.
  • For jointly Gaussian variables, uncorrelated implies independent — a special case.
Permalink

State Bayes' theorem and one intuitive use.

easy
  • P(A  ∣  B)  =  P(B  ∣  A)P(A\; \mid \;B)\; = \;P(B\; \mid \;A) P(A) / P(B).
  • Intuition: update a prior belief P(A) with new evidence B to get a posterior belief P(A  ∣  B)P(A\; \mid \;B).
  • Classic use: medical testing with base rates.
  • Even a highly sensitive/specific test on a rare disease produces mostly false positives if the disease prevalence is low.
Permalink

State Kolmogorov's three probability axioms.

easy
  • For a sample space Ω and events A: (1) P(A) ≥ 0 (non-negativity).
  • (2) P(Ω) = 1 (normalization — some outcome must occur).
  • (3) For disjoint (mutually exclusive) events A1A_{1}, A2A_{2}, ...: P(  Ai)  =  ΣP(\;A_{i})\; = \;{\Sigma} P(Ai)P(A_{i}) (countable additivity).
  • All of probability theory — conditional probability, Bayes, expectation, independence — is derived from these three axioms.
Permalink

Define conditional probability and prove Bayes' rule from it.

easy
  • P(A  ∣  B)  =  P(A    B)  /  P(B)P(A\; \mid \;B)\; = \;P(A\;\;B)\; / \;P(B) when P(B) > 0.
  • Symmetrically, P(B  ∣  A)  =  P(A    B)  /  P(A)P(B\; \mid \;A)\; = \;P(A\;\;B)\; / \;P(A) → P(A    B)  =  P(B∣A)  ⋅  P(A)P(A\;\;B)\; = \;P(B \mid A)\; \cdot \;P(A).
  • Substitute back: P(A  ∣  B)  =  P(B  ∣  A)  ⋅  P(A)  /  P(B)P(A\; \mid \;B)\; = \;P(B\; \mid \;A)\; \cdot \;P(A)\; / \;P(B).
  • This is Bayes' rule — a direct consequence of the conditional-probability definition, not a separate axiom.
Permalink

State the law of total probability.

medium
  • If {B1B_{1}, ..., BnB_{n}} partition the sample space (mutually exclusive, jointly exhaustive), then P(A) = Σ P(A  ∣  Bi)  ⋅  P(Bi)P(A\; \mid \;B_{i})\; \cdot \;P(B_{i}).
  • Used constantly: decompose a complex probability by conditioning on a partitioning variable (age, disease status, region).
  • Foundation of many probabilistic derivations, mixture models, and marginalization in Bayesian inference.
Permalink

Bernoulli distribution: parameters, PMF, mean, variance.

easy
  • X ~ Bernoulli(p): single trial with success prob p.
  • PMF: P(X=1) = p, P(X=0) = 1-p.
  • E[X] = p; Var⁡(X)  =  p(1−p)\operatorname{Var}(X)\; = \;p(1 - p) — maximum at p=0.5.
  • Foundation for binary outcomes (click / no click, pass / fail).
  • Building block: Binomial(n, p) = sum of n i.i.d.
  • Bernoulli(p).
Permalink

Binomial distribution and when to use it.

easy
  • X ~ Binomial(n, p): number of successes in n independent Bernoulli(p) trials.
  • PMF: P(X=k)  =  C(n,k)  ⋅  pk  ⋅  (1−p)P(X = k)\; = \;C(n, k)\; \cdot \;p^{k}\; \cdot \;(1 - p)^(n-k).
  • E[X] = np; Var⁡(X)  =  np(1−p)\operatorname{Var}(X)\; = \;\mathrm{np}(1 - p).
  • Use it for: A/B testing (successes out of n visitors), quality inspection (defects out of n items), any bounded count of independent trials.
  • When n is large and p is small, approximate by Poisson(np).
Permalink

Poisson distribution and its typical use cases.

medium
  • X ~ Poisson(λ): count of events in a fixed interval / area, when events happen at constant rate λ independently.
  • PMF: P(X=k)  =  λk  ⋅  eP(X = k)\; = \;{\lambda}^{k}\; \cdot \;e^(-λ) / k!.
  • E[X] = Var(X) = λ.
  • Use cases: website clicks per minute, defects per page, insurance claims per year.
  • Limit of Binomial(n, p) as n → ∞, p → 0, np → λ.
  • Assumes memorylessness — real data with time-varying rate (bursty traffic) needs negative binomial or NHPP.
Permalink

Geometric distribution: setup and mean.

medium
  • X ~ Geometric(p): number of trials until the first success in i.i.d.
  • Bernoulli(p).
  • PMF: P(X=k) = (1-p)^(k-1) * p.
  • E[X] = 1/p; Var⁡(X)  =  (1−p)/p2\operatorname{Var}(X)\; = \;(1 - p) / p^{2}.
  • Memoryless: P(X  >  n+k  ∣  X  >  n)  =  P(X  >  k)P(X\; > \;n + k\; \mid \;X\; > \;n)\; = \;P(X\; > \;k).
  • Uses: number of attempts until conversion, retries until failure.
  • Note: two conventions — some define it as failures before first success (E = (1-p)/p).
Permalink

Uniform distribution: continuous vs discrete.

easy
  • Discrete: U{a, ..., b} — each of (b-a+1) integers has probability 1/(b-a+1).
  • Continuous: U(a, b) — density 1/(b-a) on [a,b], zero elsewhere.
  • Continuous uniform: E[X] = (a+b)/2, Var⁡(X)  =  (b−a)2/12\operatorname{Var}(X)\; = \;(b - a)^{2} / 12.
  • Uses: random sampling, base for inverse transform sampling (generate  any  distribution  by  feeding  uniform  samples  through  F−1)(\mathrm{generate}\;\mathrm{any}\;\mathrm{distribution}\;\mathrm{by}\;\mathrm{feeding}\;\mathrm{uniform}\;\mathrm{samples}\;\mathrm{through}\;F^{-1}), and shuffling.
Permalink

Exponential distribution: setup, memorylessness, use cases.

medium
  • X ~ Exp(λ): time between events in a Poisson process.
  • PDF: f(x) = λ * e^(-λx) for x ≥ 0.
  • E[X] = 1/λ; Var⁡(X)  =  1/λ2\operatorname{Var}(X)\; = \;1 / {\lambda}^{2}.
  • Memoryless: P(X  >  s+t  ∣  X  >  s)  =  P(X  >  t)P(X\; > \;s + t\; \mid \;X\; > \;s)\; = \;P(X\; > \;t) — the only continuous distribution with this property.
  • Uses: waiting times (until next call, next failure), radioactive decay, survival analysis (constant hazard rate).
Permalink

Normal distribution: PDF and key properties.

easy
  • X ~ N(μ,  σ2)N({\mu}, \;{\sigma}^{2}): f(x)  =  (1  /  sqrt(2πσ2))  ⋅  exp⁡(−(x−μ)2  /  (2σ2))f(x)\; = \;(1\; / \;\mathrm{sqrt}(2{\pi}{\sigma}^{2}))\; \cdot \;\operatorname{exp}( - (x - {\mu})^{2}\; / \;(2{\sigma}^{2})).
  • Bell-shaped, symmetric around μ. ~68% of mass within 1σ, ~95% within 2σ, ~99.7% within 3σ.
  • Sum of independent normals is normal.
  • Standardized: Z = (X-μ)/σ ~ N(0,1).
  • Universally used because of CLT — many sample statistics are approximately normal for large n.
Permalink

Multivariate normal: parameters and key properties.

medium
  • X ~ N(μ, Σ), where μ is a length-d mean vector and Σ is a d×d covariance matrix (symmetric positive semi-definite).
  • PDF: |2πΣ|^(−1/2)  ⋅  exp⁡(−0.5  ⋅  (x−μ)  Σ−1  (x−μ))( - 1 / 2)\; \cdot \;\operatorname{exp}( - 0.5\; \cdot \;(x - {\mu})\;{\Sigma}^{-1}\;(x - {\mu})).
  • Marginals and conditionals of a multivariate normal are also normal.
  • Linear combinations remain normal.
  • Uncorrelated components (Σ diagonal) → independent.
  • Foundation of Gaussian processes, Kalman filters, LDA, and joint modeling.
Permalink

What is a covariance matrix and its key properties?

medium
  • For random vector X ∈ RdR^{d}: Cov⁡(X)  =  E[(X−μ)(X−μ)]\operatorname{Cov}(X)\; = \;E[(X - {\mu})(X - {\mu})], a d×d matrix.
  • Diagonal: variances Var⁡(Xi)\operatorname{Var}(X_{i}).
  • Off-diagonal: covariances Cov⁡(Xi,  Xj)\operatorname{Cov}(X_{i}, \;X_{j}).
  • Properties: (1) symmetric, (2) positive semi-definite, (3) real eigenvalues ≥ 0.
  • Standardization: correlation matrix  =  D−1\mathrm{matrix}\; = \;D^{-1} Σ D−1D^{-1} where D is a diagonal matrix of SDs.
  • PCA is eigendecomposition of Σ; Mahalanobis distance uses Σ−1{\Sigma}^{-1}.
Permalink

What is linearity of expectation?

easy
  • For any random variables X, Y and constants a, b: E[aX + bY] = a*E[X] + b*E[Y].
  • Holds regardless of dependence between X and Y.
  • Very useful because it lets you decompose expectations of complex sums into pieces even when the pieces are dependent.
  • Example: expected number of collisions in a hash table with n keys and m slots is (n choose 2)/m — via linearity over the C(n,2) pairs.
Permalink

Variance of a sum: Var⁡(X  +  Y)\operatorname{Var}(X\; + \;Y) = ?

medium
  • Var⁡(X  +  Y)  =  Var⁡(X)  +  Var⁡(Y)  +  2  ⋅  Cov⁡(X,  Y)\operatorname{Var}(X\; + \;Y)\; = \;\operatorname{Var}(X)\; + \;\operatorname{Var}(Y)\; + \;2\; \cdot \;\operatorname{Cov}(X, \;Y).
  • Only when X and Y are uncorrelated (Cov = 0) does the covariance term vanish and Var⁡(X+Y)  =  Var⁡(X)  +  Var⁡(Y)\operatorname{Var}(X + Y)\; = \;\operatorname{Var}(X)\; + \;\operatorname{Var}(Y).
  • Crucial in portfolio theory (diversification reduces variance only if assets are not perfectly correlated), A/B testing (variance of the mean depends on within-subject correlation), and RL (variance-reduced estimators).
Permalink

Write covariance and correlation formulas.

easy
  • Cov⁡(X,  Y)  =  E[(X  −  μX)(Y  −  μY)]  =  E[XY]  −  E[X]\operatorname{Cov}(X, \;Y)\; = \;E[(X\; - \;{\mu}_{X})(Y\; - \;{\mu}_{Y})]\; = \;E[\mathrm{XY}]\; - \;E[X]E[Y].
  • Corr⁡(X,  Y)  =  ρ  =  Cov⁡(X,  Y)  /  (σX  σY)\operatorname{Corr}(X, \;Y)\; = \;{\rho}\; = \;\operatorname{Cov}(X, \;Y)\; / \;({\sigma}_{X}\;{\sigma}_{Y}), in [-1, 1].
  • Sample versions: divide by n-1 (unbiased).
  • Correlation is scale-invariant; covariance has weird units (product of X and Y units).
  • Cauchy-Schwarz inequality guarantees  ∣Cov⁡(X,Y)\mathrm{guarantees}\; \mid \operatorname{Cov}(X, Y)| ≤ σX{\sigma}_{X} σY{\sigma}_{Y}, so  ∣ρ\mathrm{so}\; \mid {\rho}| ≤ 1.
Permalink

What is E[X  ∣  Y]E[X\; \mid \;Y] and its Law of Total Expectation?

medium
  • E[X  ∣  Y  =  y]E[X\; \mid \;Y\; = \;y] is the expected value of X given that Y = y.
  • As a function of Y, E[X  ∣  Y]E[X\; \mid \;Y] is itself a random variable.
  • Law of total expectation (Adam's law): E[X]  =  E[E[X  ∣  Y]]E[X]\; = \;E[E[X\; \mid \;Y]] — average the conditional expectations over the distribution of Y.
  • Foundation for regression (E[Y∣X]  is  the  best  predictor  of  Y)(E[Y \mid X]\;\mathrm{is}\;\mathrm{the}\;\mathrm{best}\;\mathrm{predictor}\;\mathrm{of}\;Y), Bayes' rule interpretation, and MCMC.
Permalink

Law of Total Variance — Var⁡(X)\operatorname{Var}(X) = ?

hard
  • Var⁡(X)  =  E[Var⁡(X  ∣  Y)]  +  Var⁡(E[X  ∣  Y])\operatorname{Var}(X)\; = \;E[\operatorname{Var}(X\; \mid \;Y)]\; + \;\operatorname{Var}(E[X\; \mid \;Y]).
  • 'Within-group' variance (average of conditional variances) plus 'between-group' variance (variance of conditional means).
  • Foundation of ANOVA (partitions total variance into between- and within-group sources), variance decomposition in regression (R2  =  explained/total)(R^{2}\; = \;\mathrm{explained} / \mathrm{total}), and mixed-effects modeling.
Permalink

Why does the Cauchy distribution have no mean?

hard
  • Cauchy(0,1) PDF: 1  /  (π(1+x2))1\; / \;({\pi}(1 + x^{2})).
  • Its tails decay only as 1/x21 / x^{2} — the integral ∫x * PDF diverges → mean is undefined.
  • Consequence: sample mean of Cauchy does not converge (LLN fails).
  • Sums of Cauchy are still Cauchy (heavy-tail preservation), so CLT doesn't apply either.
  • Use case: robust statistics use it as a heavy-tailed model for outliers; ratio of two normals has Cauchy distribution.
Permalink

State Jensen's inequality and give an ML example.

hard
  • For a convex function g: E[g(X)] ≥ g(E[X]).
  • Concave: E[g(X)] ≤ g(E[X]).
  • Consequence: E[X2]E[X^{2}] ≥ (E[X])2(E[X])^{2} → variance is non-negative.
  • In ML: (1) log-likelihood is concave — Jensen bounds expected log-likelihood ≤ log of expected likelihood; foundation of the EM algorithm's E-step.
  • (2) log⁡(mean  prediction)\operatorname{log}(\mathrm{mean}\;\mathrm{prediction}) ≤ mean log⁡(prediction)\operatorname{log}(\mathrm{prediction}) — used in ensembling analyses.
  • (3) Explains why you can't just average logs then exponentiate to recover a mean.
Permalink

What is a moment generating function and why care?

hard
  • MX(t)  =  E[e(tX)]M_{X}(t)\; = \;E[e(\mathrm{tX})].
  • When it exists in an open interval around 0, it uniquely determines the distribution — two RVs with the same MGF are identically distributed.
  • Useful because: (1) generates moments via derivatives (MX(0)  =  E[X],  MX(0)  =  E[X2],  )(MX(0)\; = \;E[X], \;MX(0)\; = \;E[X^{2}], \;); (2) MGF of sum of independents = product of MGFs → easy proofs (sum of normals normal, sum of gammas gamma).
  • Characteristic functions (imaginary argument) exist even when MGF doesn't (Cauchy).
Permalink

State Chebyshev's inequality.

medium
  • P(∣X  −  μ∣  ≥  kσ)P( \mid X\; - \;{\mu} \mid \; \ge \;k{\sigma}) ≤ 1/k21 / k^{2}.
  • Distribution-free bound on tail probabilities — at most 1/k21 / k^{2} of the mass is more than k SDs from the mean, for any distribution with finite variance.
  • Consequence: at least 75% within 2σ, 89% within 3σ.
  • Much looser than the 95/99.7% for normal — general bounds are wide.
  • Applied in concentration arguments, generalization bounds in learning theory.
Permalink

State Markov's inequality.

medium
  • For a non-negative random variable X and a > 0: P(X ≥ a) ≤ E[X] / a.
  • Even weaker than Chebyshev but requires only non-negativity and a finite mean.
  • Used to derive Chebyshev (apply  Markov  to  (X−μ)2)(\mathrm{apply}\;\mathrm{Markov}\;\mathrm{to}\;(X - {\mu})^{2}) and Chernoff bounds (apply  Markov  to  e(tX)  and  optimize  t)(\mathrm{apply}\;\mathrm{Markov}\;\mathrm{to}\;e(\mathrm{tX})\;\mathrm{and}\;\mathrm{optimize}\;t).
  • Building block of concentration inequalities.
Permalink

What is Hoeffding's inequality?

hard
  • For bounded i.i.d.
  • XiX_{i} ∈ [a, b] with mean μ: P(∣Xn  −  μ∣  ≥  t)P( \mid Xn\; - \;{\mu} \mid \; \ge \;t) ≤ 2  ⋅  exp⁡(−2n  ⋅  t2  /  (b−a)2)2\; \cdot \;\operatorname{exp}( - 2n\; \cdot \;t^{2}\; / \;(b - a)^{2}).
  • Exponential (Gaussian-like) tail bound for sample means of bounded variables.
  • Much tighter than Chebyshev.
  • Used to prove PAC learning bounds, bandit / online learning regret bounds, and to derive sample-complexity requirements: n ~ log⁡(1/δ)  /  t2\operatorname{log}(1 / {\delta})\; / \;t^{2} for confidence δ and precision t.
  • Interview classic.
Permalink

How does Monte Carlo estimation work and its convergence rate?

medium
  • To estimate E[f(X)] where X has a known distribution: draw i.i.d. samples X1X_{1}, ..., XnX_{n}, average f(Xi)f(X_{i}).
  • By LLN, the average converges to the expectation.
  • Rate: O(1/√n) — SE⁡  =  σf\operatorname{SE}\; = \;{\sigma}_{f} / √n, independent of dimension.
  • Great for high-dimensional integrals where deterministic quadrature is impossible (curse of dimensionality).
  • Variance-reduction tricks: importance sampling, control variates, antithetic variates.
  • MCMC (Markov Chain Monte Carlo) uses MC when direct sampling is hard.
Permalink

What statistical properties make maximum likelihood the default estimator?

medium
  • Given data X and a parametric family p(x; θ), MLE picks θ̂ = argmax_θ Σ log p(xi  θ)p(x_{i}\;{\theta}).
  • Equivalently: minimize negative log-likelihood.
  • Properties: consistent (θ̂ → θ*), asymptotically normal (n(θ  −  θ⋅)  →  N(0,  I−1))( \sqrt n({\theta}\; - \;{\theta} \cdot )\; \to \;N(0, \;I^{-1})), asymptotically efficient (achieves Cramér-Rao bound).
  • Foundation of parametric inference and of most ML training objectives (logistic regression, softmax cross-entropy — all MLE).
Permalink

What is Fisher information?

hard
  • I(θ) = -E[∂2  log⁡  p(X  θ)  /  ∂θ2]  =  E[(∂  log⁡  p(X  θ)/∂θ)2]E[ \partial ^{2}\;\operatorname{log}\;p(X\;{\theta})\; / \; \partial {\theta}^{2}]\; = \;E[( \partial \;\operatorname{log}\;p(X\;{\theta}) / \partial {\theta})^{2}].
  • Measures how much information the observations carry about θ.
  • Cramér-Rao bound: for any unbiased estimator, Var⁡(θ)\operatorname{Var}({\theta}) ≥ 1 / (n * I(θ)).
  • MLE asymptotic variance = 1 / (n * I(θ)) → MLE is asymptotically efficient.
  • Used to define natural gradient (natural  gradient  descent  ∝  I(θ)−1  ∇L)(\mathrm{natural}\;\mathrm{gradient}\;\mathrm{descent}\; \propto \;I({\theta})^{-1}\; \nabla L).
Permalink

Bias, variance, consistency of estimators — define.

medium
  • Bias: E[θ̂] - θ.
  • Variance: Var⁡(θ)\operatorname{Var}({\theta}).
  • MSE⁡  =  bias2  +  variance\operatorname{MSE}\; = \;\mathrm{bias}^{2}\; + \;\mathrm{variance} (bias-variance decomposition).
  • Consistency: θ̂_n → θ in probability as n → ∞.
  • Unbiased ≠ consistent (unbiased with growing variance can be inconsistent); consistent ≠ unbiased (biased with vanishing bias can be consistent).
  • MLE and MAP are typically consistent but biased for small n.
Permalink

MLE for a Bernoulli(p) — derive.

medium
  • Likelihood: L(p) = p^Σxi  ⋅  (1−p){\Sigma}x_{i}\; \cdot \;(1 - p)^(n  −  Σxi)(n\; - \;{\Sigma}x_{i}).
  • Log-lik: ℓ(p)  =  Σxi  ⋅  log⁡(p)\; = \;{\Sigma}x_{i}\; \cdot \;\operatorname{log} p  +  (n  −  Σxi)  ⋅  log⁡(1−p)p\; + \;(n\; - \;{\Sigma}x_{i})\; \cdot \;\operatorname{log}(1 - p). ∂ℓ/∂p  =  Σxi  /  p  −  (n  −  Σxi)/(1−p)  =  0p\; = \;{\Sigma}x_{i}\; / \;p\; - \;(n\; - \;{\Sigma}x_{i}) / (1 - p)\; = \;0 → p̂ = Σxi  /  n  =  X{\Sigma}x_{i}\; / \;n\; = \;X̄.
  • So the MLE of a Bernoulli parameter is just the sample proportion.
  • Also happens to be unbiased and minimum-variance (via CR bound).
  • Classic textbook derivation asked in interviews.
Permalink

MLE for Normal(μ,  σ2)\mathrm{Normal}({\mu}, \;{\sigma}^{2}) — result.

medium
  • μ̂ = X̄ (unbiased, minimum variance). σ̂² = (1/n)  ⋅  Σ(Xi  −  X)2(1 / n)\; \cdot \;{\Sigma}(X_{i}\; - \;X)^{2} — the biased MLE.
  • Bias-corrected version divides by n-1 (sample variance).
  • For large n, both converge to σ2{\sigma}^{2}.
  • Deep-learning tie-in: minimizing MSE under Gaussian noise = MLE with fixed σ.
  • Softmax cross-entropy = MLE for a categorical distribution.
  • Almost every parametric ML training objective is an MLE.
Permalink

When would you use Hoeffding's inequality for a CI?

hard
  • Distribution-free CI for the mean of a bounded random variable in [a, b]: P(∣X  −  μ∣  >  t)P( \mid X\; - \;{\mu} \mid \; > \;t) ≤ 2  ⋅  exp⁡(−2nt2  /  (b−a)2)2\; \cdot \;\operatorname{exp}( - 2\mathrm{nt}^{2}\; / \;(b - a)^{2}).
  • Gives ε for target coverage: ε = (b-a) * √(log(2/α) / (2n)).
  • Uses: bandits (UCB), online learning regret, certification without normality.
  • Weaker (wider) than CLT-based CIs for well-behaved data but valid for any n and any distribution on [a, b].
Permalink

Cramér-Rao lower bound — state it.

hard
  • For any unbiased estimator θ̂ of θ under regularity conditions: Var⁡(θ)\operatorname{Var}({\theta}) ≥ 1 / (n * I(θ)).
  • Fisher information sets the floor for estimator precision.
  • MLE achieves this bound asymptotically (asymptotically efficient).
  • Practical use: benchmark for whether an estimator can be improved.
  • For biased estimators, generalized bounds exist (van Trees).
  • Companion to the bias-variance decomposition.
Permalink

What is the influence function?

hard
  • For a functional statistic T(F): IF(x  T,  F)  =  limε→0\mathrm{IF}(x\;T, \;F)\; = \;\mathrm{lim}_{{\varepsilon} \to 0} (T((1−ε)F  +  εδx)  −  T(F)T((1 - {\varepsilon})F\; + \;{\varepsilon}{\delta}_{x})\; - \;T(F)) / ε.
  • Measures how much T changes if a single point x is added.
  • Uses: (1) derive asymptotic variance via Var⁡(n  T)\operatorname{Var}( \sqrt n\;T) → E[IF2]E[\mathrm{IF}^{2}]; (2) build robust M-estimators (Huber) with bounded IF; (3) identify influential observations.
  • Foundation of modern robust statistics and semi-parametric efficiency theory.
Permalink

Give the OLS closed-form and its variance.

medium
  • β̂ = (XX)−1(XX)^{-1} X'y.
  • Var⁡(β  ∣  X)  =  σ2  ⋅  (XX)−1\operatorname{Var}({\beta}\; \mid \;X)\; = \;{\sigma}^{2}\; \cdot \;(XX)^{-1}.
  • Estimate σ2{\sigma}^{2} with s2  =  RSS  /  (n  −  p)s^{2}\; = \;\mathrm{RSS}\; / \;(n\; - \;p).
  • SEs come from diagonal of s2  ⋅  (XX)−1s^{2}\; \cdot \;(XX)^{-1}.
  • When X'X is ill-conditioned (multicollinearity), variances blow up.
  • Solutions: ridge regression, principal components regression, or dropping correlated features.
  • This formula is the reason multicollinearity is so damaging: it inflates the variance of β̂.
Permalink

State Bayes' theorem and what each term means.

easy
  • P(θ  ∣  D)  =  P(D  ∣  θ)  ⋅  P(θ)  /  P(D)P({\theta}\; \mid \;D)\; = \;P(D\; \mid \;{\theta})\; \cdot \;P({\theta})\; / \;P(D).
  • Posterior = Likelihood * Prior / Evidence.
  • Prior encodes belief before seeing D.
  • Likelihood tells how well θ explains D.
  • Evidence P(D) = ∫ P(D  ∣  θ)P(D\; \mid \;{\theta}) P(θ) dθ normalizes.
  • In practice, we compute posterior up to proportionality: P(θ  ∣  D)P({\theta}\; \mid \;D) ∝ P(D  ∣  θ)  ⋅  P(θ)P(D\; \mid \;{\theta})\; \cdot \;P({\theta}), and use MCMC / variational methods to sample or approximate.
Permalink

Frequentist vs Bayesian — key philosophical difference.

medium
  • Frequentist: parameters are fixed but unknown; probability = long-run frequency; CIs, p-values, MLE.
  • Bayesian: parameters are random with a distribution encoding belief; probability = degree of belief; posterior, credible intervals, MAP.
  • Pragmatic: Bayesian is natural when priors are meaningful, uncertainty quantification matters, or you're doing online learning; frequentist is faster and standard in confirmatory studies.
  • Modern ML mixes both freely (MAP = MLE + regularization; Bayesian deep learning).
Permalink

Potential outcomes framework — Rubin causal model.

hard
  • For each unit i and treatment T, define Yi(0)Y_{i}(0) (outcome if untreated) and Yi(1)Y_{i}(1) (outcome if treated).
  • Individual causal effect: Yi(1)  −  Yi(0)Y_{i}(1)\; - \;Y_{i}(0) — fundamentally unobservable (fundamental problem of causal inference).
  • Estimands: ATE = E[Y(1) - Y(0)], ATT  =  E[Y(1)  −  Y(0)  ∣  T=1]\mathrm{ATT}\; = \;E[Y(1)\; - \;Y(0)\; \mid \;T = 1].
  • Identification requires: (1) SUTVA (no interference / one version of treatment), (2) unconfoundedness (Y(0),  Y(1)    T  ∣  X)(Y(0), \;Y(1)\;\;T\; \mid \;X), (3) overlap.
  • Foundation of modern causal inference.
Permalink
02

Common distributions

5 concepts

Log-normal distribution and its uses.

medium
  • X ~ LogNormal(μ,  σ2)\mathrm{LogNormal}({\mu}, \;{\sigma}^{2}) if log⁡(X)\operatorname{log}(X) ~ N(μ,  σ2)N({\mu}, \;{\sigma}^{2}).
  • Right-skewed, always positive.
  • Mean: exp⁡(μ  +  σ2/2)\operatorname{exp}({\mu}\; + \;{\sigma}^{2} / 2); Median: exp⁡(μ)\operatorname{exp}({\mu}).
  • Uses: incomes, prices, response times, file sizes — quantities that are strictly positive with right skew.
  • Products of many small independent factors (like the multiplicative CLT) tend log-normal.
  • In DS: use log-transform to make skewed features more Gaussian-like for linear models.
Permalink

What is a power-law distribution and how do you spot one?

medium
  • P(X > x) ∝ x^(-α) for large x.
  • Very heavy right tail — a few extreme observations dominate the mean.
  • Examples: word frequencies, city sizes, wealth, network degree distributions (Zipf's / Pareto's law).
  • Spot them by plotting log⁡(rank⁡)\operatorname{log}(\operatorname{rank}) vs log⁡(size)\operatorname{log}(\mathrm{size}) — a straight line indicates power law.
  • Consequences: sample means are unstable, need heavy-tailed methods (median, quantile regression, robust stats).
Permalink

Beta distribution: setup and use case.

medium
  • X ~ Beta(α, β): support [0, 1], flexible shape controlled by α, β.
  • E[X] = α / (α+β); Var⁡  =  αβ  /  ((α+β)2(α+β+1))\operatorname{Var}\; = \;{\alpha}{\beta}\; / \;(({\alpha} + {\beta})^{2}({\alpha} + {\beta} + 1)).
  • Very flexible: uniform (α=β=1), U-shaped (α, β < 1), bell (α = β > 1), skewed (unequal α, β).
  • Standard Bayesian prior for probabilities: conjugate to Bernoulli/Binomial — Beta(α, β) + k successes in n trials → Beta(α+k, β+n-k).
  • Used for A/B testing priors and click-through-rate modeling.
Permalink

Gamma distribution and when it appears.

medium
  • X ~ Gamma(k, θ): support (0, ∞).
  • PDF ∝ x^(k-1) * e^(-x/θ).
  • E[X] = kθ; Var⁡  =  kθ2\operatorname{Var}\; = \;k{\theta}^{2}.
  • Special cases: Exponential  =  Γ(1,  θ)\mathrm{Exponential}\; = \;\Gamma(1, \;{\theta}); Erlang  =  Γ(integer  k)\mathrm{Erlang}\; = \;\Gamma(\mathrm{integer}\;k).
  • Uses: waiting time until k events in Poisson process, survival times, Bayesian conjugate prior for Poisson rate and normal precision.
  • Common as a heavy-tailed regression target (Gamma GLM for positive continuous outcomes: durations, insurance claims).
Permalink

Dirichlet distribution — where does it show up in ML?

hard
  • Dirichlet(α1,  ,  αK)\mathrm{Dirichlet}({\alpha}_{1}, \;, \;{\alpha}_{K}): distribution over probability vectors on the (K-1)-simplex (positive components summing to 1).
  • Multivariate generalization of Beta.
  • Conjugate prior for the categorical / multinomial distributions.
  • Uses: topic models (LDA — Latent Dirichlet Allocation), mixture-model weights, softmax priors, RL policy priors.
  • Concentration parameter Σαi{\Sigma}{\alpha}_{i} controls how peaked (large α) vs uniform (small α) the samples are.
Permalink
03

Expectation & variance

3 concepts

Why do we use standard deviation instead of variance in interpretation?

easy
  • Variance has squared units (dollars2,  meters2)(\mathrm{dollars}^{2}, \;\mathrm{meters}^{2}), which are unintuitive.
  • Standard deviation is the square root of variance and shares the units of the data, so it can be plotted alongside the mean and interpreted directly.
  • Statistical theory tends to use variance because it decomposes nicely (sum of independent variances), while reporting typically uses SD.
Permalink

Skewness and kurtosis — what do they measure?

medium
  • Skewness  =  E[((X  −  μ)/σ)3]\mathrm{Skewness}\; = \;E[((X\; - \;{\mu}) / {\sigma})^{3}] — asymmetry of the distribution.
  • Positive: long right tail (log-normal, income).
  • Negative: long left tail.
  • Kurtosis  =  E[((X  −  μ)/σ)4]\mathrm{Kurtosis}\; = \;E[((X\; - \;{\mu}) / {\sigma})^{4}] — 'tailedness'.
  • Normal has kurtosis 3 (excess kurtosis 0).
  • Higher (leptokurtic): heavier tails, more outliers (t, Laplace, financial returns).
  • Lower (platykurtic): lighter tails than normal.
  • Both give quick diagnostic beyond mean/variance.
Permalink

Why divide by n-1 in the sample variance?

easy
  • Sample variance uses estimated mean X̄ instead of true μ.
  • Since X̄ is closer to the data than μ (it minimizes SSD by definition), Σ(Xi  −  X)2{\Sigma}(X_{i}\; - \;X)^{2} is systematically smaller than Σ(Xi  −  μ)2{\Sigma}(X_{i}\; - \;{\mu})^{2}.
  • Dividing by n-1 (not n) corrects this bias → unbiased estimator of σ2{\sigma}^{2}.
  • Formally: one degree of freedom is used to estimate the mean, leaving n-1 for variance.
  • Same reason ANOVA / regression use df corrections.
Permalink
04

Descriptive statistics & EDA

24 concepts

Mean vs median vs mode — when do you prefer each?

easy
  • Mean: sensitive to outliers, but efficient for symmetric distributions.
  • Median: robust to outliers, right choice for skewed data (income, response times).
  • Mode: most-common value, useful for categorical or multimodal distributions.
  • Rule: report both mean and median for skewed data — a big gap between them signals skew.
Permalink

What are quantiles and quartiles?

easy
  • The q-quantile is the value below which fraction q of the data falls.
  • Median = 0.5-quantile.
  • Quartiles: Q1 (0.25), Q2 (0.5, median), Q3 (0.75).
  • Interquartile range IQR = Q3 - Q1.
  • Used for robust spread measurement, boxplots, outlier detection (points > Q3 + 1.5·IQR or < Q1 - 1.5·IQR are 'outliers' by Tukey's rule).
  • Foundation of quantile regression: model any quantile of Y∣XY \mid X, not just the mean.
Permalink

How do you read a boxplot?

easy
  • Box: from Q1 to Q3 (middle 50% of data).
  • Line in box: median.
  • Whiskers: extend to farthest points within 1.5·IQR of the box.
  • Points beyond whiskers: potential outliers.
  • Compare boxplots side-by-side to see distribution shape and outlier density across groups.
  • Weakness: hides multimodality (violin plots or histograms are better for shape).
Permalink

How do you choose bin width for a histogram?

medium
  • Rules of thumb: Sturges' (log2(n)+1, small n), Scott's (3.5σ  /  n(1/3),  assumes  near−normal)(3.5{\sigma}\; / \;n(1 / 3), \;\mathrm{assumes}\;\mathrm{near} - \mathrm{normal}), Freedman-Diaconis (2⋅IQR  /  n(1/3),  robust  to  skew)(2 \cdot \mathrm{IQR}\; / \;n(1 / 3), \;\mathrm{robust}\;\mathrm{to}\;\mathrm{skew}).
  • Modern default in most libraries: Freedman-Diaconis.
  • Too few bins hide structure; too many create noisy jaggedness.
  • Best practice: try 2-3 bin widths, or use KDE (kernel density estimate) for a smooth alternative.
Permalink

What is kernel density estimation?

medium
  • Non-parametric density estimate: KDE(x) = (1/(nh)) Σ K((x  −  xi)/h)K((x\; - \;x_{i}) / h), where K is a kernel (usually Gaussian) and h is bandwidth.
  • Bandwidth choice is the key knob: too small → wiggly, too large → over-smoothed.
  • Silverman's rule: h ≈ 1.06 * σ * n^(-1/5).
  • Better: cross-validation.
  • Smoother alternative to histograms for continuous data; also foundation of some non-parametric classifiers (KDE-NB, Parzen windows).
Permalink

How do you define an outlier in practice?

medium
  • No single definition — depends on the model and question.
  • Common rules: (1) Tukey: outside Q1 - 1.5·IQR or Q3 + 1.5·IQR.
  • (2) Z-score: |z| > 3 for approximately normal data.
  • (3) Modified z-score using MAD: |z| > 3.5 (more robust).
  • (4) Isolation Forest / DBSCAN for multivariate.
  • Never remove outliers blindly — investigate whether they're data errors, natural extremes, or a signal.
Permalink

What is MAD and why is it robust?

medium
  • Median Absolute Deviation: MAD(X)  =  median(∣Xi  −  median(X)∣)\mathrm{MAD}(X)\; = \;\mathrm{median}( \mid X_{i}\; - \;\mathrm{median}(X) \mid ).
  • Robust measure of dispersion — breakdown point 50% (half the data can be arbitrarily corrupted before MAD explodes; SD breaks down at 0%).
  • Use MAD instead of SD when data has outliers.
  • Scaled MAD: 1.4826 * MAD estimates σ under normality (approximately unbiased).
  • Foundation of robust statistics.
Permalink

Describe a good EDA workflow for a new dataset.

medium
  • (1) Shape: rows × columns, dtypes, missing values.
  • (2) Univariate: histograms / boxplots / value counts per column.
  • (3) Target: distribution of the outcome, class balance.
  • (4) Bivariate: correlations, scatter plots, feature-target relationship.
  • (5) Missingness: patterns (MCAR / MAR / MNAR)?
  • (6) Duplicates and near-duplicates.
  • (7) Time / group structure.
  • (8) Sanity checks: min/max make sense? plausibility of extremes?
  • Report findings, then decide on cleaning + feature engineering.
Permalink

MCAR vs MAR vs MNAR — what's the difference?

hard
  • MCAR (Missing Completely At Random): missingness independent of both observed and unobserved data.
  • Can drop rows without bias.
  • MAR (Missing At Random): missingness depends only on observed data — imputable given the observed features.
  • MNAR (Missing Not At Random): missingness depends on the unobserved value itself (e.g., high-income people refuse to report income).
  • Only MNAR requires modeling the missingness mechanism.
  • Assumption matters — different methods assume different types.
Permalink

What imputation methods should you consider?

medium
  • (1) Drop rows: fine if MCAR and few missing.
  • (2) Mean/median/mode imputation: baseline, distorts variance.
  • (3) KNN imputation: uses similar rows' values.
  • (4) Iterative (MICE) imputation: regress each missing column on the others, iterate.
  • (5) Model-based (missForest, DL).
  • (6) Multiple imputation: create m imputed datasets, fit model on each, combine (Rubin's rules) — captures imputation uncertainty.
  • For high-stakes analysis, prefer MICE or multiple imputation.
Permalink

When and why do you log-transform a variable?

easy
  • (1) Right-skewed positive variables (income, prices, response times): log transform pulls in the tail, makes distribution more symmetric, often more Gaussian.
  • (2) Multiplicative relationships become additive after log — foundation of Box-Cox / Yeo-Johnson and log-linear models.
  • (3) Improves linearity for regression.
  • Watch: log⁡(0)\operatorname{log}(0) undefined — use log⁡(1+x)\operatorname{log}(1 + x) or add a small offset.
  • Not helpful for symmetric or already-Gaussian data.
Permalink

What is the Box-Cox transformation?

medium
  • Box−Cox(y,  λ)  =  (yλ  −  1)  /  λ\mathrm{Box} - \mathrm{Cox}(y, \;{\lambda})\; = \;(y{\lambda}\; - \;1)\; / \;{\lambda} if λ ≠ 0, log⁡(y)\operatorname{log}(y) if λ = 0.
  • Choose λ (typically via MLE) to make the transformed variable closest to normal.
  • Extended to negative values by Yeo-Johnson.
  • Uses: normalize skewed regression targets, improve linear-model residual normality.
  • In modern ML, tree models don't need it; useful for linear / GLM diagnostics.
Permalink

Pearson vs Spearman vs Kendall correlation — when do you use each?

easy
  • Pearson r: measures linear relationship, assumes approximately normal data, sensitive to outliers.
  • Spearman rho: rank-based Pearson — captures any monotonic relationship, robust to outliers.
  • Kendall tau: also rank-based, based on concordant/discordant pairs — more robust to small samples, less sensitive to outliers.
  • Rule: Pearson for linear + normal + no outliers; Spearman/Kendall for monotonic + rank-based data.
Permalink

What is an ECDF and why is it useful?

medium
  • Empirical Cumulative Distribution Function: Fn(x)  =  (1/n)  ⋅  ΣF_{n}(x)\; = \;(1 / n)\; \cdot \;{\Sigma} 1(Xi  ≤  x)1(X_{i}\; \le \;x).
  • Plots the fraction of data ≤ x for every x.
  • No binning — no arbitrary choice like histograms.
  • Useful for: comparing distributions visually (overlap two ECDFs), computing quantiles at a glance, checking distributional assumptions (compare ECDF to a theoretical CDF via Kolmogorov-Smirnov test).
Permalink

How do you read a Q-Q plot?

medium
  • Q-Q (quantile-quantile) plot: plot empirical quantiles vs theoretical quantiles (typically normal).
  • Perfect fit → points on the y=x line.
  • Deviations reveal distribution shape: (1) S-shape → heavier or lighter tails than normal; (2) curved → skew; (3) points off at the ends → outliers.
  • Fast visual check for normality / distributional fit before running parametric tests.
  • Complement with Shapiro-Wilk / Kolmogorov-Smirnov for formal tests.
Permalink

Standardization vs normalization — what's the difference?

easy
  • Standardization (Z-score): (x - μ) / σ → mean 0, SD 1.
  • Preserves shape, robust when features have similar scales after transform.
  • Normalization / MinMax: (x - min) / (max - min) → bounded [0, 1].
  • Sensitive to outliers (a single extreme value squashes everything).
  • Use standardization for linear models, PCA, k-means, neural networks.
  • Use normalization when a bounded range matters (image pixel scaling to [0,1]).
  • Robust scaling: (x - median) / IQR — outlier-robust alternative.
Permalink

What multivariate EDA plots are most useful?

medium
  • (1) Correlation heatmap: quick view of all pairwise linear relationships.
  • (2) Scatter plot matrix (pairs plot): all bivariate scatters, diagonal shows univariate — best for small-to-medium feature sets.
  • (3) Parallel coordinates: high-dim shape visualization.
  • (4) PCA / UMAP 2D projections: capture nonlinear structure.
  • (5) 2D density / hex bins: for many-point scatters.
  • (6) Facet grids: same plot across a categorical variable.
  • Choose based on dimensionality and hypothesis.
Permalink

How does EDA help catch target leakage?

medium
  • Look for features with suspicious high correlation to target: (1) any feature perfectly predictive by itself is suspect; (2) features generated after the target event should not exist at prediction time; (3) IDs / timestamps sometimes carry target information indirectly.
  • Cross-checks: check feature availability at inference time (would you have this feature when predicting?); check for target-derived aggregations (e.g., 'average order value' when predicting a churn outcome computed after the churn).
Permalink

Why should you always do group-wise EDA?

medium
  • Whole-population statistics can hide subgroup patterns that matter for the model: (1) Simpson's paradox — trends flip when you disaggregate.
  • (2) Fairness — model performance may differ by demographic group.
  • (3) Missingness patterns may vary by group.
  • (4) Different scale / variance per group.
  • (5) Distinct outlier populations.
  • Practice: pick 2-3 key categorical variables (region, cohort, product), replot univariate + bivariate patterns by group.
  • Reveals lots of hidden structure.
Permalink

What time-series-specific EDA should you do?

medium
  • (1) Plot the raw series and its rolling mean / median at multiple window sizes.
  • (2) Decompose into trend + seasonality + residual (STL).
  • (3) Autocorrelation (ACF) and partial autocorrelation (PACF) plots to detect lag structure.
  • (4) Stationarity tests (ADF, KPSS).
  • (5) Change-point detection.
  • (6) Frequency-domain view (spectrogram) for periodic components.
  • (7) Compare series across shared time windows and groups.
  • Foundation for choosing between ARIMA / Prophet / Bayesian structural time series.
Permalink

What do you look for in residual plots?

medium
  • Residuals vs fitted: no pattern = OK; funnel shape = heteroscedasticity; curvature = missed non-linearity.
  • Q-Q plot: check normality of residuals (banana → skew, sigmoid → heavy tails).
  • Residuals vs order (for time series): autocorrelation.
  • Scale-location plot: √|residuals∣  vs\mathrm{residuals} \mid \;\mathrm{vs} fitted for heteroscedasticity.
  • Leverage vs residuals: identify influential points (Cook's distance).
  • Standard 4-plot diagnostic every regression should get.
Permalink

When should you log-transform the response?

medium
  • (1) Right-skewed positive outcomes (income, prices, spend, time-on-page).
  • (2) Multiplicative errors → additive on log scale.
  • (3) Coefficients become interpretable as approximate percent changes for small β.
  • Cost: E[log Y] ≠ log E[Y] → back-transformed predictions are biased low (Duan smearing correction fixes it).
  • Alternatives: GLM with log link (models log of mean directly, avoids the bias), Box-Cox / Yeo-Johnson for automatic transformation search.
Permalink

Overall conversion went down but improved in every country. How is that possible?

hard
  • The traffic mix changed.
  • Simpson's paradox appears when the groups have different baseline rates and their relative share shifts, so an aggregate can move opposite to every subgroup.
  • Concretely, if a low-converting country grew as a fraction of traffic, the overall rate falls even though each country improved.
  • This is why aggregate metrics over a heterogeneous population are unreliable when composition is not held fixed, which happens constantly with marketing campaigns and regional launches.
  • The remedy is to compare like with like: stratify by the confounding variable and report a weighted average using a fixed reference composition, or model the outcome with the segment included.
  • In a randomized experiment this cannot happen by chance across arms, so if it does, suspect a broken assignment or differential logging.
Permalink

How do you spot selection bias in a dataset someone hands you?

hard
  • Ask how a row came to exist, which is a different question from what the row contains.
  • Every dataset is the output of a process that decided what to record, and that process is often correlated with the outcome.
  • Concretely, look for entities that could have been in the data but are not: churned customers absent from a satisfaction table, rejected loan applicants absent from a default model's training set, machines that failed before the sensor was installed.
  • Check whether the sampling rate varies by any variable you care about, and compare the dataset's marginal distributions against a known population where one exists.
  • Then ask what the missingness depends on, because data missing as a function of the unobserved outcome cannot be repaired by imputation and needs a model of the selection itself.
Permalink
05

Sampling & central limit theorem

2 concepts

What is sampling bias and how do you mitigate it?

easy
  • Sampling bias occurs when the sample is not representative of the target population — some groups over- or under-represented.
  • Causes: convenience sampling, self-selection, survivorship bias, non-response.
  • Mitigations: random sampling with a clear frame, stratified sampling to ensure coverage of subgroups, post-stratification weighting, and comparing sample demographics to population statistics.
Permalink

Your revenue-per-user metric is extremely right-skewed. Can you still use a t-test?

hard
  • Usually yes, but for a reason worth stating precisely: the t-test requires the sampling distribution of the mean to be approximately normal, not the data, and the central limit theorem delivers that at large sample sizes even for skewed data.
  • The practical caveat is that convergence is slower the heavier the tail, so with a few hundred observations and extreme outliers the test can still be unreliable.
  • Better options are to cap or winsorize the metric at a high percentile, which reduces variance dramatically at the cost of a slightly different estimand, or to bootstrap the difference in means, which makes no distributional assumption.
  • Do not switch to a rank test without thinking, because the Mann-Whitney test answers a question about stochastic ordering rather than about mean revenue, and mean revenue is what the business cares about.
Permalink
06

Hypothesis testing

33 concepts

What is a p-value, precisely?

easy
  • The probability, assuming the null hypothesis is true, of observing a test statistic at least as extreme as the one obtained.
  • It is NOT the probability that the null is true, nor the probability of a mistake.
  • A small p-value means the data are unlikely under H0.
  • Combine with effect size and confidence intervals — never rely on p-values alone.
Permalink

Type I vs Type II error — what's the difference?

easy
  • Type I error (false positive): rejecting H0 when it's actually true.
  • Its probability is alpha (significance level, often 0.05).
  • Type II error (false negative): failing to reject H0 when H1 is true.
  • Its probability is beta; power  =  1  −  β\mathrm{power}\; = \;1\; - \;\beta.
  • There's a tradeoff — lowering alpha raises beta unless you increase sample size or effect size.
Permalink

Why is multiple testing a problem and how do you correct for it?

medium
  • Running many independent tests at α  =  0.05\alpha\; = \;0.05 will produce many false positives by chance alone (5% per test).
  • Corrections: Bonferroni (divide alpha by number of tests — very conservative), Holm-Bonferroni (stepwise, better power), or control the False Discovery Rate with Benjamini-Hochberg (best for discovery-style work with many tests).
Permalink

How do you formulate null and alternative hypotheses?

easy
  • Null H0: the 'no effect' / 'no difference' hypothesis (μA  =  μB,  β  =  0,  treatment  has  no  impact)({\mu}_{A}\; = \;{\mu}_{B}, \;{\beta}\; = \;0, \;\mathrm{treatment}\;\mathrm{has}\;\mathrm{no}\;\mathrm{impact}).
  • Alternative H1: what we suspect (μA  ≠  μB  two−sided,  or  μA  >  μB  one−sided)({\mu}_{A}\; \ne \;{\mu}_{B}\;\mathrm{two} - \mathrm{sided}, \;\mathrm{or}\;{\mu}_{A}\; > \;{\mu}_{B}\;\mathrm{one} - \mathrm{sided}).
  • One-sided tests are more powerful but only appropriate when you commit ex ante to the direction.
  • Rule: define H0 / H1 before seeing the data.
  • Post-hoc directional testing inflates type I error.
Permalink

One-sample t-test: setup and assumptions.

easy
  • H0: μ  =  μ0{\mu}\; = \;{\mu}_{0}.
  • Statistic: t  =  (X  −  μ0)  /  (s  /  n)t\; = \;(X\; - \;{\mu}_{0})\; / \;(s\; / \; \sqrt n), where s is sample SD.
  • Under H0 and normality, t ~ tn−1t_{n - 1}.
  • Assumptions: (1) i.i.d. samples; (2) approximately normal (robust for n > 30 due to CLT).
  • Compare t to t-distribution quantiles or compute a p-value.
  • Use case: is the mean of a metric different from a target?
Permalink

Two-sample t-test: pooled vs Welch — when to use each?

medium
  • Compare two group means.
  • Pooled (Student's): assumes equal variances → uses pooled SD, df  =  n1  +  n2  −  2\mathrm{df}\; = \;n_{1}\; + \;n_{2}\; - \;2.
  • Welch's: doesn't assume equal variances → uses Satterthwaite df correction.
  • In practice, Welch is the safer default (behavior similar to Student when variances are equal, better when they aren't).
  • Both assume i.i.d. within groups and approximate normality (CLT-robust for larger n).
Permalink

When do you use a paired t-test?

easy
  • Same subjects measured under two conditions (before / after treatment), or naturally paired observations (twins, matched pairs).
  • Compute differences di  =  XAi  −  XBid_{i}\; = \;X_{A}i\; - \;X_{B}i, then one-sample t on d.
  • Much more powerful than an unpaired test when the within-pair correlation is high — cancels out subject-level variation.
  • Assumption: differences approximately normal (CLT-robust for n > 30).
Permalink

z-test vs t-test — when to pick each?

easy
  • z-test: known population variance (rare in practice), or very large n (>~100) where t-distribution ≈ normal. t-test: unknown population variance (usual case).
  • For proportions with large n: z-test for a proportion or two-proportion z-test.
  • The 'z-test for a proportion' is common in A/B testing.
  • Modern practice: default to t-test unless you specifically know σ.
Permalink

Chi-square test of independence — what does it do?

medium
  • Tests whether two categorical variables are independent, given an r × c contingency table.
  • Statistic: χ2  =  Σ{\chi}^{2}\; = \;{\Sigma} (Oij  −  Eij)2  /  Eij(O_{\mathrm{ij}}\; - \;E_{\mathrm{ij}})^{2}\; / \;E_{\mathrm{ij}}, where Eij  =  rowi  ⋅  colj  /  NE_{\mathrm{ij}}\; = \;\mathrm{row}_{i}\; \cdot \;\mathrm{col}_{j}\; / \;N under independence. df = (r-1)(c-1).
  • Requires EijE_{\mathrm{ij}} ≥ 5 for validity.
  • Use case: is conversion rate independent of region?
  • For small counts, use Fisher's exact test instead.
Permalink

Chi-square goodness-of-fit test — setup.

medium
  • Tests whether observed frequencies match an expected distribution: χ2  =  Σ{\chi}^{2}\; = \;{\Sigma} (Oi  −  Ei)2  /  Ei(O_{i}\; - \;E_{i})^{2}\; / \;E_{i}, df = k - 1 - (parameters estimated).
  • Uses: is a die fair?
  • Does a dataset follow Poisson?
  • Requires EiE_{i} ≥ 5 (aggregate small buckets).
  • Modern alternatives: G-test (likelihood ratio), Kolmogorov-Smirnov (continuous), Anderson-Darling (heavier weight on tails).
Permalink

What does one-way ANOVA test?

medium
  • H0: μ1  =  μ2{\mu}_{1}\; = \;{\mu}_{2} = ... = μk{\mu}_{k} (all group means equal).
  • Statistic: F = between-group variance / within-group variance ~ Fk−1,  N−kF_{k - 1, \;N - k} under H0.
  • Assumptions: (1) i.i.d. within groups; (2) normally distributed; (3) equal variances (Welch's ANOVA relaxes this).
  • If F is significant, follow-up with post-hoc tests (Tukey HSD, Bonferroni) to identify which groups differ.
  • Non-parametric alternative: Kruskal-Wallis.
Permalink

Two-way ANOVA and what interactions mean.

hard
  • Tests three effects at once: main effect of factor A, main effect of factor B, and A×B interaction.
  • Interaction: the effect of A depends on the level of B.
  • Significant interaction means main effects are hard to interpret in isolation (plot cell means to visualize).
  • Assumptions: normality + equal variances + independence within cells.
  • Foundation of factorial experimental design.
Permalink

Mann-Whitney U (Wilcoxon rank-sum) — when and why?

medium
  • Non-parametric test comparing two independent groups.
  • Combines all data, ranks it, compares rank sums between groups.
  • Doesn't assume normality — good for skewed, ordinal, or small-sample data.
  • Effectively tests whether one distribution is shifted vs the other.
  • Less powerful than t-test when normality holds; more robust when it doesn't.
  • Not exactly a 'test of medians' — it's a stochastic-dominance test.
Permalink

Wilcoxon signed-rank test — setup.

medium
  • Non-parametric alternative to the paired t-test.
  • Compute differences did_{i}, rank⁡  ∣di\operatorname{rank}\; \mid d_{i}|, sum ranks separately for positive and negative differences.
  • Null: distribution of differences is symmetric around zero.
  • Robust to non-normal differences, sensitive to outliers still (unlike sign test).
  • Uses ranks so drops information but keeps power when normality is doubtful.
Permalink

Kruskal-Wallis — non-parametric ANOVA analog.

medium
  • Extends Mann-Whitney to k > 2 groups.
  • Rank all data across groups, compute H = 12/(N(N+1)) * Σ (Ri2  /  ni)  −  3(N+1)(R_{i}^{2}\; / \;n_{i})\; - \;3(N + 1) ~ χ2k−1{\chi}^{2}k - 1 approximately.
  • Null: all groups drawn from same distribution.
  • If significant, follow up with pairwise Mann-Whitney with Bonferroni.
  • Use for skewed / ordinal / non-normal data with 3+ groups.
Permalink

Friedman test and its use case.

hard
  • Non-parametric alternative to repeated-measures ANOVA.
  • Same subjects rated on k treatments; rank within each subject, sum ranks per treatment.
  • Null: no treatment effect.
  • Useful for within-subject designs where normality is violated.
  • Follow-up: pairwise Wilcoxon with correction.
  • Common in ML benchmarking (rank algorithms across datasets).
Permalink

Kolmogorov-Smirnov test — one- and two-sample.

medium
  • Statistic: D  =  max⁡x  ∣F1(x)  −  F2(x)D\; = \;\operatorname{max}_{x}\; \mid F_{1}(x)\; - \;F_{2}(x)| — max vertical distance between two ECDFs (two-sample) or between empirical and theoretical CDF (one-sample).
  • Distribution-free, works for continuous data, sensitive to shifts in location, scale, or shape.
  • Weakness: less sensitive in the tails.
  • Use in drift detection (compare distribution today vs baseline).
Permalink

Shapiro-Wilk normality test — when useful?

medium
  • Tests whether data comes from a normal distribution.
  • Very sensitive for n < 50; over-powered for large n (reports significant non-normality for trivial deviations).
  • Practical rule: for large samples, rely more on Q-Q plots than on the p-value.
  • Alternatives: Anderson-Darling (weighted toward tails), Kolmogorov-Smirnov (less sensitive), Lilliefors (KS variant with estimated params).
Permalink

How do you test equality of variances?

medium
  • Bartlett: assumes normality; sensitive to non-normality.
  • Levene: uses absolute deviations from mean or median (Brown-Forsythe uses median); more robust.
  • F-test of variances: strictly two-sample under normality.
  • Use Levene (with median) as the safe default.
  • Purpose: check equal-variance assumption for ANOVA / pooled t-test; if violated, use Welch's variants.
Permalink

Why report effect size alongside p-values?

easy
  • P-values conflate signal size with sample size — huge n makes trivial effects significant.
  • Effect size quantifies practical importance regardless of n.
  • Common measures: Cohen's d (standardized mean difference: 0.2 small, 0.5 medium, 0.8 large); odds ratio / relative risk (binary outcomes); Pearson r  /  R2r\; / \;R^{2}; Cliff's delta (non-parametric).
  • Always report effect size with CIs and p-values.
Permalink

What is power analysis and how do you use it?

medium
  • Power  =  P(reject  H0  ∣  H1  true)  =  1  −  β\mathrm{Power}\; = \;P(\mathrm{reject}\;\mathrm{H0}\; \mid \;\mathrm{H1}\;\mathrm{true})\; = \;1\; - \;{\beta}.
  • Power analysis: given effect size, α, and power (typically 0.8), solve for required sample size.
  • Or: given n, α, effect size, solve for achieved power.
  • Uses: (1) plan A/B tests (how many users do we need?); (2) diagnose 'no significant effect' — was the study under-powered?
  • Free tools: G*Power, statsmodels.power.
  • Always do this before running an experiment.
Permalink

What is MDE (minimum detectable effect)?

medium
  • Smallest effect size the experiment can detect with a target power (usually 0.8) given the sample size and α.
  • Computed via power formula.
  • Practical use: 'with N=10000 users per arm at α=0.05 and power 0.8, the MDE is a 2% lift'.
  • If your product intuition says the effect might be 1%, you need to grow n.
  • Cornerstone of A/B testing capacity planning.
Permalink

Bonferroni correction — how does it work?

medium
  • For k independent tests at family-wise error rate α: use αadjusted  =  α  /  k{\alpha}_{\mathrm{adjusted}}\; = \;{\alpha}\; / \;k per test.
  • Guarantees P(any false rejection) ≤ α under independence, ≤ α always (union bound).
  • Very conservative — loses power quickly as k grows.
  • Use for a small number of confirmatory tests (< 20).
  • Holm's step-down variant is uniformly more powerful with the same guarantee.
Permalink

How does the Benjamini-Hochberg procedure work?

hard
  • Controls the FDR (expected fraction of false positives among rejections) at level q.
  • Steps: (1) Sort p-values p_(1) ≤ ... ≤ p_(m).
  • (2) Find largest k such that p_(k) ≤ (k/m) * q.
  • (3) Reject all H_(i) with i ≤ k.
  • Much more powerful than Bonferroni for large m — accepts a small fraction of false positives to detect many true effects.
  • Standard in genomics, feature selection, high-dim testing.
Permalink

What is a permutation test?

medium
  • Non-parametric test based on the exchangeability principle: shuffle group labels, recompute the test statistic, build a null distribution empirically.
  • Compute p-value = fraction of permutations with statistic ≥ observed.
  • Works for any statistic (mean,  median,  difference  in  R2)(\mathrm{mean}, \;\mathrm{median}, \;\mathrm{difference}\;\mathrm{in}\;R^{2}).
  • Distribution-free, only assumes exchangeability under H0.
  • Slow (usually 10k+ shuffles) but very general.
  • Modern default when you want strong assumption independence.
Permalink

What does McNemar's test do?

hard
  • Paired categorical (usually binary) test.
  • Compares proportions in matched pairs.
  • From a 2×2 table of concordant / discordant pairs, χ2  =  (b  −  c)2  /  (b  +  c){\chi}^{2}\; = \;(b\; - \;c)^{2}\; / \;(b\; + \;c) approximately (or exact binomial for small counts).
  • Uses: classifier comparison on the same test set (does model A get more test items right than model B?), before / after intervention on the same subjects.
Permalink

When to use Fisher's exact test?

medium
  • 2×2 (or r×c) contingency table with small expected counts (E < 5) where chi-square approximation fails.
  • Computes an exact p-value from the hypergeometric distribution.
  • Slow for large tables.
  • Standard for small clinical trials, rare-event categorical data.
  • Modern alternative for larger tables: Monte Carlo χ2{\chi}^{2} (simulate null distribution).
Permalink

What is p-hacking and how do you prevent it?

medium
  • Running many analyses (test variants, subgroups, transformations) until you find p < 0.05, then reporting only the significant results.
  • Inflates false-positive rate dramatically.
  • Prevention: (1) pre-register hypotheses + analysis plan; (2) Bonferroni / FDR on all tested variants; (3) separate exploration (open) from confirmation (locked test set + pre-registered stat plan); (4) report all comparisons, not just significant ones.
  • Foundation of the replication crisis.
Permalink

FWER vs FDR — which do you control when?

hard
  • FWER (Family-Wise Error Rate): P(any false rejection).
  • Controlled by Bonferroni / Holm — safe when even a single false positive is costly (safety, regulatory).
  • FDR (False Discovery Rate): E[false / total rejections].
  • Controlled by Benjamini-Hochberg — accepts some false positives to keep power in discovery / exploratory settings.
  • Rule: FWER for confirmatory / high-stakes; FDR for exploratory / genomics / feature selection.
Permalink

How is a CI related to a hypothesis test?

medium
  • A 95% CI for a parameter includes all null values that would NOT be rejected at α = 0.05 in a two-sided test.
  • Equivalently: if 0 is outside the 95% CI of the difference, then reject H0: no difference at α = 0.05.
  • CIs are more informative — they give you the range of plausible values, not just accept / reject.
  • Always report CIs.
Permalink

Why is peeking at running A/B tests dangerous?

hard
  • Peeking = repeatedly running a test and stopping when p < 0.05 → hugely inflates false-positive rate (up to ~40% for one-sided at α=0.05).
  • Fixes: (1) commit to sample size ex ante and don't peek.
  • (2) Sequential testing: mSPRT, α-spending (Pocock, O'Brien-Fleming), Bayesian sequential decisions.
  • (3) Always-valid inference (Howard & Ramdas): give up 10-20% efficiency for the freedom to check any time.
  • Optimizely / Statsig / Eppo implement always-valid or sequential correctly.
Permalink

What is alpha-spending in sequential testing?

hard
  • Divide the total α (0.05) across scheduled interim looks so cumulative false-positive rate stays capped.
  • Pocock: uniform per look (conservative).
  • O'Brien-Fleming: tiny α early, most saved for the final look (encourages waiting).
  • Haybittle-Peto: small α at interims, α - 0.001 at final.
  • All maintain family-wise error at 0.05 across multiple looks — enables ethical stopping in clinical trials.
Permalink

A product manager asks what a p-value of 0.03 means. What do you say?

easy
  • Say that if the change truly had no effect, you would see a difference this large or larger about 3% of the time by chance alone.
  • Then say what it does not mean, because that is where decisions go wrong: it is not the probability that the change works, and it is not the probability that the null hypothesis is true.
  • Add the part that actually matters for the decision, which is the effect size and its confidence interval, since a significant result whose interval spans from trivially small to large does not justify a launch.
  • Frame the conclusion as a decision under uncertainty, weighing the cost of shipping a neutral change against the cost of missing a real one, rather than as a verdict delivered by the threshold.
Permalink
07

Non-parametric tests

1 concept

How many bootstrap resamples do you need, and which statistics does it handle?

medium
  • Given data X1X_{1},...,XnX_{n}, sample n observations with replacement to form X*, compute the statistic θ̂*, repeat B ≈ 1000-10000 times to build an empirical distribution of θ̂.
  • Use it to compute SEs, CIs, and bias.
  • Works for any statistic (median,  quantiles,  R2,  ratios)(\mathrm{median}, \;\mathrm{quantiles}, \;R^{2}, \;\mathrm{ratios}), no distributional assumption.
  • Weaknesses: bad for extreme statistics (max), heavy tails, small n, and violated i.i.d.
Permalink
08

Power, effect size & multiple testing

5 concepts

What techniques reduce variance in A/B tests?

hard
  • (1) CUPED: covariate adjustment using a pre-experiment baseline metric — 30-70% variance reduction on retention metrics.
  • (2) Stratification: guarantee balanced enrollment by strata (region, device).
  • (3) Doubly-robust estimators.
  • (4) Ratio metrics with delta-method variance.
  • (5) Winsorize extreme values to reduce heavy-tail variance.
  • All target the same goal: smaller CIs at the same n → detect smaller effects.
  • CUPED is Microsoft / Meta / Netflix standard.
Permalink

How does CUPED reduce variance in A/B tests?

hard
  • CUPED (Controlled-experiment Using Pre-Experiment Data, Deng et al. 2013): use a pre-experiment covariate X (e.g. same metric, 30 days pre-treatment) to explain part of Y's variance.
  • Adjusted metric: Y' = Y - θ * (X - E[X]), where θ = Cov(Y, X)/Var(X).
  • Var⁡(Y)  =  Var⁡(Y)  ⋅  (1  −  Corr⁡(Y,  X)2)\operatorname{Var}(Y)\; = \;\operatorname{Var}(Y)\; \cdot \;(1\; - \;\operatorname{Corr}(Y, \;X)^{2}).
  • Typical 30-70% variance reduction on retention / spend metrics → 3-5x fewer users for same MDE.
  • Standard at Microsoft, Meta, Netflix, Booking.
Permalink

Why is checking an A/B test daily and stopping when it turns significant wrong?

medium
  • Because the false positive rate is not the nominal 5% under repeated looks; every additional peek is another chance for random fluctuation to cross the threshold, and with daily checks over a few weeks the real error rate rises to something like 20 to 30%.
  • Fixed-horizon tests assume exactly one analysis at a predetermined sample size, and stopping on the first significant result systematically selects the fluctuations that happened to be extreme.
  • The honest options are to fix the sample size in advance from a power calculation and look only once, or to use a method designed for continuous monitoring: a sequential test with alpha spending such as a group sequential design, or always-valid confidence sequences.
  • If you must peek for operational safety, peek at guardrails and not at the primary metric.
Permalink

What do you need to know to compute a sample size for an experiment?

medium
  • Four things, and the hard one is not statistical.
  • The baseline rate of the metric, its variance, which for a proportion follows from the baseline but for a revenue metric must be estimated and is usually much larger than people expect.
  • The minimum detectable effect, meaning the smallest improvement that would actually change a decision, which is a business judgement and the input most often chosen to make the number look acceptable.
  • Then the significance level and power, conventionally 5% and 80%, where power is the probability of detecting the effect if it is real.
  • Sample size scales inversely with the square of the effect size, so halving the detectable effect quadruples the sample, which is why heavy-tailed revenue metrics often cannot be tested directly and are replaced by a capped or binary proxy.
Permalink

Your experiment tracks 20 metrics and one is significant. What now?

medium
  • Treat it as expected noise until proven otherwise, because with 20 independent tests at the 5% level you expect about one false positive even when nothing changed.
  • The structural fix is to declare a single primary metric before the experiment, with the rest labelled as secondary or guardrail, which turns a fishing expedition into a hypothesis test.
  • If several metrics genuinely need decision authority, control the error rate explicitly: Bonferroni is conservative and fine for a handful, and the Benjamini-Hochberg procedure controls the false discovery rate and is the better choice when you have many.
  • For the metric that fired, ask whether it moved in a direction consistent with a plausible mechanism and with the other metrics, and if it is isolated and surprising, the correct action is to replicate rather than to launch.
Permalink
09

Estimation: MLE, MoM, MAP

18 concepts

What does a 95% confidence interval mean?

medium
  • If we repeated the sampling and interval construction procedure many times, about 95% of the resulting intervals would contain the true parameter.
  • It's NOT 'there is a 95% probability the parameter is in this specific interval' — the parameter is fixed and the interval is random.
  • Interpret CIs as long-run coverage guarantees.
Permalink

What is the bootstrap and when do you use it?

medium
  • Bootstrap resamples the data with replacement to approximate the sampling distribution of an estimator.
  • You get standard errors, confidence intervals, and bias estimates without strong parametric assumptions.
  • Use it when analytical formulas are unavailable or unreliable (medians, ratios, complex estimators).
  • Rule of thumb: 1,000-10,000 bootstrap samples.
Permalink

What is Maximum Likelihood Estimation?

medium
  • Choose parameter values that make the observed data most probable — i.e., maximize the likelihood function L(θ)  =  P(data  ∣  θ)L(\theta)\; = \;P(\mathrm{data}\; \mid \;\theta).
  • In practice, maximize log L (numerically stable, turns products into sums).
  • MLEs are consistent, asymptotically normal, and efficient under regularity conditions.
  • Add regularization/priors to obtain MAP estimation (Bayesian penalty).
Permalink

What is the Method of Moments (MoM)?

medium
  • Set sample moments equal to theoretical moments and solve for parameters.
  • Example: Gamma(α, β) with mean α/β and variance α/β2{\alpha} / {\beta}^{2} → set sample mean = α/β and sample var  =  α/β2\mathrm{var}\; = \;{\alpha} / {\beta}^{2}, solve for α̂ and β̂.
  • Simple and consistent, but often less efficient than MLE.
  • Modern use: warm-start MLE; robust starting points; GMM (generalized method of moments) in econometrics.
Permalink

MAP vs MLE — key difference.

medium
  • MLE: θ̂ = argmax p(D  ∣  θ)p(D\; \mid \;{\theta}).
  • MAP: θ̂ = argmax p(θ  ∣  D)  =  argmax⁡p({\theta}\; \mid \;D)\; = \;\operatorname{argmax} p(D  ∣  θ)  ⋅  p(θ)p(D\; \mid \;{\theta})\; \cdot \;p({\theta}).
  • MAP adds a prior — reduces to MLE with uniform prior.
  • In log space, MAP = MLE + log-prior term → acts as regularization.
  • L2 regularization = Gaussian prior; L1 = Laplace prior.
  • Not fully Bayesian: gives a point estimate, not the full posterior.
Permalink

Explain the EM algorithm.

hard
  • For MLE with latent variables (Z): (E) compute q(Z)  =  p(Z  ∣  X,  θold)q(Z)\; = \;p(Z\; \mid \;X, \;{\theta}_{\mathrm{old}}); (M) update θnew  =  argmax⁡{\theta}_{\mathrm{new}}\; = \;\operatorname{argmax}_θ Eq[log⁡  p(X,  Z  ∣  θ)]E_{q}[\operatorname{log}\;p(X, \;Z\; \mid \;{\theta})].
  • Guaranteed to increase the log-likelihood at each iteration, converges to a local max.
  • Uses: Gaussian mixture models (soft clustering), HMMs (Baum-Welch), missing data imputation.
  • Sensitive to init → try multiple starts.
Permalink

Standard error vs standard deviation — the difference.

easy
  • SD: spread of the data (√variance) — describes the population.
  • SE: spread of a statistic across hypothetical repeated samples — describes the estimator's precision.
  • Rule for the mean: SE⁡(X)  =  SD\operatorname{SE}(X)\; = \;\mathrm{SD} / √n.
  • As n grows, SD stays roughly the same, SE shrinks — that's why bigger samples give tighter estimates without changing what the population looks like.
Permalink

What is the delta method?

hard
  • For a differentiable function g and asymptotically normal θ̂: g(θ̂) ~ N(g(θ),  g(θ)2  ⋅  Var⁡(θ))N(g({\theta}), \;g({\theta})^{2}\; \cdot \;\operatorname{Var}({\theta})) approximately.
  • Use to derive SE of transformations (ratios, log-odds).
  • Example: Var⁡(log⁡(p))\operatorname{Var}(\operatorname{log}(p)) ≈ Var⁡(p)  /  p\operatorname{Var}(p)\; / \;p̂².
  • Central for A/B testing ratio metrics (CTR per user, revenue per user with per-user denominator).
  • Fast alternative to bootstrap when you have a closed-form derivative.
Permalink

What is the jackknife?

hard
  • Leave-one-out precursor of the bootstrap.
  • For each i, compute θ̂_(-i) = statistic on data without observation i.
  • Jackknife SE = √((n-1)/n * Σ (θ(−i)  −  θ)2({\theta}( - i)\; - \;{\theta})^{2}).
  • Bias correction: n * θ̂ - (n-1) * θ̄.
  • Cheaper than bootstrap (n evaluations vs B ≈ 1000), but less general — works only for smooth statistics (fails for medians, quantiles).
  • Foundation of influence-function-based robust statistics.
Permalink

What is a conjugate prior?

hard
  • A prior p(θ) is conjugate to a likelihood p(x  ∣  θ)p(x\; \mid \;{\theta}) if the posterior p(θ  ∣  x)p({\theta}\; \mid \;x) belongs to the same family.
  • Examples: Beta prior + Bernoulli likelihood → Beta posterior; Gamma prior + Poisson → Gamma; Normal prior + Normal (known σ) → Normal; Dirichlet + categorical → Dirichlet.
  • Enables closed-form Bayesian updates — cornerstone of Thompson sampling, online Bayesian inference, contextual bandits.
Permalink

Beta-Bernoulli conjugate update — derive.

medium
  • Prior: p ~ Beta(α, β).
  • Data: n Bernoulli trials with k successes.
  • Posterior: p  ∣  datap\; \mid \;\mathrm{data} ~ Beta(α + k, β + n - k).
  • Posterior mean = (α + k) / (α + β + n) → smoothly interpolates between prior mean α/(α+β) and MLE k/n as n grows.
  • Uses: Thompson sampling for binary bandits, small-sample click-through rate estimation, MAP with a fair-coin prior Beta(1, 1) = Uniform.
Permalink

What is an uninformative prior?

hard
  • Prior meant to encode minimal prior belief.
  • Options: (1) Flat / uniform (Beta(1,1)) — feels natural but is not invariant to reparameterization.
  • (2) Jeffreys prior — proportional to √det⁡(I(θ))\operatorname{det}(I({\theta})); invariant to reparameterization (Beta(0.5, 0.5) for Bernoulli).
  • (3) Reference prior (Bernardo).
  • For Bayesian A/B testing with weak prior belief, Jeffreys is a defensible default; results converge to MLE as n grows.
Permalink

What is Bayesian shrinkage?

hard
  • Posterior estimates are pulled ('shrunk') from noisy per-group MLEs toward the prior / group mean.
  • Effect: reduces variance at the cost of small bias — Stein's paradox shows shrinkage dominates MLE in ≥ 3 dimensions.
  • Uses: multi-arm bandits, small-sample rate estimation (individual player batting averages, per-store conversion), empirical Bayes hierarchical models.
  • Frequentist equivalents: ridge regression, James-Stein estimator, empirical Bayes.
Permalink

What is empirical Bayes?

hard
  • Estimate the hyperparameters of the prior from the data itself (usually by marginal maximum likelihood), then use that prior for the posterior.
  • Cheap approximation to a full hierarchical Bayesian model — avoids MCMC on hyperpriors.
  • Uses: shrinkage for large-scale multiple testing (Efron's LFDR), gene expression, hierarchical rate estimation.
  • Weakness: ignores uncertainty in the hyperparameter → slightly narrower posteriors than full Bayes.
Permalink

How does Thompson sampling work?

medium
  • For each arm, maintain posterior over its reward.
  • To choose an action, draw θj{\theta}_{j} ~ p(θj  ∣  data)p({\theta}_{j}\; \mid \;\mathrm{data}) for each arm, pick arm with highest sampled θj{\theta}_{j}.
  • Randomness is 'baked in' via sampling → automatic exploration-exploitation tradeoff.
  • Optimal frequentist regret bounds (Bayesian and worst-case).
  • Standard in online experimentation, recommender systems, contextual bandits.
  • Bayes / Beta-Bernoulli conjugate makes update O(1) per pull.
Permalink

Why do conjugate priors matter in production?

medium
  • Closed-form posterior updates → no MCMC needed, O(1) update per new observation → scales to streaming / online settings.
  • Examples: (1) Beta-Bernoulli for CTR bandits, (2) Gamma-Poisson for arrival-rate estimation, (3) Normal-Normal for personalization.
  • Even when the model isn't strictly conjugate, industry teams often prefer a well-tuned conjugate approximation to full MCMC for latency reasons.
Permalink

What does a 95% confidence interval actually assert?

medium
  • That the procedure generating it captures the true parameter in 95% of hypothetical repetitions of the study.
  • The confidence is a property of the method across repetitions, not of the particular interval you computed, which either contains the parameter or does not.
  • This is why saying there is a 95% probability that the parameter lies inside a specific interval is a frequentist error, even though it is the interpretation everyone wants; a Bayesian credible interval is the object that supports that statement, and it requires a prior.
  • In practice the useful reading is about precision: a narrow interval near zero says the effect is small, which is informative, while a wide interval crossing zero says the study could not distinguish a meaningful effect from nothing, which is not the same as evidence of no effect.
Permalink

When is a Bayesian analysis genuinely worth the extra effort?

medium
  • When you have real prior information, when the sample is small, or when you need the probability statement itself.
  • With little data, a weakly informative prior stabilizes estimates that maximum likelihood pushes to absurd values, which is why hierarchical models are standard for many small groups such as per-store or per-user effects: partial pooling shrinks noisy groups toward the overall mean by exactly as much as the data warrants.
  • It is also the right frame when a decision needs the probability that an effect exceeds a threshold, since that is a statement about the parameter and only a posterior supports it.
  • Where it is not worth it is a large-sample, well-identified problem with a single parameter, because the posterior will match the likelihood and you will have added computational and communication cost for no inferential gain.
Permalink
10

Confidence intervals & bootstrap

8 concepts

95% CI for a mean — formula and interpretation.

easy
  • CI = X̄ ± tn−1,  0.975  ⋅  (s  /  n)t_{n - 1, \;0.975}\; \cdot \;(s\; / \; \sqrt n).
  • Interpretation: if we repeated the experiment many times, 95% of the CIs would contain the true μ.
  • NOT: 'there's a 95% chance μ is in this specific interval' (that's Bayesian credible interval language).
  • For large n, t ≈ z = 1.96.
  • Foundational for A/B testing and any point estimate reporting.
Permalink

What CI methods exist for a proportion?

medium
  • Wald (naive): p̂ ± z * √(p̂(1-p̂)/n) — bad when p̂ is near 0/1 or n small (can even exceed [0,1]).
  • Wilson score: preferred default; more accurate coverage.
  • Clopper-Pearson (exact): guaranteed conservative coverage but wider.
  • Agresti-Coull: quick fix (add 2 successes + 2 failures).
  • Modern practice: use Wilson or Clopper-Pearson, avoid Wald except for very large n and mid-range p.
Permalink

Bootstrap CI methods — percentile vs BCa vs studentized.

hard
  • Percentile: use 2.5th and 97.5th quantiles of bootstrap distribution — simple but biased when the sampling distribution is skewed.
  • BCa (bias-corrected & accelerated): corrects bias and skewness → generally best default.
  • Studentized (bootstrap-t): asymptotically most accurate but requires an SE estimate per bootstrap sample.
  • Basic (pivotal): 2*θ̂ - Q2.5Q_{2.5}, 2*θ̂ - Q97.5Q_{97.5}.
  • Rule: default BCa; percentile only if you know the sampling distribution is symmetric.
Permalink

Why do you need block bootstrap for time series?

hard
  • Standard bootstrap resamples individual points, destroying temporal dependence and grossly under-estimating variance for autocorrelated data.
  • Block bootstrap resamples contiguous blocks (moving or stationary) to preserve short-range dependence.
  • Block length L chosen so autocorrelation dies within L (rule  of  thumb  n(1/3))(\mathrm{rule}\;\mathrm{of}\;\mathrm{thumb}\;n(1 / 3)).
  • Foundational for CIs in time-series metrics, hedge-fund performance, drift detection.
Permalink

Prediction interval vs confidence interval — the difference.

medium
  • CI: uncertainty in a parameter (μ, β, p) — shrinks with n.
  • PI: uncertainty in a new individual observation Ynew  =  μ  +  εY_{\mathrm{new}}\; = \;{\mu}\; + \;{\varepsilon} — includes both parameter uncertainty AND irreducible noise σ, so wider than CI and doesn't shrink to zero (bounded by σ).
  • Practical: CI answers 'what's the average?' → tightens with n.
  • PI answers 'where will the next point land?' → bounded by irreducible variance.
Permalink

Credible interval vs confidence interval.

medium
  • CI (frequentist): 'if we repeated the experiment many times, 95% of these intervals would contain θ' — probability statement about the procedure.
  • Credible interval (Bayesian): 'given the data + prior, P(θ  ∈  [a,  b]  ∣  data)  =  95P({\theta}\; \in \;[a, \;b]\; \mid \;\mathrm{data})\; = \;95%' — probability statement about θ.
  • Numerically they can coincide with uninformative priors and large n, but they answer different questions.
  • Users usually want the credible interval interpretation.
Permalink

Why do ratio metrics need the delta method?

hard
  • Ratios like CTR = clicks / impressions have per-user numerators AND denominators → sample-averaged ratio has non-trivial variance.
  • Naive user-level SE is wrong (ignores denominator variability).
  • Delta method gives: Var⁡(N/D)\operatorname{Var}(N / D) ≈ (μN/μD)2  ⋅  [Var⁡(N)/μN2  +  Var⁡(D)/μD2  −  2⋅Cov⁡(N,D)/(μN⋅μD)]  /  n({\mu}_{N} / {\mu}_{D})^{2}\; \cdot \;[\operatorname{Var}(N) / {\mu}_{N}^{2}\; + \;\operatorname{Var}(D) / {\mu}_{D}^{2}\; - \;2 \cdot \operatorname{Cov}(N, D) / ({\mu}_{N} \cdot {\mu}_{D})]\; / \;n.
  • Alternative: bootstrap on user level.
  • Every serious A/B platform (Statsig, Eppo, Facebook's own) uses this for CTR / click-through / like-through.
Permalink

When does the bootstrap give you the wrong answer?

hard
  • It fails where the statistic depends on the extreme tail of the distribution, because resampling can never produce a value larger than the largest observation, so the maximum and near-extreme quantiles are badly estimated.
  • It fails with small samples, since the empirical distribution is a poor stand-in for the population and the intervals come out too narrow.
  • It fails when observations are dependent, which is the most common real-world violation: naive resampling of time series or clustered data destroys the correlation structure and understates variance badly, and you need a block or cluster bootstrap instead.
  • It also struggles with statistics that are not smooth functions of the data, such as the median in small samples with ties.
  • The general rule is that it substitutes computation for a distributional assumption, but not for independence.
Permalink
11

Regression assumptions

25 concepts

What are the Gauss-Markov assumptions for OLS?

medium
  • (1) Linearity: E[Y  ∣  X]  =  XβE[Y\; \mid \;X]\; = \;X{\beta}.
  • (2) Strict exogeneity: E[ε  ∣  X]  =  0E[{\varepsilon}\; \mid \;X]\; = \;0 (no omitted variable bias).
  • (3) Homoscedasticity: Var⁡(ε  ∣  X)  =  σ2\operatorname{Var}({\varepsilon}\; \mid \;X)\; = \;{\sigma}^{2}.
  • (4) No autocorrelation: Cov⁡(εi,  εj)  =  0\operatorname{Cov}({\varepsilon}_{i}, \;{\varepsilon}_{j})\; = \;0 for i ≠ j.
  • (5) No perfect multicollinearity.
  • Under (1)-(5), OLS is BLUE (Best Linear Unbiased Estimator).
  • Normality of ε is NOT required for BLUE, only for exact-sample t / F inference — CLT-robust for large n.
Permalink

Heteroscedasticity — what breaks and how do you fix it?

medium
  • Var⁡(ε  ∣  X)\operatorname{Var}({\varepsilon}\; \mid \;X) depends on X.
  • OLS β̂ stays unbiased and consistent, but SEs and t-tests are wrong → invalid inference.
  • Fixes: (1) Robust ('sandwich', HC0-HC3) SEs — most common fix in econometrics.
  • (2) Weighted Least Squares with weights ∝ 1/Var⁡(ε  ∣  X)1 / \operatorname{Var}({\varepsilon}\; \mid \;X).
  • (3) Cluster-robust SEs for panel / clustered data.
  • (4) Transform the response (log for right-skewed outcomes).
  • (5) Model the variance directly (GAMLSS, HGLM).
  • Detect with Breusch-Pagan / White tests or residual-vs-fitted plots.
Permalink

How do you detect and handle multicollinearity?

medium
  • Symptoms: unstable β̂ across subsets, huge SEs, opposite-sign coefficients from univariate expectation, high pairwise correlation.
  • Diagnostics: VIF  =  1  /  (1  −  Rj2)\mathrm{VIF}\; = \;1\; / \;(1\; - \;R_{j}^{2}) — VIF > 5-10 is concerning.
  • Condition number of X'X (>30 problematic).
  • Fixes: drop redundant features, PCA / partial-least-squares, ridge regression (shrinks coefficients), domain-based feature engineering.
  • Doesn't hurt PREDICTION accuracy on new data much — mainly wrecks coefficient interpretability and inference.
Permalink

R2R^{2} vs adjusted R2R^{2} — when do you use each?

easy
  • R2  =  1  −  RSS/TSSR^{2}\; = \;1\; - \;\mathrm{RSS} / \mathrm{TSS}: fraction of variance explained; always increases when you add features (even garbage ones).
  • Adjusted R2  =  1  −  (1  −  R2)  ⋅  (n−1)/(n−p−1)R^{2}\; = \;1\; - \;(1\; - \;R^{2})\; \cdot \;(n - 1) / (n - p - 1): penalizes model complexity; only increases if the new feature improves fit beyond noise.
  • Use adjusted R2R^{2} to compare models with different numbers of features.
  • For prediction focus, use AIC / BIC / cross-validated R2R^{2}.
  • Warning: neither is a substitute for holdout evaluation.
Permalink

Cook's distance, leverage, DFBETAS — define.

hard
  • Leverage hiih_{\mathrm{ii}}: diagonal of hat matrix H  =  X(XX)−1H\; = \;X(XX)^{-1}X'; measures how extreme xix_{i} is in X-space.
  • Cook's distance: how much β̂ changes if point i is removed; > 4/n is a flag.
  • DFBETASi\mathrm{DFBETAS}_{i},j: change in β̂_j (in SE units) after removing i; > 2/√n flag.
  • DFFITS: change in fitted value.
  • All identify points that unduly drive the fit — investigate before dropping (often signal of a real edge case or a bug in the data pipeline).
Permalink

Standardized vs studentized residuals — the difference.

hard
  • Standardized: ri  /  (s  ⋅  (1  −  hii))r_{i}\; / \;(s\; \cdot \; \sqrt (1\; - \;h_{\mathrm{ii}})) — divides raw residual by its SE.
  • Studentized (internally): same but using s (uses point i to estimate σ).
  • Externally studentized: uses s_(-i) (deletes point i from σ estimate) → follows tn−p−1t_{n - p - 1} exactly, standard for outlier tests.
  • Rule: |externally studentized| > 3 usually flags an outlier.
  • Any regression diagnostics package reports both.
Permalink

Logistic regression: why MLE and not OLS?

medium
  • Y ∈ {0, 1} → OLS predicts outside [0, 1] and residuals are heteroscedastic (Var = p(1-p)).
  • Logistic: p = σ(x'β), maximize Σ yiy_{i} log pi  +  (1  −  yi)p_{i}\; + \;(1\; - \;y_{i}) log⁡(1  −  pi)\operatorname{log}(1\; - \;p_{i}).
  • No closed form → IRLS or gradient descent.
  • Softmax = generalization to k classes.
  • Interpretation: exp⁡(βj)  =  odds\operatorname{exp}({\beta}_{j})\; = \;\mathrm{odds} ratio for a unit change in xjx_{j}.
  • Foundation of most tabular binary classifiers and the top layer of neural nets.
Permalink

Odds ratio — interpret and derive.

medium
  • Odds = p / (1 - p).
  • Odds ratio (OR)  =  oddsA  /  oddsB(\mathrm{OR})\; = \;\mathrm{odds}_{A}\; / \;\mathrm{odds}_{B}.
  • In logistic regression, log⁡(odds)  =  x\operatorname{log}(\mathrm{odds})\; = \;x'β → exp⁡(βj)\operatorname{exp}({\beta}_{j}) is the multiplicative change in odds for a 1-unit increase in xjx_{j} (holding others fixed).
  • OR > 1: positive association; OR = 1: no association; OR < 1: negative.
  • Not a ratio of probabilities.
  • For rare outcomes, OR ≈ relative risk; for common outcomes, OR overstates RR.
Permalink

GLM: link function and exponential family — connection.

hard
  • GLM: Y ~ exponential family with mean μ  =  E[Y  ∣  X]{\mu}\; = \;E[Y\; \mid \;X], and link g(μ) = x'β.
  • Canonical links: Gaussian → identity, Bernoulli → logit, Poisson → log, Gamma → inverse.
  • Fit by IRLS (iteratively re-weighted least squares).
  • Advantages over OLS: handles counts, proportions, positive-only outcomes; correct variance structure baked in.
  • Foundation of Poisson regression, logistic regression, gamma regression.
Permalink

Poisson regression: setup and pitfalls.

hard
  • Y ~ Poisson(λ), log λ = x'β.
  • Mean = variance = λ (equidispersion).
  • Uses: count outcomes (clicks, visits, defects).
  • Common pitfall: real data usually over-disperses (Var > Mean).
  • Fix: (1) negative binomial regression (extra dispersion parameter), (2) quasi-Poisson (inflates SEs by dispersion factor).
  • For zero-heavy counts, use zero-inflated Poisson / negative binomial or hurdle models.
  • Always check dispersion after fitting.
Permalink

Negative binomial regression — why use it?

hard
  • Adds dispersion parameter θ: Var⁡  =  μ  +  μ2/θ\operatorname{Var}\; = \;{\mu}\; + \;{\mu}^{2} / {\theta}.
  • Handles over-dispersion (Var > Mean) which is the norm for real counts (customer visits, defect counts, rare events).
  • Fit by MLE.
  • When θ → ∞ → reduces to Poisson.
  • Interpretation of exp⁡(β)\operatorname{exp}({\beta}) same as Poisson (multiplicative rate ratio).
  • Modern default for count regression in industry.
Permalink

What is quasi-likelihood?

hard
  • Rather than a full parametric distribution, specify only a mean-variance relationship: E[Y] = μ, Var⁡(Y)  =  φ  ⋅  V(μ)\operatorname{Var}(Y)\; = \;{\varphi}\; \cdot \;V({\mu}).
  • Quasi-scores solve exact same equations as MLE for the mean, but SEs use estimated dispersion φ.
  • Quasi-Poisson: V(μ) = μ, φ estimated → same β as Poisson but corrected SEs.
  • Cheap fix when the distribution isn't quite right — used extensively in industry counting problems.
Permalink

What is a Generalized Additive Model (GAM)?

hard
  • g(E[Y])  =  β0  +  f1(x1)  +  f2(x2)g(E[Y])\; = \;{\beta}_{0}\; + \;f_{1}(x_{1})\; + \;f_{2}(x_{2}) + ..., where each fjf_{j} is a smooth (usually spline).
  • Extends GLM by letting each feature enter through a non-parametric smooth.
  • Interpretable (partial dependence plot per feature), captures non-linearity without explicit interactions.
  • Fit via penalized splines (mgcv in R, pygam in Python).
  • Modern uses: interpretable ML (competes with GBDT on tabular), forecasting (Prophet is a GAM).
Permalink

Splines: knots, degrees of freedom, regularization.

hard
  • Spline = piecewise polynomial with continuity at knots.
  • Cubic splines: 3rd-degree pieces, continuous through 2nd derivative.
  • Natural spline: linear beyond boundary knots (avoids wild extrapolation).
  • Knot placement: at quantiles of X (uniform coverage).
  • Regularization: penalized splines add smoothing penalty λ * ∫ (f(x))2(f(x))^{2} dx → tune λ by REML / GCV.
  • Standard smoother in GAMs.
Permalink

How do you include and interpret interactions in regression?

medium
  • Include x1  ⋅  x2x_{1}\; \cdot \;x_{2} (product term) plus main effects.
  • Interpretation: coefficient of x1x_{1} depends on level of x2x_{2}.
  • Center predictors before interaction to avoid multicollinearity between the main effect and the product term.
  • For categorical × numeric interactions: separate slopes per category.
  • Test with joint F-test or LRT.
  • In production ML, tree-based methods learn interactions automatically — GLM interactions matter mostly for interpretation.
Permalink

What is a mixed-effects model?

hard
  • y = Xβ + Zu + ε, where β = fixed effects (population-level slopes), u = random effects (subject / cluster deviations), u ~ N(0, G), ε ~ N(0,  σ2I)N(0, \;{\sigma}^{2}I).
  • Uses: repeated measures on subjects, hierarchical data (students in schools, users in accounts), longitudinal panels.
  • Handles within-cluster correlation → correct SEs.
  • Fit by REML or Laplace-approximated MLE (lme4, statsmodels, brms).
  • Foundation of hierarchical Bayesian modeling.
Permalink

Random intercepts vs random slopes — the difference.

hard
  • Random intercept: each cluster has its own baseline level around the global β0{\beta}_{0}.
  • Random slope: each cluster's slope for a predictor varies around the global β.
  • Test which you need with LRT or AIC.
  • Rule: if the effect of X differs meaningfully across clusters (users respond differently to price changes), use random slopes.
  • Random slopes typically require more clusters (~30+) to fit reliably.
Permalink

Cluster-robust standard errors — when to use?

hard
  • Observations within clusters (schools, accounts, sessions) are correlated → default OLS SEs are too small → false positives.
  • Cluster-robust ('Liang-Zeger', 'sandwich') SEs correct for within-cluster correlation.
  • Rule: cluster at the level of treatment assignment or the highest level of correlation.
  • Needs ≥ 30-50 clusters to be reliable — with few clusters, use wild cluster bootstrap.
  • Standard in econometrics / A/B testing on clusters.
Permalink

Mixed-effects vs cluster-robust SEs — which do you pick?

hard
  • Both handle cluster correlation, but differently.
  • Mixed-effects: fully specifies covariance structure → efficient if the model is correct, but misspecification biases estimates.
  • Cluster-robust SEs: model-agnostic → less efficient but robust to structure misspecification.
  • Rule of thumb: use mixed effects when the hierarchical structure is real and you want cluster-specific predictions; use cluster-robust when you just want valid SEs on the fixed effects without committing to a full model.
Permalink

In practice, how do you handle correlated features in regression?

medium
  • (1) Domain knowledge: keep the more meaningful one (age vs birthyear).
  • (2) Combine: sum, difference, or PCA / PLS.
  • (3) Ridge: L2 shrinks correlated coefficients toward each other without dropping.
  • (4) Lasso: L1 arbitrarily picks one of a correlated pair.
  • (5) Elastic net: middle ground — spreads coefficients across correlated groups.
  • (6) Feature importance-based selection.
  • Rule: if inference matters, ridge / elastic net + domain pruning; if pure prediction, tree ensembles usually don't care about collinearity.
Permalink

Durbin-Watson test — what does it test?

hard
  • Tests for first-order autocorrelation in regression residuals: DW  =  Σ(et  −  et−1)2  /  Σet2\mathrm{DW}\; = \;{\Sigma}(e_{t}\; - \;e_{t - 1})^{2}\; / \;{\Sigma}e_{t}^{2}.
  • Range ~ [0, 4].
  • DW ≈ 2: no autocorr.
  • DW < 1.5: positive autocorr (common in time series).
  • DW > 2.5: negative autocorr.
  • Only detects lag-1 correlation — use Breusch-Godfrey for higher orders.
  • When positive: fix with Cochrane-Orcutt, Prais-Winsten, GLS with AR(1) errors, or explicit ARIMA modeling.
Permalink

Two-stage least squares (2SLS) — the recipe.

hard
  • Stage 1: regress T on instrument Z (and controls X) → predicted T̂.
  • Stage 2: regress Y on T̂ (and X) → coefficient on T̂ is IV estimate.
  • Equivalent to OLS-with-instrument formula βIV  =  Cov⁡(Z,  Y)/Cov⁡(Z,  T){\beta}_{\mathrm{IV}}\; = \;\operatorname{Cov}(Z, \;Y) / \operatorname{Cov}(Z, \;T) in the simple case.
  • Use robust or cluster SEs; standard 'ivreg2' / 'AER' / linearmodels packages compute correct SEs automatically.
  • Never use naive 2-step OLS SEs from stage 2 — they're wrong.
Permalink

Your model reports R2  =  0.92R^{2}\; = \;0.92. What can go wrong with celebrating that?

medium
  • Plenty.
  • R2R^{2} rises mechanically with every predictor you add, whether or not it helps, so a high value on training data says nothing about generalization and adjusted R2R^{2} only partially compensates.
  • It is scale-free but not comparable across datasets, because it depends on the variance of the target: the same model achieves a high value on a heterogeneous population and a low one on a narrow slice.
  • On time series with a trend, a high R2R^{2} is nearly automatic and often means only that both series drift, which is why differencing before evaluating matters.
  • It also says nothing about whether the residuals are well behaved, so a value of 0.92 is compatible with obvious structure the model missed.
  • Report out-of-sample error in the units of the problem alongside it, and always look at the residual plot.
Permalink

Two predictors correlate at 0.95. Does it matter?

medium
  • It depends entirely on what you want from the model.
  • For prediction, collinearity is largely harmless: the fitted values and out-of-sample error are barely affected, because the pair jointly carries the information regardless of how the coefficient is split between them.
  • For interpretation it matters a great deal, since the coefficients become unstable, their standard errors inflate, and the split between the two is nearly arbitrary, so a sign can flip when you add a few observations.
  • Diagnose it with the variance inflation factor rather than pairwise correlation, which misses multi-way collinearity.
  • Remedies are to drop one, combine them into a single index, or use ridge regularization, which trades a little bias for a large reduction in coefficient variance and is the standard answer when you must keep both.
Permalink

You have 50,000 measurements from 200 patients. What is your sample size?

hard
  • Closer to 200 than to 50,000, because observations within a patient are correlated and each additional measurement from the same patient carries much less new information than a measurement from a new patient.
  • Treating all 50,000 as independent shrinks the standard errors dramatically and produces confidence intervals far too narrow, which is how clustered data generates confident nonsense.
  • The effective sample size depends on the intraclass correlation, and even a modest value collapses the benefit of many repeats per subject.
  • The correct treatments are a mixed effects model with a random intercept per patient, cluster-robust standard errors, or aggregating to one summary per patient and analyzing those.
  • Randomization must also happen at the patient level, otherwise the treatment is entangled with the cluster.
Permalink
12

Mixed effects & clustering

1 concept

What is a hierarchical Bayesian model?

hard
  • Multi-level model: parameters θj{\theta}_{j} vary across groups j and are themselves drawn from a shared 'population' distribution θj{\theta}_{j} ~ N(μ,  τ2)N({\mu}, \;{\tau}^{2}).
  • Partial pooling: group estimates borrow strength from each other via the shared prior — automatic shrinkage.
  • Uses: hospital effects, per-user models, radon by county (Gelman's classic example).
  • Solves the 'estimate 10,000 individual users' problem: no-pool overfits, full-pool ignores individuality, partial-pool balances them.
Permalink
13

Bayesian methods

16 concepts

What is the posterior predictive distribution?

hard
  • p(ynew  ∣  D)p(y_{\mathrm{new}}\; \mid \;D) = ∫ p(ynew  ∣  θ)p(y_{\mathrm{new}}\; \mid \;{\theta}) p(θ  ∣  D)p({\theta}\; \mid \;D) dθ.
  • Predicts new data marginalizing over posterior uncertainty.
  • Unlike MLE + plug-in prediction, it accounts for parameter uncertainty → wider (better-calibrated) prediction intervals, especially with small n.
  • In practice: draw θ^(s) from posterior, sample ynewy_{\mathrm{new}} ~ p(y  ∣  θ(s))p(y\; \mid \;{\theta}(s)).
  • Foundation of Bayesian forecasting and probabilistic programming.
Permalink

What is the marginal likelihood (evidence) and why is it hard?

hard
  • p(D) = ∫ p(D  ∣  θ)p(D\; \mid \;{\theta}) p(θ) dθ.
  • Normalizing constant for the posterior.
  • High-dimensional integral → intractable in closed form for most models.
  • Bayesian model comparison: Bayes factor  =  p(D  ∣  M1)  /  p(D  ∣  M2)\mathrm{factor}\; = \;p(D\; \mid \;M_{1})\; / \;p(D\; \mid \;M_{2}).
  • Computing p(D): bridge sampling, thermodynamic integration, Chib's method, variational lower bounds (ELBO ≤ log p(D)).
  • MCMC alone doesn't give p(D) — special methods required.
Permalink

How do you interpret a Bayes factor?

hard
  • BF12  =  p(D  ∣  M1)  /  p(D  ∣  M2)\mathrm{BF}_{12}\; = \;p(D\; \mid \;M_{1})\; / \;p(D\; \mid \;M_{2}) — ratio of marginal likelihoods → posterior odds = BF × prior odds.
  • Jeffreys scale: BF > 10 strong evidence for M1M_{1}, > 100 decisive; BF < 1/10 flip.
  • Unlike p-values, symmetric — can support H0H_{0}.
  • Weakness: hypersensitive to prior choice for nuisance parameters (Lindley's paradox).
  • Alternative in practice: LOO-CV, WAIC, PSIS-LOO for predictive model comparison.
Permalink

How does Bayesian A/B testing work?

medium
  • Model: pAp_{A}, pBp_{B} ~ Beta prior; observe successes / failures → Beta posteriors.
  • Metric: P(pB  >  pA  ∣  data)P(p_{B}\; > \;p_{A}\; \mid \;\mathrm{data}) via Monte Carlo (sample from both posteriors, count fraction).
  • Also useful: posterior of the lift (pB  −  pA)(p_{B}\; - \;p_{A}) or of the relative uplift.
  • Advantages: interpretable ('72% chance B is better') and safe under peeking (no p-hacking inflation, decisions optimal under a loss function).
  • Standard in growth teams alongside frequentist.
Permalink

How does Metropolis-Hastings work?

hard
  • MCMC: build a Markov chain whose stationary distribution is the posterior.
  • Given current θ, propose θ' ~ q(θ  ∣  θ)q({\theta}\; \mid \;{\theta}).
  • Accept with probability α  =  min⁡(1,  [p(θ  ∣  D)  q(θ  ∣  θ)]  /  [p(θ  ∣  D)  q(θ  ∣  θ)]){\alpha}\; = \;\operatorname{min}(1, \;[p({\theta}\; \mid \;D)\;q({\theta}\; \mid \;{\theta})]\; / \;[p({\theta}\; \mid \;D)\;q({\theta}\; \mid \;{\theta})]).
  • Symmetric q simplifies to α  =  min⁡(1,  p(θ  ∣  D)  /  p(θ  ∣  D)){\alpha}\; = \;\operatorname{min}(1, \;p({\theta}\; \mid \;D)\; / \;p({\theta}\; \mid \;D)).
  • Only needs the posterior up to a constant → dodges intractable evidence.
  • Weaknesses: slow mixing in high dims; sensitive to proposal scale.
Permalink

What is Gibbs sampling?

hard
  • Special MCMC: sample each parameter (or block) from its full conditional p(θj  ∣  θ−j,  D)p({\theta}_{j}\; \mid \;{\theta}_{- j}, \;D), one at a time.
  • Guaranteed acceptance (α = 1) when full conditionals are known.
  • Works well for conjugate hierarchical models, Bayesian networks, LDA.
  • Weak in strongly correlated posteriors — slow mixing.
  • Modern alternatives (HMC, NUTS) are usually better default, but Gibbs is still standard when conjugacy makes full conditionals trivial.
Permalink

Hamiltonian Monte Carlo — intuition.

hard
  • Uses gradients of log-posterior to propose long, informed steps → efficient in high dimensions.
  • Introduces auxiliary momentum p, runs Hamiltonian dynamics (leapfrog) to propose new (θ, p), MH-accepts.
  • Advantages: dramatically better mixing than random-walk MH in high dimensions.
  • NUTS (No-U-Turn Sampler): auto-tunes trajectory length; default in Stan, PyMC, NumPyro.
  • Requires differentiable posterior — great for continuous models, doesn't handle discrete latents directly.
Permalink

How do you diagnose MCMC convergence?

hard
  • (1) R̂ (Gelman-Rubin) < 1.01 across multiple chains → chains agree.
  • (2) ESS (effective sample size) ≥ 400 per parameter.
  • (3) Trace plots: should look like fuzzy caterpillars, no drift or stickiness.
  • (4) Autocorrelation plots decay quickly.
  • (5) Divergences in NUTS: indicate hard posterior geometry — reparameterize (non-centered) or increase adaptδ\mathrm{adapt}_{\delta}.
  • Never trust a single chain's samples without these checks.
Permalink

Burn-in and thinning — why?

medium
  • Burn-in: discard initial iterations before the chain reaches stationarity.
  • Typical: 10-50% of chain.
  • Thinning: keep every k-th sample to reduce autocorrelation.
  • Modern view: thinning wastes information unless memory is tight — better to use all samples with autocorrelation-aware summaries.
  • Use ESS-based decisions rather than fixed burn-in / thinning rules.
  • NUTS + adaptive warmup replaces manual burn-in in practice.
Permalink

Variational inference vs MCMC.

hard
  • VI: approximate posterior p(θ  ∣  D)p({\theta}\; \mid \;D) with a simpler family q_φ(θ) (e.g. mean-field Gaussian) by maximizing ELBO  =  Eq[log⁡  p(D,  θ)]  −  Eq[log⁡  q(θ)]\mathrm{ELBO}\; = \;E_{q}[\operatorname{log}\;p(D, \;{\theta})]\; - \;E_{q}[\operatorname{log}\;q({\theta})].
  • Advantages: much faster, scales to big data (SVI, mini-batches), differentiable → fits on GPU.
  • Weaknesses: biased (limited by family); mean-field VI under-covers uncertainty and drops correlations.
  • Use for large-scale problems where speed > accuracy.
  • Modern: normalizing flows for richer q.
Permalink

Derive the ELBO for VI.

hard
  • log p(D) = log ∫ p(D, θ) dθ = log ∫ q(θ) * p(D, θ)/q(θ) dθ ≥ Eq[log⁡  p(D,  θ)/q(θ)]E_{q}[\operatorname{log}\;p(D, \;{\theta}) / q({\theta})] (Jensen).
  • ELBO = E_q[log p(D, θ)] - E_q[log q(θ)] = E_q[log p(D | θ)] - KL(q || p(θ)). log p(D)  −  ELBO  =  KL⁡(q  ∣∣  p(θ  ∣  D))p(D)\; - \;\mathrm{ELBO}\; = \;\operatorname{KL}(q\; \mid \mid \;p({\theta}\; \mid \;D)) ≥ 0 → maximizing ELBO minimizes KL to the posterior.
  • Same objective drives VAE training in deep learning.
Permalink

Informative vs non-informative priors — the tradeoff.

medium
  • Non-informative (flat, Jeffreys): 'let the data speak' — safe when you have plenty of data, but can be worse than an informative prior at small n.
  • Informative (based on historical data, domain knowledge, similar experiments): dramatically improves inference in small-sample settings, but can also inject bias if wrong.
  • Rule: (1) when n is large, prior barely matters.
  • (2) When n is small, informative priors are your friend if you have credible data.
Permalink

WAIC and LOO-CV — Bayesian model comparison.

hard
  • Both estimate out-of-sample predictive performance.
  • WAIC = -2  ⋅  (lppd  −  pWAIC)2\; \cdot \;(\mathrm{lppd}\; - \;p_{\mathrm{WAIC}}) where lppd = expected log posterior predictive on training data, pWAIC  =  effectivep_{\mathrm{WAIC}}\; = \;\mathrm{effective} number of parameters via posterior variance.
  • LOO-CV: exact leave-one-out log predictive; approximated efficiently via PSIS-LOO (Vehtari et al.) using importance sampling — modern default in Stan / PyMC / Arviz.
  • Both replace AIC / BIC / Bayes factor for model comparison in most cases.
Permalink

What is a prior predictive check?

hard
  • Simulate data from the prior alone: θ ~ p(θ), y ~ p(y  ∣  θ)p(y\; \mid \;{\theta}).
  • Check whether generated y is plausible given domain knowledge (e.g. simulated coin flip probabilities in [0, 1]; simulated log-revenues aren't in the trillions).
  • Catches absurd priors before running expensive inference.
  • Modern Bayesian workflow (Gelman et al.): prior predictive → fit → posterior predictive → repeat.
  • Essential in real modeling.
Permalink

Posterior predictive check — how do you use it?

hard
  • Simulate replicated datasets yrepy_{\mathrm{rep}} from posterior predictive; compare their summary statistics (mean, quantiles, distribution shape) to observed data.
  • If observed data looks like a plausible draw from yrepy_{\mathrm{rep}}, the model captures the data — otherwise, refine.
  • Formalized via Bayesian p-values: fraction of yrepy_{\mathrm{rep}} with statistic ≥ observed.
  • Standard diagnostic step in ArviZ, brms, Stan.
Permalink

When should you use a bandit instead of an A/B test?

hard
  • Bandit (Thompson sampling / UCB) automatically shifts traffic toward the winning arm during the test → minimizes regret.
  • Use when: (1) short-lived items (news, promotions) where you can't afford to send 50% traffic to the loser; (2) many arms (>10) with fast feedback; (3) time-decaying decisions.
  • Don't use when: (1) you need clean inference on effect size / statistical significance for stakeholder communication; (2) delayed feedback; (3) SUTVA-violating settings.
  • Bandits are for optimization, A/B is for learning.
Permalink
14

Causal inference basics

24 concepts

Why doesn't correlation imply causation?

medium
  • Two variables can move together because A causes B, B causes A, both are caused by a third variable Z (confounder), or by pure coincidence.
  • Establishing causation requires either a randomized experiment (breaks confounding by design) or careful causal inference methods (DAGs, matching, instrumental variables, difference-in-differences, regression discontinuity, do-calculus).
Permalink

What is Simpson's paradox?

hard
  • A trend that appears in several subgroups can reverse when the groups are combined.
  • Classic example: a treatment looks better in each subgroup but worse overall, because group sizes and baseline rates vary.
  • It's a warning to always inspect subgroup patterns and think causally — the right decomposition depends on the confounding structure, not on which cut looks prettier.
Permalink

Users who adopt feature X churn less. Can you say the feature reduces churn?

easy
  • Not from that association alone, because four mechanisms produce it without the feature doing anything: (1) confounding — engaged users both adopt features and stay, so engagement causes both, (2) reverse causation — users who were already going to stay are the ones who explore features, (3) selection — the population you measured was filtered in a way that creates the link, (4) coincidence, which matters when the sample is small.
  • To make the causal claim you need randomization, meaning an experiment that offers the feature to a random subset, or a credible identification strategy: instrumental variables, regression discontinuity, difference-in-differences, or matching under an explicitly stated ignorability assumption.
  • Some version of this question appears in every statistics interview.
Permalink

What is SUTVA?

hard
  • Stable Unit Treatment Value Assumption: (1) no interference between units — one unit's treatment doesn't affect another's outcome (breaks in networks, marketplaces, viral products), (2) no hidden treatment variants — one 'treatment' is well-defined.
  • Violated in social networks (spillovers), two-sided marketplaces (Uber: driver-side changes affect rider outcomes), auctions (competition), any system with global equilibrium effects.
  • Fixes: cluster / market-level randomization, switchback experiments.
Permalink

How do you handle confounders in observational studies?

medium
  • (1) Include them in regression (X  in  E[Y  ∣  T,  X])(X\;\mathrm{in}\;E[Y\; \mid \;T, \;X]) if measured — assumes correct functional form.
  • (2) Matching / propensity score matching — trims to overlap region.
  • (3) Inverse probability weighting — reweight sample so treated and control look similar.
  • (4) Doubly robust: regression + IPW (consistent if either is correct).
  • (5) Instrumental variables — bypass unobserved confounders.
  • Unmeasured confounders are the fatal weakness — that's why RCTs are the gold standard.
Permalink

An aggregate trend reverses once you stratify. Which conditioning set is correct?

medium
  • An association observed at the aggregate level reverses when data is split by a confounder.
  • Classic example: Berkeley admissions — men admitted at higher aggregate rate but women admitted at higher rate within every department (women applied to more competitive departments).
  • Resolution: pick the right conditioning set based on the causal DAG.
  • Blindly conditioning or ignoring a confounder both cause errors.
  • Textbook case for why 'always condition on more variables' is wrong.
Permalink

What is a collider and why is conditioning on it bad?

hard
  • In a DAG A → C ← B: C is a collider (both A and B point into it).
  • A and B are marginally independent, but conditioning on C creates a spurious association.
  • Classic example: 'talent and beauty are anticorrelated among Hollywood stars, because being famous (a collider caused by both) has been implicitly selected on'.
  • Same mechanism drives selection bias, sample-conditioning artifacts, restaurant-only surveys (studies that condition on 'made it to production' etc.).
Permalink

What is Pearl's backdoor criterion?

hard
  • To identify P(Y  ∣  do(T))P(Y\; \mid \;\mathrm{do}(T)) from observational data, find a set of variables Z such that: (1) Z blocks all backdoor paths from T to Y (paths starting with an arrow into T), (2) Z contains no descendant of T.
  • Then P(Y  ∣  do(T))  =  ΣzP(Y\; \mid \;\mathrm{do}(T))\; = \;{\Sigma}_{z} P(Y  ∣  T,  Z=z)P(Y\; \mid \;T, \;Z = z) P(Z=z).
  • Effectively: condition on all confounders, don't condition on mediators or colliders.
  • Foundation of causal DAG-based analysis.
Permalink

When is the frontdoor criterion useful?

hard
  • When unmeasured confounders make backdoor impossible, but a fully-mediating variable M exists.
  • E.g.
  • T → M → Y where all T-Y effect goes through M and no confounder acts on M.
  • Frontdoor: P(Y  ∣  do(T))  =  ΣmP(Y\; \mid \;\mathrm{do}(T))\; = \;{\Sigma}_{m} P(m  ∣  T)  ⋅  ΣtP(m\; \mid \;T)\; \cdot \;{\Sigma}_{t}' P(Y  ∣  m,  t)P(Y\; \mid \;m, \;t) P(t').
  • Classic example: smoking → tar → cancer, if we could measure tar but not the unobserved 'smoking-gene' confounder.
  • Rarely applicable in practice but a landmark result in causal inference theory.
Permalink

What is a propensity score?

hard
  • e(x)  =  P(T  =  1  ∣  X  =  x)e(x)\; = \;P(T\; = \;1\; \mid \;X\; = \;x).
  • Rosenbaum-Rubin: under unconfoundedness, adjusting for e(x) alone is sufficient (dimension  reduction  from  ∣X∣  to  1)(\mathrm{dimension}\;\mathrm{reduction}\;\mathrm{from}\; \mid X \mid \;\mathrm{to}\;1).
  • Uses: (1) matching on e(x), (2) IPW: weight treated by 1/e(x), control by 1/(1-e(x)) → weighted average = ATE.
  • (3) Stratification by e(x) quintiles.
  • Estimated with logistic regression or ML.
  • Never use predicted probabilities of 0 or 1 (violates overlap → wild variance).
Permalink

Inverse Probability Weighting — how does it work?

hard
  • Weight each treated unit by 1/e(x) and each control by 1/(1-e(x)) → weighted population is balanced on X.
  • Marginal ATE estimator: Σ (Ti  Yi  /  ei)  −  Σ(T_{i}\;Y_{i}\; / \;e_{i})\; - \;{\Sigma} ((1−Ti)  Yi  /  (1−ei))  /  n((1 - T_{i})\;Y_{i}\; / \;(1 - e_{i}))\; / \;n.
  • Consistent under unconfoundedness + overlap + correct e(x) model.
  • Weakness: high-variance when e(x) near 0 or 1 → truncate weights or use stabilized IPW / doubly robust estimators.
  • Standard in observational causal inference.
Permalink

Why is a doubly-robust estimator useful?

hard
  • Combines outcome model μ̂(x) and propensity model ê(x).
  • Formula: ATE = E[μ̂(1, X) - μ̂(0, X) + T(Y - μ̂(1, X))/ê(x) - (1-T)(Y - μ̂(0, X))/(1-ê(x))].
  • Consistent if EITHER μ̂ OR ê(x) is correctly specified — hence 'doubly robust'.
  • AIPW, TMLE are standard implementations.
  • Modern default in observational causal inference; also basis of DR-learners / DML with ML nuisance models.
Permalink

What makes a valid instrumental variable?

hard
  • Z is a valid IV for T → Y if: (1) Relevance: Z affects T (Cov(Z, T) ≠ 0).
  • (2) Exclusion: Z affects Y only through T (no direct Z → Y).
  • (3) Exogeneity: Z ⊥ unobserved confounders.
  • Estimand: LATE (Local Average Treatment Effect) = Cov(Z, Y) / Cov(Z, T) via 2SLS.
  • Classic examples: judge leniency, distance to hospital, weather as an instrument for economic activity.
  • Weak instruments (small Cov(Z, T)) → biased 2SLS with wide CIs.
Permalink

Difference-in-differences — setup and assumption.

hard
  • Compare change in outcome for treated group vs change for control group before and after intervention.
  • Estimand: (Ytreated,post  −  Ytreated,pre)  −  (Ycontrol,post  −  Ycontrol,pre)(Y_{\mathrm{treated}}, \mathrm{post}\; - \;Y_{\mathrm{treated}}, \mathrm{pre})\; - \;(Y_{\mathrm{control}}, \mathrm{post}\; - \;Y_{\mathrm{control}}, \mathrm{pre}).
  • Runs as OLS with unit + time fixed effects and an interaction Treated × Post.
  • Key assumption: parallel trends — without treatment, both groups would have evolved similarly.
  • Verify with pre-trend tests, event studies, robustness checks.
  • Foundational in policy analysis and observational A/B.
Permalink

How do you defend the parallel-trends assumption?

hard
  • (1) Plot pre-treatment trends of treated and control — visually parallel?
  • (2) Event study: interact treatment indicator with lead / lag indicators; pre-treatment lead coefficients ≈ 0.
  • (3) Placebo tests: pretend treatment happened at an earlier fake date — no effect should appear.
  • (4) Multiple control groups.
  • (5) Synthetic control if only one treated unit.
  • Rule: never present DID without explicit pre-trend evidence.
Permalink

Regression discontinuity design — how does it work?

hard
  • Treatment assigned by a threshold on a continuous running variable X (e.g. score ≥ 60 → scholarship).
  • Compare Y just above vs just below cutoff — near cutoff, individuals are 'as-if random'.
  • Sharp RDD: treatment deterministic at cutoff.
  • Fuzzy RDD: probability jumps but not to 1 — combine with IV.
  • Local linear regression around cutoff + optimal bandwidth (Imbens-Kalyanaraman).
  • Foundational in policy evaluation, admission thresholds, credit-score cutoffs.
Permalink

What is synthetic control?

hard
  • Construct a weighted combination of untreated units that reproduces the pre-treatment trajectory of the treated unit.
  • Weights ≥ 0, sum to 1, chosen to minimize pre-treatment fit error on X and Y.
  • Post-treatment gap between treated and synthetic = causal effect estimate.
  • Uses: single-treated-unit policy analysis (California smoking ban, German reunification).
  • Modern extensions: generalized synthetic control, augmented synthetic control (Ben-Michael), matrix completion methods.
Permalink

Mediation analysis — direct vs indirect effects.

hard
  • Path: T → M → Y (with possible direct T → Y).
  • Decomposition: total effect = direct effect (T → Y not through M) + indirect effect (T → M → Y).
  • Baron-Kenny style regression coefficients or, better, potential-outcomes-based Natural Direct / Indirect Effects (Pearl, Robins).
  • Identification requires no unmeasured T-M, T-Y, or M-Y confounders.
  • Key application: understanding causal mechanism, not just total effect.
Permalink

How do you estimate heterogeneous treatment effects (HTE)?

hard
  • CATE(x)  =  E[Y(1)  −  Y(0)  ∣  X  =  x]\mathrm{CATE}(x)\; = \;E[Y(1)\; - \;Y(0)\; \mid \;X\; = \;x] varies with covariates.
  • Methods: (1) interaction terms in regression (Y ~ T + T×X); (2) causal forests / GRF (Wager & Athey); (3) meta-learners: T-learner (separate models), S-learner (single joint), X-learner (better in imbalanced treatment), R-learner; (4) doubly-robust CATE.
  • Uses: personalized recommendations, targeted policies (Uplift modeling in marketing).
  • Requires overlap + unconfoundedness (or RCT).
Permalink

How does Simpson's paradox strike A/B tests?

hard
  • Test wins overall but loses in every user segment (or vice versa) → aggregate confounded by mid-experiment composition change.
  • Common causes: (1) SRM by segment (imbalanced assignment across segments over time), (2) traffic mix drift while experiment runs, (3) new-vs-returning composition shift.
  • Fix: (1) fix SRM; (2) analyze weighted by pre-experiment segment shares (post-stratification); (3) run segment-level analyses; (4) if segment-level results all point the same way but opposite to aggregate → trust the segments.
Permalink

How do you test in a marketplace / network with SUTVA violations?

hard
  • (1) Cluster randomization (randomize at community / graph-community level so spillovers stay within cluster).
  • (2) Switchback experiments (turn treatment on/off over time, randomize periods).
  • (3) Ego-cluster tests (treat user + their neighbors together).
  • (4) Two-sided experiments (rider-side + driver-side simultaneously in Uber).
  • (5) Structural models to extrapolate to full launch.
  • Standard practice at Uber, Lyft, Airbnb, DoorDash.
  • Naive user-level tests bias effect estimates.
Permalink

Quantile treatment effect (QTE) vs ATE.

hard
  • ATE captures mean effect.
  • QTE(τ)  =  FY(1)\mathrm{QTE}({\tau})\; = \;F_{Y}(1)^(-1)(τ)  −  FY(0)({\tau})\; - \;F_{Y}(0)^(-1)(τ) captures effect on the τ-th quantile — useful when the mean is misleading (heavy tails, revenue metrics dominated by top 1%).
  • Example: an A/B test with +0.01ATEbut−0.01 ATE but -1 at the median → most users hurt, a few whales lift the average.
  • Estimation: quantile regression, IPW-quantile, causal forests.
  • Standard tail-metric analysis at big tech.
Permalink

Real interview: your model performs great in A/B but flops post-launch. Why?

hard
  • Common reasons: (1) Selection bias — the A/B population is not the launch population (early adopters, engaged users).
  • (2) SUTVA violation — 50% traffic doesn't scale to 100% (marketplace saturation, ad auction dynamics).
  • (3) Novelty effect not de-biased.
  • (4) Winner's curse — effect regressed to a smaller true value.
  • (5) Metric divergence — A/B primary metric doesn't align with long-term OKR.
  • (6) Reflex reaction from competitors / operations.
  • Debug by: revisiting SUTVA, holdout at 1%, long-term surrogate, and re-measuring at launch.
Permalink

You cannot randomize. What is the strongest causal claim you can still make?

hard
  • One conditional on an assumption you state explicitly, which is the honest form of every observational causal claim.
  • Begin by drawing the assumed causal structure, because which variables to adjust for is a question about that structure, not about which improve fit; conditioning on a collider or a mediator introduces bias rather than removing it.
  • If you can argue that all confounders are measured, adjustment or matching gives an effect estimate under that assumption.
  • Stronger designs exploit structure instead: a difference-in-differences comparison if you have pre-period data and a plausible parallel trend, an instrumental variable if something shifts treatment without affecting the outcome directly, or a regression discontinuity if assignment follows a threshold.
  • Then test the assumption, with placebo outcomes and pre-trend checks, and report sensitivity to unmeasured confounding.
Permalink
15

Experimentation & A/B testing

9 concepts

What are the pillars of a solid A/B test design?

medium
  • (1) Clear primary metric + guardrail metrics chosen in advance.
  • (2) Sample size / MDE via power analysis.
  • (3) Randomization unit chosen at correct level (user, session, cluster).
  • (4) Sample ratio check (SRC): assignment counts should match target ratio.
  • (5) A/A test as sanity check on randomizer.
  • (6) Pre-registered analysis plan.
  • (7) Fixed test duration + no-peeking rule (or sequential test).
  • (8) Multiple-testing correction if many metrics.
Permalink

What is Sample Ratio Mismatch (SRM) and why check it?

medium
  • Chi-square test that observed enrollment ratio (say 50/50) matches expected ratio. p < 0.005 → strong evidence something's wrong with the randomizer (bot traffic, ID mismatch, bug in assignment logic, biased filtering).
  • Any downstream analysis with SRM is invalid — you can't trust the effect estimate.
  • Standard early-warning check in every experimentation platform (Optimizely, Statsig, Eppo).
Permalink

What are guardrail metrics?

easy
  • Secondary metrics you commit to monitor to catch bad tradeoffs: latency, error rate, retention, revenue-per-user, complaint rate, security signals.
  • Test can 'win' on the primary metric but must not degrade guardrails beyond a preset threshold.
  • Standard practice: bidirectional CI on each guardrail must not cross the harm threshold.
  • Prevents 'winning' by cannibalizing another team's metric or long-term health.
Permalink

OEC — Overall Evaluation Criterion — what is it?

hard
  • A single scalar composite of relevant metrics chosen ex ante as the experiment's success criterion.
  • Simplifies decision-making (one number).
  • Should reflect business value, be sensitive at reasonable sample sizes, and align long-term with company OKRs.
  • Examples: Bing's 'sessions per user', Netflix engagement composite.
  • Cost: hard to design well; teams often mis-weight components → dumb decisions.
  • Modern practice: use OEC + a small guardrail suite rather than OEC alone.
Permalink

Novelty vs primacy effects in long experiments.

hard
  • Novelty: users react positively to any change initially, then revert → early effect inflated, long-run effect smaller.
  • Primacy: existing users hate change initially, then adapt → early effect deflated, long-run effect larger.
  • Fixes: (1) run experiments long enough (2-4 weeks minimum) to observe stabilization; (2) segment by user tenure (new vs existing); (3) look at the trend of daily effect over time; (4) always plan for ramp-up + steady-state phases in analysis.
Permalink

Switchback experiments — when and how?

hard
  • Alternate treatment on/off over time windows across the whole market → each window randomly assigned.
  • Estimator: difference in outcomes between treated and control windows, with cluster-robust or time-block SEs.
  • Uses: full-market experiments where user-level randomization fails (surge pricing at Uber, driver dispatch at DoorDash).
  • Carryover between windows biases the estimate — use burn-in periods within each window and appropriate window length.
Permalink

What is the winner's curse in A/B testing?

hard
  • Because we only 'ship' experiments that cross the significance / effect threshold, the observed effect on the winners is a biased over-estimate of the true effect (selection on the outcome).
  • Empirical rule at Microsoft / Bing: shipped effects shrink 10-40% on re-measurement.
  • Fixes: (1) empirical Bayes shrinkage of shipped estimates; (2) look at long-run replicable effects; (3) hold-out samples to re-estimate on unbiased data.
  • Foundation of realistic ROI accounting.
Permalink

How do you measure long-term effects when A/B tests are short?

hard
  • (1) Long holdout experiment: keep 1% of traffic unchanged for months → measure long-term differential.
  • (2) Surrogate metrics: proxy predicting long-term outcomes (predicted LTV, 28-day retention as surrogate for annual retention).
  • (3) Two-stage experiment: short-term A/B + Bayesian imputation of long-term with historical priors.
  • (4) Meta-analyses across many past experiments.
  • Modern practice: combine surrogate + long holdout at big platforms.
Permalink

How does a company scale to running 1000+ experiments concurrently?

hard
  • (1) Layered assignment (users get one exposure per orthogonal layer) — Google's classic approach.
  • (2) Mutually exclusive groups when interactions are expected.
  • (3) Centralized experimentation platform (Optimizely, Statsig, Eppo, in-house Google Optimize, Meta Deltoid).
  • (4) Metric standardization + guardrails.
  • (5) FDR control across the metric-experiment matrix.
  • (6) A/A tests continuously to monitor platform health.
  • (7) Culture of pre-registered hypotheses + shipping thresholds.
  • Big-tech DS interviews probe this.
Permalink
You finished the lesson
Now test what stuck.
Interview flashcardsTake the quizBrowse other topics