41 interview questions on probability foundations, each answered in full. Free to read, no account needed.
State the Central Limit Theorem in one sentence. easy For independent, identically distributed samples with finite mean mu and variance σ 2 \sigma^{2} σ 2 N ( μ , σ 2 / n ) N(\mu, \;\sigma^{2}\; / \;n) N ( μ , σ 2 / n )
Bayesian vs frequentist — what's the core difference? medium Frequentists treat parameters as fixed unknowns and estimate them via long-run frequency properties (unbiased estimators, confidence intervals, p-values). Bayesians treat parameters as random variables with a prior distribution, update with data via Bayes' rule to get a posterior, and summarize with credible intervals. Bayesian methods handle prior knowledge naturally and give posterior probability statements directly.
State the Law of Large Numbers. easy As the sample size grows, the sample mean converges to the true expected value. Weak LLN: convergence in probability. Strong LLN: convergence almost surely. This justifies estimating expectations by averaging samples — the foundation of Monte Carlo estimation.
Independence vs uncorrelatedness — what's the difference? medium Independent means P(A, B) = P(A) P(B): no relationship at all. Uncorrelated means covariance = 0: no linear relationship. Independence implies uncorrelated, but the reverse is false in general — X and X 2 X^{2} X 2 For jointly Gaussian variables, uncorrelated implies independent — a special case.
State Bayes' theorem and one intuitive use. easy P ( A ∣ B ) = P ( B ∣ A ) P(A\; \mid \;B)\; = \;P(B\; \mid \;A) P ( A ∣ B ) = P ( B ∣ A ) Intuition: update a prior belief P(A) with new evidence B to get a posterior belief P ( A ∣ B ) P(A\; \mid \;B) P ( A ∣ B ) Classic use: medical testing with base rates. Even a highly sensitive/specific test on a rare disease produces mostly false positives if the disease prevalence is low.
State Kolmogorov's three probability axioms. easy For a sample space Ω and events A: (1) P(A) ≥ 0 (non-negativity). (2) P(Ω) = 1 (normalization — some outcome must occur). (3) For disjoint (mutually exclusive) events A 1 A_{1} A 1 A 2 A_{2} A 2 P ( A i ) = Σ P(\;A_{i})\; = \;{\Sigma} P ( A i ) = Σ P ( A i ) P(A_{i}) P ( A i ) All of probability theory — conditional probability, Bayes, expectation, independence — is derived from these three axioms.
Define conditional probability and prove Bayes' rule from it. easy P ( A ∣ B ) = P ( A B ) / P ( B ) P(A\; \mid \;B)\; = \;P(A\;\;B)\; / \;P(B) P ( A ∣ B ) = P ( A B ) / P ( B ) Symmetrically, P ( B ∣ A ) = P ( A B ) / P ( A ) P(B\; \mid \;A)\; = \;P(A\;\;B)\; / \;P(A) P ( B ∣ A ) = P ( A B ) / P ( A ) P ( A B ) = P ( B ∣ A ) ⋅ P ( A ) P(A\;\;B)\; = \;P(B \mid A)\; \cdot \;P(A) P ( A B ) = P ( B ∣ A ) ⋅ P ( A ) Substitute back: P ( A ∣ B ) = P ( B ∣ A ) ⋅ P ( A ) / P ( B ) P(A\; \mid \;B)\; = \;P(B\; \mid \;A)\; \cdot \;P(A)\; / \;P(B) P ( A ∣ B ) = P ( B ∣ A ) ⋅ P ( A ) / P ( B ) This is Bayes' rule — a direct consequence of the conditional-probability definition, not a separate axiom.
State the law of total probability. medium If {B 1 B_{1} B 1 B n B_{n} B n P ( A ∣ B i ) ⋅ P ( B i ) P(A\; \mid \;B_{i})\; \cdot \;P(B_{i}) P ( A ∣ B i ) ⋅ P ( B i ) Used constantly: decompose a complex probability by conditioning on a partitioning variable (age, disease status, region). Foundation of many probabilistic derivations, mixture models, and marginalization in Bayesian inference.
Bernoulli distribution: parameters, PMF, mean, variance. easy X ~ Bernoulli(p): single trial with success prob p. PMF: P(X=1) = p, P(X=0) = 1-p. E[X] = p; Var ( X ) = p ( 1 − p ) \operatorname{Var}(X)\; = \;p(1 - p) Var ( X ) = p ( 1 − p ) Foundation for binary outcomes (click / no click, pass / fail). Building block: Binomial(n, p) = sum of n i.i.d. Bernoulli(p).
Binomial distribution and when to use it. easy X ~ Binomial(n, p): number of successes in n independent Bernoulli(p) trials. PMF: P ( X = k ) = C ( n , k ) ⋅ p k ⋅ ( 1 − p ) P(X = k)\; = \;C(n, k)\; \cdot \;p^{k}\; \cdot \;(1 - p) P ( X = k ) = C ( n , k ) ⋅ p k ⋅ ( 1 − p ) E[X] = np; Var ( X ) = n p ( 1 − p ) \operatorname{Var}(X)\; = \;\mathrm{np}(1 - p) Var ( X ) = np ( 1 − p ) Use it for: A/B testing (successes out of n visitors), quality inspection (defects out of n items), any bounded count of independent trials. When n is large and p is small, approximate by Poisson(np).
Poisson distribution and its typical use cases. medium X ~ Poisson(λ): count of events in a fixed interval / area, when events happen at constant rate λ independently. PMF: P ( X = k ) = λ k ⋅ e P(X = k)\; = \;{\lambda}^{k}\; \cdot \;e P ( X = k ) = λ k ⋅ e E[X] = Var(X) = λ. Use cases: website clicks per minute, defects per page, insurance claims per year. Limit of Binomial(n, p) as n → ∞, p → 0, np → λ. Assumes memorylessness — real data with time-varying rate (bursty traffic) needs negative binomial or NHPP.
Geometric distribution: setup and mean. medium X ~ Geometric(p): number of trials until the first success in i.i.d. Bernoulli(p). PMF: P(X=k) = (1-p)^(k-1) * p. E[X] = 1/p; Var ( X ) = ( 1 − p ) / p 2 \operatorname{Var}(X)\; = \;(1 - p) / p^{2} Var ( X ) = ( 1 − p ) / p 2 Memoryless: P ( X > n + k ∣ X > n ) = P ( X > k ) P(X\; > \;n + k\; \mid \;X\; > \;n)\; = \;P(X\; > \;k) P ( X > n + k ∣ X > n ) = P ( X > k ) Uses: number of attempts until conversion, retries until failure. Note: two conventions — some define it as failures before first success (E = (1-p)/p).
Uniform distribution: continuous vs discrete. easy Discrete: U{a, ..., b} — each of (b-a+1) integers has probability 1/(b-a+1). Continuous: U(a, b) — density 1/(b-a) on [a,b], zero elsewhere. Continuous uniform: E[X] = (a+b)/2, Var ( X ) = ( b − a ) 2 / 12 \operatorname{Var}(X)\; = \;(b - a)^{2} / 12 Var ( X ) = ( b − a ) 2 /12 Uses: random sampling, base for inverse transform sampling ( g e n e r a t e a n y d i s t r i b u t i o n b y f e e d i n g u n i f o r m s a m p l e s t h r o u g h F − 1 ) (\mathrm{generate}\;\mathrm{any}\;\mathrm{distribution}\;\mathrm{by}\;\mathrm{feeding}\;\mathrm{uniform}\;\mathrm{samples}\;\mathrm{through}\;F^{-1}) ( generate any distribution by feeding uniform samples through F − 1 )
Exponential distribution: setup, memorylessness, use cases. medium X ~ Exp(λ): time between events in a Poisson process. PDF: f(x) = λ * e^(-λx) for x ≥ 0. E[X] = 1/λ; Var ( X ) = 1 / λ 2 \operatorname{Var}(X)\; = \;1 / {\lambda}^{2} Var ( X ) = 1/ λ 2 Memoryless: P ( X > s + t ∣ X > s ) = P ( X > t ) P(X\; > \;s + t\; \mid \;X\; > \;s)\; = \;P(X\; > \;t) P ( X > s + t ∣ X > s ) = P ( X > t ) Uses: waiting times (until next call, next failure), radioactive decay, survival analysis (constant hazard rate).
Normal distribution: PDF and key properties. easy X ~ N ( μ , σ 2 ) N({\mu}, \;{\sigma}^{2}) N ( μ , σ 2 ) f ( x ) = ( 1 / s q r t ( 2 π σ 2 ) ) ⋅ exp ( − ( x − μ ) 2 / ( 2 σ 2 ) ) f(x)\; = \;(1\; / \;\mathrm{sqrt}(2{\pi}{\sigma}^{2}))\; \cdot \;\operatorname{exp}( - (x - {\mu})^{2}\; / \;(2{\sigma}^{2})) f ( x ) = ( 1 / sqrt ( 2 π σ 2 )) ⋅ exp ( − ( x − μ ) 2 / ( 2 σ 2 )) Bell-shaped, symmetric around μ. ~68% of mass within 1σ, ~95% within 2σ, ~99.7% within 3σ. Sum of independent normals is normal. Standardized: Z = (X-μ)/σ ~ N(0,1). Universally used because of CLT — many sample statistics are approximately normal for large n.
Multivariate normal: parameters and key properties. medium X ~ N(μ, Σ), where μ is a length-d mean vector and Σ is a d×d covariance matrix (symmetric positive semi-definite). PDF: |2πΣ|^( − 1 / 2 ) ⋅ exp ( − 0.5 ⋅ ( x − μ ) Σ − 1 ( x − μ ) ) ( - 1 / 2)\; \cdot \;\operatorname{exp}( - 0.5\; \cdot \;(x - {\mu})\;{\Sigma}^{-1}\;(x - {\mu})) ( − 1/2 ) ⋅ exp ( − 0.5 ⋅ ( x − μ ) Σ − 1 ( x − μ )) Marginals and conditionals of a multivariate normal are also normal. Linear combinations remain normal. Uncorrelated components (Σ diagonal) → independent. Foundation of Gaussian processes, Kalman filters, LDA, and joint modeling.
What is a covariance matrix and its key properties? medium For random vector X ∈ R d R^{d} R d Cov ( X ) = E [ ( X − μ ) ( X − μ ) ] \operatorname{Cov}(X)\; = \;E[(X - {\mu})(X - {\mu})] Cov ( X ) = E [( X − μ ) ( X − μ )] Diagonal: variances Var ( X i ) \operatorname{Var}(X_{i}) Var ( X i ) Off-diagonal: covariances Cov ( X i , X j ) \operatorname{Cov}(X_{i}, \;X_{j}) Cov ( X i , X j ) Properties: (1) symmetric, (2) positive semi-definite, (3) real eigenvalues ≥ 0. Standardization: correlation m a t r i x = D − 1 \mathrm{matrix}\; = \;D^{-1} matrix = D − 1 D − 1 D^{-1} D − 1 PCA is eigendecomposition of Σ; Mahalanobis distance uses Σ − 1 {\Sigma}^{-1} Σ − 1
What is linearity of expectation? easy For any random variables X, Y and constants a, b: E[aX + bY] = a*E[X] + b*E[Y]. Holds regardless of dependence between X and Y. Very useful because it lets you decompose expectations of complex sums into pieces even when the pieces are dependent. Example: expected number of collisions in a hash table with n keys and m slots is (n choose 2)/m — via linearity over the C(n,2) pairs. Variance of a sum: Var ( X + Y ) \operatorname{Var}(X\; + \;Y) Var ( X + Y ) medium Var ( X + Y ) = Var ( X ) + Var ( Y ) + 2 ⋅ Cov ( X , Y ) \operatorname{Var}(X\; + \;Y)\; = \;\operatorname{Var}(X)\; + \;\operatorname{Var}(Y)\; + \;2\; \cdot \;\operatorname{Cov}(X, \;Y) Var ( X + Y ) = Var ( X ) + Var ( Y ) + 2 ⋅ Cov ( X , Y ) Only when X and Y are uncorrelated (Cov = 0) does the covariance term vanish and Var ( X + Y ) = Var ( X ) + Var ( Y ) \operatorname{Var}(X + Y)\; = \;\operatorname{Var}(X)\; + \;\operatorname{Var}(Y) Var ( X + Y ) = Var ( X ) + Var ( Y ) Crucial in portfolio theory (diversification reduces variance only if assets are not perfectly correlated), A/B testing (variance of the mean depends on within-subject correlation), and RL (variance-reduced estimators).
Write covariance and correlation formulas. easy Cov ( X , Y ) = E [ ( X − μ X ) ( Y − μ Y ) ] = E [ X Y ] − E [ X ] \operatorname{Cov}(X, \;Y)\; = \;E[(X\; - \;{\mu}_{X})(Y\; - \;{\mu}_{Y})]\; = \;E[\mathrm{XY}]\; - \;E[X] Cov ( X , Y ) = E [( X − μ X ) ( Y − μ Y )] = E [ XY ] − E [ X ] Corr ( X , Y ) = ρ = Cov ( X , Y ) / ( σ X σ Y ) \operatorname{Corr}(X, \;Y)\; = \;{\rho}\; = \;\operatorname{Cov}(X, \;Y)\; / \;({\sigma}_{X}\;{\sigma}_{Y}) Corr ( X , Y ) = ρ = Cov ( X , Y ) / ( σ X σ Y ) Sample versions: divide by n-1 (unbiased). Correlation is scale-invariant; covariance has weird units (product of X and Y units). Cauchy-Schwarz inequality g u a r a n t e e s ∣ Cov ( X , Y ) \mathrm{guarantees}\; \mid \operatorname{Cov}(X, Y) guarantees ∣ Cov ( X , Y ) σ X {\sigma}_{X} σ X σ Y {\sigma}_{Y} σ Y s o ∣ ρ \mathrm{so}\; \mid {\rho} so ∣ ρ What is E [ X ∣ Y ] E[X\; \mid \;Y] E [ X ∣ Y ] medium E [ X ∣ Y = y ] E[X\; \mid \;Y\; = \;y] E [ X ∣ Y = y ] As a function of Y, E [ X ∣ Y ] E[X\; \mid \;Y] E [ X ∣ Y ] Law of total expectation (Adam's law): E [ X ] = E [ E [ X ∣ Y ] ] E[X]\; = \;E[E[X\; \mid \;Y]] E [ X ] = E [ E [ X ∣ Y ]] Foundation for regression ( E [ Y ∣ X ] i s t h e b e s t p r e d i c t o r o f Y ) (E[Y \mid X]\;\mathrm{is}\;\mathrm{the}\;\mathrm{best}\;\mathrm{predictor}\;\mathrm{of}\;Y) ( E [ Y ∣ X ] is the best predictor of Y ) Law of Total Variance — Var ( X ) \operatorname{Var}(X) Var ( X ) hard Var ( X ) = E [ Var ( X ∣ Y ) ] + Var ( E [ X ∣ Y ] ) \operatorname{Var}(X)\; = \;E[\operatorname{Var}(X\; \mid \;Y)]\; + \;\operatorname{Var}(E[X\; \mid \;Y]) Var ( X ) = E [ Var ( X ∣ Y )] + Var ( E [ X ∣ Y ]) 'Within-group' variance (average of conditional variances) plus 'between-group' variance (variance of conditional means). Foundation of ANOVA (partitions total variance into between- and within-group sources), variance decomposition in regression ( R 2 = e x p l a i n e d / t o t a l ) (R^{2}\; = \;\mathrm{explained} / \mathrm{total}) ( R 2 = explained / total )
Why does the Cauchy distribution have no mean? hard Cauchy(0,1) PDF: 1 / ( π ( 1 + x 2 ) ) 1\; / \;({\pi}(1 + x^{2})) 1 / ( π ( 1 + x 2 )) Its tails decay only as 1 / x 2 1 / x^{2} 1/ x 2 Consequence: sample mean of Cauchy does not converge (LLN fails). Sums of Cauchy are still Cauchy (heavy-tail preservation), so CLT doesn't apply either. Use case: robust statistics use it as a heavy-tailed model for outliers; ratio of two normals has Cauchy distribution.
State Jensen's inequality and give an ML example. hard For a convex function g: E[g(X)] ≥ g(E[X]). Concave: E[g(X)] ≤ g(E[X]). Consequence: E [ X 2 ] E[X^{2}] E [ X 2 ] ( E [ X ] ) 2 (E[X])^{2} ( E [ X ] ) 2 In ML: (1) log-likelihood is concave — Jensen bounds expected log-likelihood ≤ log of expected likelihood; foundation of the EM algorithm's E-step. (2) log ( m e a n p r e d i c t i o n ) \operatorname{log}(\mathrm{mean}\;\mathrm{prediction}) log ( mean prediction ) log ( p r e d i c t i o n ) \operatorname{log}(\mathrm{prediction}) log ( prediction ) (3) Explains why you can't just average logs then exponentiate to recover a mean.
What is a moment generating function and why care? hard M X ( t ) = E [ e ( t X ) ] M_{X}(t)\; = \;E[e(\mathrm{tX})] M X ( t ) = E [ e ( tX )] When it exists in an open interval around 0, it uniquely determines the distribution — two RVs with the same MGF are identically distributed. Useful because: (1) generates moments via derivatives ( M X ( 0 ) = E [ X ] , M X ( 0 ) = E [ X 2 ] , ) (MX(0)\; = \;E[X], \;MX(0)\; = \;E[X^{2}], \;) ( M X ( 0 ) = E [ X ] , M X ( 0 ) = E [ X 2 ] , ) Characteristic functions (imaginary argument) exist even when MGF doesn't (Cauchy).
State Chebyshev's inequality. medium P ( ∣ X − μ ∣ ≥ k σ ) P( \mid X\; - \;{\mu} \mid \; \ge \;k{\sigma}) P ( ∣ X − μ ∣ ≥ k σ ) 1 / k 2 1 / k^{2} 1/ k 2 Distribution-free bound on tail probabilities — at most 1 / k 2 1 / k^{2} 1/ k 2 Consequence: at least 75% within 2σ, 89% within 3σ. Much looser than the 95/99.7% for normal — general bounds are wide. Applied in concentration arguments, generalization bounds in learning theory.
State Markov's inequality. medium For a non-negative random variable X and a > 0: P(X ≥ a) ≤ E[X] / a. Even weaker than Chebyshev but requires only non-negativity and a finite mean. Used to derive Chebyshev ( a p p l y M a r k o v t o ( X − μ ) 2 ) (\mathrm{apply}\;\mathrm{Markov}\;\mathrm{to}\;(X - {\mu})^{2}) ( apply Markov to ( X − μ ) 2 ) ( a p p l y M a r k o v t o e ( t X ) a n d o p t i m i z e t ) (\mathrm{apply}\;\mathrm{Markov}\;\mathrm{to}\;e(\mathrm{tX})\;\mathrm{and}\;\mathrm{optimize}\;t) ( apply Markov to e ( tX ) and optimize t ) Building block of concentration inequalities.
What is Hoeffding's inequality? hard For bounded i.i.d. X i X_{i} X i P ( ∣ X n − μ ∣ ≥ t ) P( \mid Xn\; - \;{\mu} \mid \; \ge \;t) P ( ∣ X n − μ ∣ ≥ t ) 2 ⋅ exp ( − 2 n ⋅ t 2 / ( b − a ) 2 ) 2\; \cdot \;\operatorname{exp}( - 2n\; \cdot \;t^{2}\; / \;(b - a)^{2}) 2 ⋅ exp ( − 2 n ⋅ t 2 / ( b − a ) 2 ) Exponential (Gaussian-like) tail bound for sample means of bounded variables. Much tighter than Chebyshev. Used to prove PAC learning bounds, bandit / online learning regret bounds, and to derive sample-complexity requirements: n ~ log ( 1 / δ ) / t 2 \operatorname{log}(1 / {\delta})\; / \;t^{2} log ( 1/ δ ) / t 2 Interview classic.
How does Monte Carlo estimation work and its convergence rate? medium To estimate E[f(X)] where X has a known distribution: draw i.i.d. samples X 1 X_{1} X 1 X n X_{n} X n f ( X i ) f(X_{i}) f ( X i ) By LLN, the average converges to the expectation. Rate: O(1/√n) — SE = σ f \operatorname{SE}\; = \;{\sigma}_{f} SE = σ f Great for high-dimensional integrals where deterministic quadrature is impossible (curse of dimensionality). Variance-reduction tricks: importance sampling, control variates, antithetic variates. MCMC (Markov Chain Monte Carlo) uses MC when direct sampling is hard.
What statistical properties make maximum likelihood the default estimator? medium Given data X and a parametric family p(x; θ), MLE picks θ̂ = argmax_θ Σ log p ( x i θ ) p(x_{i}\;{\theta}) p ( x i θ ) Equivalently: minimize negative log-likelihood. Properties: consistent (θ̂ → θ*), asymptotically normal ( n ( θ − θ ⋅ ) → N ( 0 , I − 1 ) ) ( \sqrt n({\theta}\; - \;{\theta} \cdot )\; \to \;N(0, \;I^{-1})) ( n ( θ − θ ⋅ ) → N ( 0 , I − 1 )) Foundation of parametric inference and of most ML training objectives (logistic regression, softmax cross-entropy — all MLE).
What is Fisher information? hard I(θ) = -E [ ∂ 2 log p ( X θ ) / ∂ θ 2 ] = E [ ( ∂ log p ( X θ ) / ∂ θ ) 2 ] E[ \partial ^{2}\;\operatorname{log}\;p(X\;{\theta})\; / \; \partial {\theta}^{2}]\; = \;E[( \partial \;\operatorname{log}\;p(X\;{\theta}) / \partial {\theta})^{2}] E [ ∂ 2 log p ( X θ ) / ∂ θ 2 ] = E [( ∂ log p ( X θ ) / ∂ θ ) 2 ] Measures how much information the observations carry about θ. Cramér-Rao bound: for any unbiased estimator, Var ( θ ) \operatorname{Var}({\theta}) Var ( θ ) MLE asymptotic variance = 1 / (n * I(θ)) → MLE is asymptotically efficient. Used to define natural gradient ( n a t u r a l g r a d i e n t d e s c e n t ∝ I ( θ ) − 1 ∇ L ) (\mathrm{natural}\;\mathrm{gradient}\;\mathrm{descent}\; \propto \;I({\theta})^{-1}\; \nabla L) ( natural gradient descent ∝ I ( θ ) − 1 ∇ L )
Bias, variance, consistency of estimators — define. medium Bias: E[θ̂] - θ. Variance: Var ( θ ) \operatorname{Var}({\theta}) Var ( θ ) MSE = b i a s 2 + v a r i a n c e \operatorname{MSE}\; = \;\mathrm{bias}^{2}\; + \;\mathrm{variance} MSE = bias 2 + variance Consistency: θ̂_n → θ in probability as n → ∞. Unbiased ≠ consistent (unbiased with growing variance can be inconsistent); consistent ≠ unbiased (biased with vanishing bias can be consistent). MLE and MAP are typically consistent but biased for small n.
MLE for a Bernoulli(p) — derive. medium Likelihood: L(p) = p^Σ x i ⋅ ( 1 − p ) {\Sigma}x_{i}\; \cdot \;(1 - p) Σ x i ⋅ ( 1 − p ) ( n − Σ x i ) (n\; - \;{\Sigma}x_{i}) ( n − Σ x i ) Log-lik: ℓ( p ) = Σ x i ⋅ log (p)\; = \;{\Sigma}x_{i}\; \cdot \;\operatorname{log} ( p ) = Σ x i ⋅ log p + ( n − Σ x i ) ⋅ log ( 1 − p ) p\; + \;(n\; - \;{\Sigma}x_{i})\; \cdot \;\operatorname{log}(1 - p) p + ( n − Σ x i ) ⋅ log ( 1 − p ) p = Σ x i / p − ( n − Σ x i ) / ( 1 − p ) = 0 p\; = \;{\Sigma}x_{i}\; / \;p\; - \;(n\; - \;{\Sigma}x_{i}) / (1 - p)\; = \;0 p = Σ x i / p − ( n − Σ x i ) / ( 1 − p ) = 0 Σ x i / n = X {\Sigma}x_{i}\; / \;n\; = \;X Σ x i / n = X So the MLE of a Bernoulli parameter is just the sample proportion. Also happens to be unbiased and minimum-variance (via CR bound). Classic textbook derivation asked in interviews. MLE for N o r m a l ( μ , σ 2 ) \mathrm{Normal}({\mu}, \;{\sigma}^{2}) Normal ( μ , σ 2 ) medium μ̂ = X̄ (unbiased, minimum variance). σ̂² = ( 1 / n ) ⋅ Σ ( X i − X ) 2 (1 / n)\; \cdot \;{\Sigma}(X_{i}\; - \;X)^{2} ( 1/ n ) ⋅ Σ ( X i − X ) 2 Bias-corrected version divides by n-1 (sample variance). For large n, both converge to σ 2 {\sigma}^{2} σ 2 Deep-learning tie-in: minimizing MSE under Gaussian noise = MLE with fixed σ. Softmax cross-entropy = MLE for a categorical distribution. Almost every parametric ML training objective is an MLE.
When would you use Hoeffding's inequality for a CI? hard Distribution-free CI for the mean of a bounded random variable in [a, b]: P ( ∣ X − μ ∣ > t ) P( \mid X\; - \;{\mu} \mid \; > \;t) P ( ∣ X − μ ∣ > t ) 2 ⋅ exp ( − 2 n t 2 / ( b − a ) 2 ) 2\; \cdot \;\operatorname{exp}( - 2\mathrm{nt}^{2}\; / \;(b - a)^{2}) 2 ⋅ exp ( − 2 nt 2 / ( b − a ) 2 ) Gives ε for target coverage: ε = (b-a) * √(log(2/α) / (2n)). Uses: bandits (UCB), online learning regret, certification without normality. Weaker (wider) than CLT-based CIs for well-behaved data but valid for any n and any distribution on [a, b].
Cramér-Rao lower bound — state it. hard For any unbiased estimator θ̂ of θ under regularity conditions: Var ( θ ) \operatorname{Var}({\theta}) Var ( θ ) Fisher information sets the floor for estimator precision. MLE achieves this bound asymptotically (asymptotically efficient). Practical use: benchmark for whether an estimator can be improved. For biased estimators, generalized bounds exist (van Trees). Companion to the bias-variance decomposition.
What is the influence function? hard For a functional statistic T(F): I F ( x T , F ) = l i m ε → 0 \mathrm{IF}(x\;T, \;F)\; = \;\mathrm{lim}_{{\varepsilon} \to 0} IF ( x T , F ) = lim ε → 0 T ( ( 1 − ε ) F + ε δ x ) − T ( F ) T((1 - {\varepsilon})F\; + \;{\varepsilon}{\delta}_{x})\; - \;T(F) T (( 1 − ε ) F + ε δ x ) − T ( F ) Measures how much T changes if a single point x is added. Uses: (1) derive asymptotic variance via Var ( n T ) \operatorname{Var}( \sqrt n\;T) Var ( n T ) E [ I F 2 ] E[\mathrm{IF}^{2}] E [ IF 2 ] Foundation of modern robust statistics and semi-parametric efficiency theory.
Give the OLS closed-form and its variance. medium β̂ = ( X X ) − 1 (XX)^{-1} ( X X ) − 1 Var ( β ∣ X ) = σ 2 ⋅ ( X X ) − 1 \operatorname{Var}({\beta}\; \mid \;X)\; = \;{\sigma}^{2}\; \cdot \;(XX)^{-1} Var ( β ∣ X ) = σ 2 ⋅ ( X X ) − 1 Estimate σ 2 {\sigma}^{2} σ 2 s 2 = R S S / ( n − p ) s^{2}\; = \;\mathrm{RSS}\; / \;(n\; - \;p) s 2 = RSS / ( n − p ) SEs come from diagonal of s 2 ⋅ ( X X ) − 1 s^{2}\; \cdot \;(XX)^{-1} s 2 ⋅ ( X X ) − 1 When X'X is ill-conditioned (multicollinearity), variances blow up. Solutions: ridge regression, principal components regression, or dropping correlated features. This formula is the reason multicollinearity is so damaging: it inflates the variance of β̂.
State Bayes' theorem and what each term means. easy P ( θ ∣ D ) = P ( D ∣ θ ) ⋅ P ( θ ) / P ( D ) P({\theta}\; \mid \;D)\; = \;P(D\; \mid \;{\theta})\; \cdot \;P({\theta})\; / \;P(D) P ( θ ∣ D ) = P ( D ∣ θ ) ⋅ P ( θ ) / P ( D ) Posterior = Likelihood * Prior / Evidence. Prior encodes belief before seeing D. Likelihood tells how well θ explains D. Evidence P(D) = ∫ P ( D ∣ θ ) P(D\; \mid \;{\theta}) P ( D ∣ θ ) In practice, we compute posterior up to proportionality: P ( θ ∣ D ) P({\theta}\; \mid \;D) P ( θ ∣ D ) P ( D ∣ θ ) ⋅ P ( θ ) P(D\; \mid \;{\theta})\; \cdot \;P({\theta}) P ( D ∣ θ ) ⋅ P ( θ )
Frequentist vs Bayesian — key philosophical difference. medium Frequentist: parameters are fixed but unknown; probability = long-run frequency; CIs, p-values, MLE. Bayesian: parameters are random with a distribution encoding belief; probability = degree of belief; posterior, credible intervals, MAP. Pragmatic: Bayesian is natural when priors are meaningful, uncertainty quantification matters, or you're doing online learning; frequentist is faster and standard in confirmatory studies. Modern ML mixes both freely (MAP = MLE + regularization; Bayesian deep learning).
Potential outcomes framework — Rubin causal model. hard For each unit i and treatment T, define Y i ( 0 ) Y_{i}(0) Y i ( 0 ) Y i ( 1 ) Y_{i}(1) Y i ( 1 ) Individual causal effect: Y i ( 1 ) − Y i ( 0 ) Y_{i}(1)\; - \;Y_{i}(0) Y i ( 1 ) − Y i ( 0 ) Estimands: ATE = E[Y(1) - Y(0)], A T T = E [ Y ( 1 ) − Y ( 0 ) ∣ T = 1 ] \mathrm{ATT}\; = \;E[Y(1)\; - \;Y(0)\; \mid \;T = 1] ATT = E [ Y ( 1 ) − Y ( 0 ) ∣ T = 1 ] Identification requires: (1) SUTVA (no interference / one version of treatment), (2) unconfoundedness ( Y ( 0 ) , Y ( 1 ) T ∣ X ) (Y(0), \;Y(1)\;\;T\; \mid \;X) ( Y ( 0 ) , Y ( 1 ) T ∣ X ) Foundation of modern causal inference.