EasyDeepLearn
Statistics Fundamentals · section 9 of 15

Estimation: MLE, MoM, MAP

18 interview questions on estimation: mle, mom, map, each answered in full. Free to read, no account needed.

What does a 95% confidence interval mean?

medium
  • If we repeated the sampling and interval construction procedure many times, about 95% of the resulting intervals would contain the true parameter.
  • It's NOT 'there is a 95% probability the parameter is in this specific interval' — the parameter is fixed and the interval is random.
  • Interpret CIs as long-run coverage guarantees.
#confidence-interval#estimationPermalink & quiz →

What is the bootstrap and when do you use it?

medium
  • Bootstrap resamples the data with replacement to approximate the sampling distribution of an estimator.
  • You get standard errors, confidence intervals, and bias estimates without strong parametric assumptions.
  • Use it when analytical formulas are unavailable or unreliable (medians, ratios, complex estimators).
  • Rule of thumb: 1,000-10,000 bootstrap samples.
#bootstrap#estimationPermalink & quiz →

What is Maximum Likelihood Estimation?

medium
  • Choose parameter values that make the observed data most probable — i.e., maximize the likelihood function L(θ)  =  P(data  ∣  θ)L(\theta)\; = \;P(\mathrm{data}\; \mid \;\theta).
  • In practice, maximize log L (numerically stable, turns products into sums).
  • MLEs are consistent, asymptotically normal, and efficient under regularity conditions.
  • Add regularization/priors to obtain MAP estimation (Bayesian penalty).
#estimationPermalink & quiz →

What is the Method of Moments (MoM)?

medium
  • Set sample moments equal to theoretical moments and solve for parameters.
  • Example: Gamma(α, β) with mean α/β and variance α/β2{\alpha} / {\beta}^{2} → set sample mean = α/β and sample var  =  α/β2\mathrm{var}\; = \;{\alpha} / {\beta}^{2}, solve for α̂ and β̂.
  • Simple and consistent, but often less efficient than MLE.
  • Modern use: warm-start MLE; robust starting points; GMM (generalized method of moments) in econometrics.
#estimationPermalink & quiz →

MAP vs MLE — key difference.

medium
  • MLE: θ̂ = argmax p(D  ∣  θ)p(D\; \mid \;{\theta}).
  • MAP: θ̂ = argmax p(θ  ∣  D)  =  argmax⁡p({\theta}\; \mid \;D)\; = \;\operatorname{argmax} p(D  ∣  θ)  ⋅  p(θ)p(D\; \mid \;{\theta})\; \cdot \;p({\theta}).
  • MAP adds a prior — reduces to MLE with uniform prior.
  • In log space, MAP = MLE + log-prior term → acts as regularization.
  • L2 regularization = Gaussian prior; L1 = Laplace prior.
  • Not fully Bayesian: gives a point estimate, not the full posterior.
#estimation#bayesianPermalink & quiz →

Explain the EM algorithm.

hard
  • For MLE with latent variables (Z): (E) compute q(Z)  =  p(Z  ∣  X,  θold)q(Z)\; = \;p(Z\; \mid \;X, \;{\theta}_{\mathrm{old}}); (M) update θnew  =  argmax⁡{\theta}_{\mathrm{new}}\; = \;\operatorname{argmax}_θ Eq[log⁡  p(X,  Z  ∣  θ)]E_{q}[\operatorname{log}\;p(X, \;Z\; \mid \;{\theta})].
  • Guaranteed to increase the log-likelihood at each iteration, converges to a local max.
  • Uses: Gaussian mixture models (soft clustering), HMMs (Baum-Welch), missing data imputation.
  • Sensitive to init → try multiple starts.
#estimation#bayesianPermalink & quiz →

Standard error vs standard deviation — the difference.

easy
  • SD: spread of the data (√variance) — describes the population.
  • SE: spread of a statistic across hypothetical repeated samples — describes the estimator's precision.
  • Rule for the mean: SE⁡(X)  =  SD\operatorname{SE}(X)\; = \;\mathrm{SD} / √n.
  • As n grows, SD stays roughly the same, SE shrinks — that's why bigger samples give tighter estimates without changing what the population looks like.
#estimation#confidence-intervalPermalink & quiz →

What is the delta method?

hard
  • For a differentiable function g and asymptotically normal θ̂: g(θ̂) ~ N(g(θ),  g(θ)2  ⋅  Var⁡(θ))N(g({\theta}), \;g({\theta})^{2}\; \cdot \;\operatorname{Var}({\theta})) approximately.
  • Use to derive SE of transformations (ratios, log-odds).
  • Example: Var⁡(log⁡(p))\operatorname{Var}(\operatorname{log}(p)) ≈ Var⁡(p)  /  p\operatorname{Var}(p)\; / \;p̂².
  • Central for A/B testing ratio metrics (CTR per user, revenue per user with per-user denominator).
  • Fast alternative to bootstrap when you have a closed-form derivative.
#estimation#confidence-interval#ab-testingPermalink & quiz →

What is the jackknife?

hard
  • Leave-one-out precursor of the bootstrap.
  • For each i, compute θ̂_(-i) = statistic on data without observation i.
  • Jackknife SE = √((n-1)/n * Σ (θ(−i)  −  θ)2({\theta}( - i)\; - \;{\theta})^{2}).
  • Bias correction: n * θ̂ - (n-1) * θ̄.
  • Cheaper than bootstrap (n evaluations vs B ≈ 1000), but less general — works only for smooth statistics (fails for medians, quantiles).
  • Foundation of influence-function-based robust statistics.
#bootstrap#estimationPermalink & quiz →

What is a conjugate prior?

hard
  • A prior p(θ) is conjugate to a likelihood p(x  ∣  θ)p(x\; \mid \;{\theta}) if the posterior p(θ  ∣  x)p({\theta}\; \mid \;x) belongs to the same family.
  • Examples: Beta prior + Bernoulli likelihood → Beta posterior; Gamma prior + Poisson → Gamma; Normal prior + Normal (known σ) → Normal; Dirichlet + categorical → Dirichlet.
  • Enables closed-form Bayesian updates — cornerstone of Thompson sampling, online Bayesian inference, contextual bandits.
#bayesian#estimationPermalink & quiz →

Beta-Bernoulli conjugate update — derive.

medium
  • Prior: p ~ Beta(α, β).
  • Data: n Bernoulli trials with k successes.
  • Posterior: p  ∣  datap\; \mid \;\mathrm{data} ~ Beta(α + k, β + n - k).
  • Posterior mean = (α + k) / (α + β + n) → smoothly interpolates between prior mean α/(α+β) and MLE k/n as n grows.
  • Uses: Thompson sampling for binary bandits, small-sample click-through rate estimation, MAP with a fair-coin prior Beta(1, 1) = Uniform.
#bayesian#estimationPermalink & quiz →

What is an uninformative prior?

hard
  • Prior meant to encode minimal prior belief.
  • Options: (1) Flat / uniform (Beta(1,1)) — feels natural but is not invariant to reparameterization.
  • (2) Jeffreys prior — proportional to √det⁡(I(θ))\operatorname{det}(I({\theta})); invariant to reparameterization (Beta(0.5, 0.5) for Bernoulli).
  • (3) Reference prior (Bernardo).
  • For Bayesian A/B testing with weak prior belief, Jeffreys is a defensible default; results converge to MLE as n grows.
#bayesian#estimationPermalink & quiz →

What is Bayesian shrinkage?

hard
  • Posterior estimates are pulled ('shrunk') from noisy per-group MLEs toward the prior / group mean.
  • Effect: reduces variance at the cost of small bias — Stein's paradox shows shrinkage dominates MLE in ≥ 3 dimensions.
  • Uses: multi-arm bandits, small-sample rate estimation (individual player batting averages, per-store conversion), empirical Bayes hierarchical models.
  • Frequentist equivalents: ridge regression, James-Stein estimator, empirical Bayes.
#bayesian#estimationPermalink & quiz →

What is empirical Bayes?

hard
  • Estimate the hyperparameters of the prior from the data itself (usually by marginal maximum likelihood), then use that prior for the posterior.
  • Cheap approximation to a full hierarchical Bayesian model — avoids MCMC on hyperpriors.
  • Uses: shrinkage for large-scale multiple testing (Efron's LFDR), gene expression, hierarchical rate estimation.
  • Weakness: ignores uncertainty in the hyperparameter → slightly narrower posteriors than full Bayes.
#bayesian#estimationPermalink & quiz →

How does Thompson sampling work?

medium
  • For each arm, maintain posterior over its reward.
  • To choose an action, draw θj{\theta}_{j} ~ p(θj  ∣  data)p({\theta}_{j}\; \mid \;\mathrm{data}) for each arm, pick arm with highest sampled θj{\theta}_{j}.
  • Randomness is 'baked in' via sampling → automatic exploration-exploitation tradeoff.
  • Optimal frequentist regret bounds (Bayesian and worst-case).
  • Standard in online experimentation, recommender systems, contextual bandits.
  • Bayes / Beta-Bernoulli conjugate makes update O(1) per pull.
#bayesian#estimationPermalink & quiz →

Why do conjugate priors matter in production?

medium
  • Closed-form posterior updates → no MCMC needed, O(1) update per new observation → scales to streaming / online settings.
  • Examples: (1) Beta-Bernoulli for CTR bandits, (2) Gamma-Poisson for arrival-rate estimation, (3) Normal-Normal for personalization.
  • Even when the model isn't strictly conjugate, industry teams often prefer a well-tuned conjugate approximation to full MCMC for latency reasons.
#bayesian#estimationPermalink & quiz →

What does a 95% confidence interval actually assert?

medium
  • That the procedure generating it captures the true parameter in 95% of hypothetical repetitions of the study.
  • The confidence is a property of the method across repetitions, not of the particular interval you computed, which either contains the parameter or does not.
  • This is why saying there is a 95% probability that the parameter lies inside a specific interval is a frequentist error, even though it is the interpretation everyone wants; a Bayesian credible interval is the object that supports that statement, and it requires a prior.
  • In practice the useful reading is about precision: a narrow interval near zero says the effect is small, which is informative, while a wide interval crossing zero says the study could not distinguish a meaningful effect from nothing, which is not the same as evidence of no effect.
#confidence-interval#estimationPermalink & quiz →

When is a Bayesian analysis genuinely worth the extra effort?

medium
  • When you have real prior information, when the sample is small, or when you need the probability statement itself.
  • With little data, a weakly informative prior stabilizes estimates that maximum likelihood pushes to absurd values, which is why hierarchical models are standard for many small groups such as per-store or per-user effects: partial pooling shrinks noisy groups toward the overall mean by exactly as much as the data warrants.
  • It is also the right frame when a decision needs the probability that an effect exceeds a threshold, since that is a statement about the parameter and only a posterior supports it.
  • Where it is not worth it is a large-sample, well-identified problem with a single parameter, because the posterior will match the likelihood and you will have added computational and communication cost for no inferential gain.
#bayesian#estimationPermalink & quiz →

Practise Statistics Fundamentals

All 15 sectionsFlashcardsQuiz