EasyDeepLearn
Statistics Fundamentals · section 10 of 15

Confidence intervals & bootstrap

8 interview questions on confidence intervals & bootstrap, each answered in full. Free to read, no account needed.

95% CI for a mean — formula and interpretation.

easy
  • CI = X̄ ± tn−1,  0.975  ⋅  (s  /  n)t_{n - 1, \;0.975}\; \cdot \;(s\; / \; \sqrt n).
  • Interpretation: if we repeated the experiment many times, 95% of the CIs would contain the true μ.
  • NOT: 'there's a 95% chance μ is in this specific interval' (that's Bayesian credible interval language).
  • For large n, t ≈ z = 1.96.
  • Foundational for A/B testing and any point estimate reporting.
#confidence-intervalPermalink & quiz →

What CI methods exist for a proportion?

medium
  • Wald (naive): p̂ ± z * √(p̂(1-p̂)/n) — bad when p̂ is near 0/1 or n small (can even exceed [0,1]).
  • Wilson score: preferred default; more accurate coverage.
  • Clopper-Pearson (exact): guaranteed conservative coverage but wider.
  • Agresti-Coull: quick fix (add 2 successes + 2 failures).
  • Modern practice: use Wilson or Clopper-Pearson, avoid Wald except for very large n and mid-range p.
#confidence-intervalPermalink & quiz →

Bootstrap CI methods — percentile vs BCa vs studentized.

hard
  • Percentile: use 2.5th and 97.5th quantiles of bootstrap distribution — simple but biased when the sampling distribution is skewed.
  • BCa (bias-corrected & accelerated): corrects bias and skewness → generally best default.
  • Studentized (bootstrap-t): asymptotically most accurate but requires an SE estimate per bootstrap sample.
  • Basic (pivotal): 2*θ̂ - Q2.5Q_{2.5}, 2*θ̂ - Q97.5Q_{97.5}.
  • Rule: default BCa; percentile only if you know the sampling distribution is symmetric.
#bootstrap#confidence-intervalPermalink & quiz →

Why do you need block bootstrap for time series?

hard
  • Standard bootstrap resamples individual points, destroying temporal dependence and grossly under-estimating variance for autocorrelated data.
  • Block bootstrap resamples contiguous blocks (moving or stationary) to preserve short-range dependence.
  • Block length L chosen so autocorrelation dies within L (rule  of  thumb  n(1/3))(\mathrm{rule}\;\mathrm{of}\;\mathrm{thumb}\;n(1 / 3)).
  • Foundational for CIs in time-series metrics, hedge-fund performance, drift detection.
#bootstrap#confidence-intervalPermalink & quiz →

Prediction interval vs confidence interval — the difference.

medium
  • CI: uncertainty in a parameter (μ, β, p) — shrinks with n.
  • PI: uncertainty in a new individual observation Ynew  =  μ  +  εY_{\mathrm{new}}\; = \;{\mu}\; + \;{\varepsilon} — includes both parameter uncertainty AND irreducible noise σ, so wider than CI and doesn't shrink to zero (bounded by σ).
  • Practical: CI answers 'what's the average?' → tightens with n.
  • PI answers 'where will the next point land?' → bounded by irreducible variance.
#confidence-interval#regressionPermalink & quiz →

Credible interval vs confidence interval.

medium
  • CI (frequentist): 'if we repeated the experiment many times, 95% of these intervals would contain θ' — probability statement about the procedure.
  • Credible interval (Bayesian): 'given the data + prior, P(θ  ∈  [a,  b]  ∣  data)  =  95P({\theta}\; \in \;[a, \;b]\; \mid \;\mathrm{data})\; = \;95%' — probability statement about θ.
  • Numerically they can coincide with uninformative priors and large n, but they answer different questions.
  • Users usually want the credible interval interpretation.
#confidence-interval#bayesianPermalink & quiz →

Why do ratio metrics need the delta method?

hard
  • Ratios like CTR = clicks / impressions have per-user numerators AND denominators → sample-averaged ratio has non-trivial variance.
  • Naive user-level SE is wrong (ignores denominator variability).
  • Delta method gives: Var⁡(N/D)\operatorname{Var}(N / D) ≈ (μN/μD)2  ⋅  [Var⁡(N)/μN2  +  Var⁡(D)/μD2  −  2⋅Cov⁡(N,D)/(μN⋅μD)]  /  n({\mu}_{N} / {\mu}_{D})^{2}\; \cdot \;[\operatorname{Var}(N) / {\mu}_{N}^{2}\; + \;\operatorname{Var}(D) / {\mu}_{D}^{2}\; - \;2 \cdot \operatorname{Cov}(N, D) / ({\mu}_{N} \cdot {\mu}_{D})]\; / \;n.
  • Alternative: bootstrap on user level.
  • Every serious A/B platform (Statsig, Eppo, Facebook's own) uses this for CTR / click-through / like-through.
#ab-testing#confidence-intervalPermalink & quiz →

When does the bootstrap give you the wrong answer?

hard
  • It fails where the statistic depends on the extreme tail of the distribution, because resampling can never produce a value larger than the largest observation, so the maximum and near-extreme quantiles are badly estimated.
  • It fails with small samples, since the empirical distribution is a poor stand-in for the population and the intervals come out too narrow.
  • It fails when observations are dependent, which is the most common real-world violation: naive resampling of time series or clustered data destroys the correlation structure and understates variance badly, and you need a block or cluster bootstrap instead.
  • It also struggles with statistics that are not smooth functions of the data, such as the median in small samples with ties.
  • The general rule is that it substitutes computation for a distributional assumption, but not for independence.
#bootstrap#confidence-intervalPermalink & quiz →

Practise Statistics Fundamentals

All 15 sectionsFlashcardsQuiz