EasyDeepLearn
Statistics Fundamentals · section 13 of 15

Bayesian methods

16 interview questions on bayesian methods, each answered in full. Free to read, no account needed.

What is the posterior predictive distribution?

hard
  • p(ynew  ∣  D)p(y_{\mathrm{new}}\; \mid \;D) = ∫ p(ynew  ∣  θ)p(y_{\mathrm{new}}\; \mid \;{\theta}) p(θ  ∣  D)p({\theta}\; \mid \;D) dθ.
  • Predicts new data marginalizing over posterior uncertainty.
  • Unlike MLE + plug-in prediction, it accounts for parameter uncertainty → wider (better-calibrated) prediction intervals, especially with small n.
  • In practice: draw θ^(s) from posterior, sample ynewy_{\mathrm{new}} ~ p(y  ∣  θ(s))p(y\; \mid \;{\theta}(s)).
  • Foundation of Bayesian forecasting and probabilistic programming.
#bayesianPermalink & quiz →

What is the marginal likelihood (evidence) and why is it hard?

hard
  • p(D) = ∫ p(D  ∣  θ)p(D\; \mid \;{\theta}) p(θ) dθ.
  • Normalizing constant for the posterior.
  • High-dimensional integral → intractable in closed form for most models.
  • Bayesian model comparison: Bayes factor  =  p(D  ∣  M1)  /  p(D  ∣  M2)\mathrm{factor}\; = \;p(D\; \mid \;M_{1})\; / \;p(D\; \mid \;M_{2}).
  • Computing p(D): bridge sampling, thermodynamic integration, Chib's method, variational lower bounds (ELBO ≤ log p(D)).
  • MCMC alone doesn't give p(D) — special methods required.
#bayesianPermalink & quiz →

How do you interpret a Bayes factor?

hard
  • BF12  =  p(D  ∣  M1)  /  p(D  ∣  M2)\mathrm{BF}_{12}\; = \;p(D\; \mid \;M_{1})\; / \;p(D\; \mid \;M_{2}) — ratio of marginal likelihoods → posterior odds = BF × prior odds.
  • Jeffreys scale: BF > 10 strong evidence for M1M_{1}, > 100 decisive; BF < 1/10 flip.
  • Unlike p-values, symmetric — can support H0H_{0}.
  • Weakness: hypersensitive to prior choice for nuisance parameters (Lindley's paradox).
  • Alternative in practice: LOO-CV, WAIC, PSIS-LOO for predictive model comparison.
#bayesianPermalink & quiz →

How does Bayesian A/B testing work?

medium
  • Model: pAp_{A}, pBp_{B} ~ Beta prior; observe successes / failures → Beta posteriors.
  • Metric: P(pB  >  pA  ∣  data)P(p_{B}\; > \;p_{A}\; \mid \;\mathrm{data}) via Monte Carlo (sample from both posteriors, count fraction).
  • Also useful: posterior of the lift (pB  −  pA)(p_{B}\; - \;p_{A}) or of the relative uplift.
  • Advantages: interpretable ('72% chance B is better') and safe under peeking (no p-hacking inflation, decisions optimal under a loss function).
  • Standard in growth teams alongside frequentist.
#bayesian#ab-testingPermalink & quiz →

How does Metropolis-Hastings work?

hard
  • MCMC: build a Markov chain whose stationary distribution is the posterior.
  • Given current θ, propose θ' ~ q(θ  ∣  θ)q({\theta}\; \mid \;{\theta}).
  • Accept with probability α  =  min⁡(1,  [p(θ  ∣  D)  q(θ  ∣  θ)]  /  [p(θ  ∣  D)  q(θ  ∣  θ)]){\alpha}\; = \;\operatorname{min}(1, \;[p({\theta}\; \mid \;D)\;q({\theta}\; \mid \;{\theta})]\; / \;[p({\theta}\; \mid \;D)\;q({\theta}\; \mid \;{\theta})]).
  • Symmetric q simplifies to α  =  min⁡(1,  p(θ  ∣  D)  /  p(θ  ∣  D)){\alpha}\; = \;\operatorname{min}(1, \;p({\theta}\; \mid \;D)\; / \;p({\theta}\; \mid \;D)).
  • Only needs the posterior up to a constant → dodges intractable evidence.
  • Weaknesses: slow mixing in high dims; sensitive to proposal scale.
#bayesian#mcmcPermalink & quiz →

What is Gibbs sampling?

hard
  • Special MCMC: sample each parameter (or block) from its full conditional p(θj  ∣  θ−j,  D)p({\theta}_{j}\; \mid \;{\theta}_{- j}, \;D), one at a time.
  • Guaranteed acceptance (α = 1) when full conditionals are known.
  • Works well for conjugate hierarchical models, Bayesian networks, LDA.
  • Weak in strongly correlated posteriors — slow mixing.
  • Modern alternatives (HMC, NUTS) are usually better default, but Gibbs is still standard when conjugacy makes full conditionals trivial.
#bayesian#mcmcPermalink & quiz →

Hamiltonian Monte Carlo — intuition.

hard
  • Uses gradients of log-posterior to propose long, informed steps → efficient in high dimensions.
  • Introduces auxiliary momentum p, runs Hamiltonian dynamics (leapfrog) to propose new (θ, p), MH-accepts.
  • Advantages: dramatically better mixing than random-walk MH in high dimensions.
  • NUTS (No-U-Turn Sampler): auto-tunes trajectory length; default in Stan, PyMC, NumPyro.
  • Requires differentiable posterior — great for continuous models, doesn't handle discrete latents directly.
#bayesian#mcmcPermalink & quiz →

How do you diagnose MCMC convergence?

hard
  • (1) R̂ (Gelman-Rubin) < 1.01 across multiple chains → chains agree.
  • (2) ESS (effective sample size) ≥ 400 per parameter.
  • (3) Trace plots: should look like fuzzy caterpillars, no drift or stickiness.
  • (4) Autocorrelation plots decay quickly.
  • (5) Divergences in NUTS: indicate hard posterior geometry — reparameterize (non-centered) or increase adaptδ\mathrm{adapt}_{\delta}.
  • Never trust a single chain's samples without these checks.
#bayesian#mcmcPermalink & quiz →

Burn-in and thinning — why?

medium
  • Burn-in: discard initial iterations before the chain reaches stationarity.
  • Typical: 10-50% of chain.
  • Thinning: keep every k-th sample to reduce autocorrelation.
  • Modern view: thinning wastes information unless memory is tight — better to use all samples with autocorrelation-aware summaries.
  • Use ESS-based decisions rather than fixed burn-in / thinning rules.
  • NUTS + adaptive warmup replaces manual burn-in in practice.
#bayesian#mcmcPermalink & quiz →

Variational inference vs MCMC.

hard
  • VI: approximate posterior p(θ  ∣  D)p({\theta}\; \mid \;D) with a simpler family q_φ(θ) (e.g. mean-field Gaussian) by maximizing ELBO  =  Eq[log⁡  p(D,  θ)]  −  Eq[log⁡  q(θ)]\mathrm{ELBO}\; = \;E_{q}[\operatorname{log}\;p(D, \;{\theta})]\; - \;E_{q}[\operatorname{log}\;q({\theta})].
  • Advantages: much faster, scales to big data (SVI, mini-batches), differentiable → fits on GPU.
  • Weaknesses: biased (limited by family); mean-field VI under-covers uncertainty and drops correlations.
  • Use for large-scale problems where speed > accuracy.
  • Modern: normalizing flows for richer q.
#bayesian#mcmcPermalink & quiz →

Derive the ELBO for VI.

hard
  • log p(D) = log ∫ p(D, θ) dθ = log ∫ q(θ) * p(D, θ)/q(θ) dθ ≥ Eq[log⁡  p(D,  θ)/q(θ)]E_{q}[\operatorname{log}\;p(D, \;{\theta}) / q({\theta})] (Jensen).
  • ELBO = E_q[log p(D, θ)] - E_q[log q(θ)] = E_q[log p(D | θ)] - KL(q || p(θ)). log p(D)  −  ELBO  =  KL⁡(q  ∣∣  p(θ  ∣  D))p(D)\; - \;\mathrm{ELBO}\; = \;\operatorname{KL}(q\; \mid \mid \;p({\theta}\; \mid \;D)) ≥ 0 → maximizing ELBO minimizes KL to the posterior.
  • Same objective drives VAE training in deep learning.
#bayesian#mcmcPermalink & quiz →

Informative vs non-informative priors — the tradeoff.

medium
  • Non-informative (flat, Jeffreys): 'let the data speak' — safe when you have plenty of data, but can be worse than an informative prior at small n.
  • Informative (based on historical data, domain knowledge, similar experiments): dramatically improves inference in small-sample settings, but can also inject bias if wrong.
  • Rule: (1) when n is large, prior barely matters.
  • (2) When n is small, informative priors are your friend if you have credible data.
#bayesianPermalink & quiz →

WAIC and LOO-CV — Bayesian model comparison.

hard
  • Both estimate out-of-sample predictive performance.
  • WAIC = -2  ⋅  (lppd  −  pWAIC)2\; \cdot \;(\mathrm{lppd}\; - \;p_{\mathrm{WAIC}}) where lppd = expected log posterior predictive on training data, pWAIC  =  effectivep_{\mathrm{WAIC}}\; = \;\mathrm{effective} number of parameters via posterior variance.
  • LOO-CV: exact leave-one-out log predictive; approximated efficiently via PSIS-LOO (Vehtari et al.) using importance sampling — modern default in Stan / PyMC / Arviz.
  • Both replace AIC / BIC / Bayes factor for model comparison in most cases.
#bayesianPermalink & quiz →

What is a prior predictive check?

hard
  • Simulate data from the prior alone: θ ~ p(θ), y ~ p(y  ∣  θ)p(y\; \mid \;{\theta}).
  • Check whether generated y is plausible given domain knowledge (e.g. simulated coin flip probabilities in [0, 1]; simulated log-revenues aren't in the trillions).
  • Catches absurd priors before running expensive inference.
  • Modern Bayesian workflow (Gelman et al.): prior predictive → fit → posterior predictive → repeat.
  • Essential in real modeling.
#bayesianPermalink & quiz →

Posterior predictive check — how do you use it?

hard
  • Simulate replicated datasets yrepy_{\mathrm{rep}} from posterior predictive; compare their summary statistics (mean, quantiles, distribution shape) to observed data.
  • If observed data looks like a plausible draw from yrepy_{\mathrm{rep}}, the model captures the data — otherwise, refine.
  • Formalized via Bayesian p-values: fraction of yrepy_{\mathrm{rep}} with statistic ≥ observed.
  • Standard diagnostic step in ArviZ, brms, Stan.
#bayesianPermalink & quiz →

When should you use a bandit instead of an A/B test?

hard
  • Bandit (Thompson sampling / UCB) automatically shifts traffic toward the winning arm during the test → minimizes regret.
  • Use when: (1) short-lived items (news, promotions) where you can't afford to send 50% traffic to the loser; (2) many arms (>10) with fast feedback; (3) time-decaying decisions.
  • Don't use when: (1) you need clean inference on effect size / statistical significance for stakeholder communication; (2) delayed feedback; (3) SUTVA-violating settings.
  • Bandits are for optimization, A/B is for learning.
#ab-testing#bayesianPermalink & quiz →

Practise Statistics Fundamentals

All 15 sectionsFlashcardsQuiz