25 interview questions on regression assumptions, each answered in full. Free to read, no account needed.
What are the Gauss-Markov assumptions for OLS? medium (1) Linearity: E [ Y ∣ X ] = X β E[Y\; \mid \;X]\; = \;X{\beta} E [ Y ∣ X ] = X β (2) Strict exogeneity: E [ ε ∣ X ] = 0 E[{\varepsilon}\; \mid \;X]\; = \;0 E [ ε ∣ X ] = 0 (3) Homoscedasticity: Var ( ε ∣ X ) = σ 2 \operatorname{Var}({\varepsilon}\; \mid \;X)\; = \;{\sigma}^{2} Var ( ε ∣ X ) = σ 2 (4) No autocorrelation: Cov ( ε i , ε j ) = 0 \operatorname{Cov}({\varepsilon}_{i}, \;{\varepsilon}_{j})\; = \;0 Cov ( ε i , ε j ) = 0 (5) No perfect multicollinearity. Under (1)-(5), OLS is BLUE (Best Linear Unbiased Estimator). Normality of ε is NOT required for BLUE, only for exact-sample t / F inference — CLT-robust for large n.
Heteroscedasticity — what breaks and how do you fix it? medium Var ( ε ∣ X ) \operatorname{Var}({\varepsilon}\; \mid \;X) Var ( ε ∣ X ) OLS β̂ stays unbiased and consistent, but SEs and t-tests are wrong → invalid inference. Fixes: (1) Robust ('sandwich', HC0-HC3) SEs — most common fix in econometrics. (2) Weighted Least Squares with weights ∝ 1 / Var ( ε ∣ X ) 1 / \operatorname{Var}({\varepsilon}\; \mid \;X) 1/ Var ( ε ∣ X ) (3) Cluster-robust SEs for panel / clustered data. (4) Transform the response (log for right-skewed outcomes). (5) Model the variance directly (GAMLSS, HGLM). Detect with Breusch-Pagan / White tests or residual-vs-fitted plots.
How do you detect and handle multicollinearity? medium Symptoms: unstable β̂ across subsets, huge SEs, opposite-sign coefficients from univariate expectation, high pairwise correlation. Diagnostics: V I F = 1 / ( 1 − R j 2 ) \mathrm{VIF}\; = \;1\; / \;(1\; - \;R_{j}^{2}) VIF = 1 / ( 1 − R j 2 ) Condition number of X'X (>30 problematic). Fixes: drop redundant features, PCA / partial-least-squares, ridge regression (shrinks coefficients), domain-based feature engineering. Doesn't hurt PREDICTION accuracy on new data much — mainly wrecks coefficient interpretability and inference. R 2 R^{2} R 2 R 2 R^{2} R 2 easy R 2 = 1 − R S S / T S S R^{2}\; = \;1\; - \;\mathrm{RSS} / \mathrm{TSS} R 2 = 1 − RSS / TSS Adjusted R 2 = 1 − ( 1 − R 2 ) ⋅ ( n − 1 ) / ( n − p − 1 ) R^{2}\; = \;1\; - \;(1\; - \;R^{2})\; \cdot \;(n - 1) / (n - p - 1) R 2 = 1 − ( 1 − R 2 ) ⋅ ( n − 1 ) / ( n − p − 1 ) Use adjusted R 2 R^{2} R 2 For prediction focus, use AIC / BIC / cross-validated R 2 R^{2} R 2 Warning: neither is a substitute for holdout evaluation.
Cook's distance, leverage, DFBETAS — define. hard Leverage h i i h_{\mathrm{ii}} h ii H = X ( X X ) − 1 H\; = \;X(XX)^{-1} H = X ( X X ) − 1 x i x_{i} x i Cook's distance: how much β̂ changes if point i is removed; > 4/n is a flag. D F B E T A S i \mathrm{DFBETAS}_{i} DFBETAS i DFFITS: change in fitted value. All identify points that unduly drive the fit — investigate before dropping (often signal of a real edge case or a bug in the data pipeline).
Standardized vs studentized residuals — the difference. hard Standardized: r i / ( s ⋅ ( 1 − h i i ) ) r_{i}\; / \;(s\; \cdot \; \sqrt (1\; - \;h_{\mathrm{ii}})) r i / ( s ⋅ ( 1 − h ii )) Studentized (internally): same but using s (uses point i to estimate σ). Externally studentized: uses s_(-i) (deletes point i from σ estimate) → follows t n − p − 1 t_{n - p - 1} t n − p − 1 Rule: |externally studentized| > 3 usually flags an outlier. Any regression diagnostics package reports both.
Logistic regression: why MLE and not OLS? medium Y ∈ {0, 1} → OLS predicts outside [0, 1] and residuals are heteroscedastic (Var = p(1-p)). Logistic: p = σ(x'β), maximize Σ y i y_{i} y i p i + ( 1 − y i ) p_{i}\; + \;(1\; - \;y_{i}) p i + ( 1 − y i ) log ( 1 − p i ) \operatorname{log}(1\; - \;p_{i}) log ( 1 − p i ) No closed form → IRLS or gradient descent. Softmax = generalization to k classes. Interpretation: exp ( β j ) = o d d s \operatorname{exp}({\beta}_{j})\; = \;\mathrm{odds} exp ( β j ) = odds x j x_{j} x j Foundation of most tabular binary classifiers and the top layer of neural nets.
Odds ratio — interpret and derive. medium Odds = p / (1 - p). Odds ratio ( O R ) = o d d s A / o d d s B (\mathrm{OR})\; = \;\mathrm{odds}_{A}\; / \;\mathrm{odds}_{B} ( OR ) = odds A / odds B In logistic regression, log ( o d d s ) = x \operatorname{log}(\mathrm{odds})\; = \;x log ( odds ) = x exp ( β j ) \operatorname{exp}({\beta}_{j}) exp ( β j ) x j x_{j} x j OR > 1: positive association; OR = 1: no association; OR < 1: negative. Not a ratio of probabilities. For rare outcomes, OR ≈ relative risk; for common outcomes, OR overstates RR.
GLM: link function and exponential family — connection. hard GLM: Y ~ exponential family with mean μ = E [ Y ∣ X ] {\mu}\; = \;E[Y\; \mid \;X] μ = E [ Y ∣ X ] Canonical links: Gaussian → identity, Bernoulli → logit, Poisson → log, Gamma → inverse. Fit by IRLS (iteratively re-weighted least squares). Advantages over OLS: handles counts, proportions, positive-only outcomes; correct variance structure baked in. Foundation of Poisson regression, logistic regression, gamma regression.
Poisson regression: setup and pitfalls. hard Y ~ Poisson(λ), log λ = x'β. Mean = variance = λ (equidispersion). Uses: count outcomes (clicks, visits, defects). Common pitfall: real data usually over-disperses (Var > Mean). Fix: (1) negative binomial regression (extra dispersion parameter), (2) quasi-Poisson (inflates SEs by dispersion factor). For zero-heavy counts, use zero-inflated Poisson / negative binomial or hurdle models. Always check dispersion after fitting.
Negative binomial regression — why use it? hard Adds dispersion parameter θ: Var = μ + μ 2 / θ \operatorname{Var}\; = \;{\mu}\; + \;{\mu}^{2} / {\theta} Var = μ + μ 2 / θ Handles over-dispersion (Var > Mean) which is the norm for real counts (customer visits, defect counts, rare events). Fit by MLE. When θ → ∞ → reduces to Poisson. Interpretation of exp ( β ) \operatorname{exp}({\beta}) exp ( β ) Modern default for count regression in industry.
What is quasi-likelihood? hard Rather than a full parametric distribution, specify only a mean-variance relationship: E[Y] = μ, Var ( Y ) = φ ⋅ V ( μ ) \operatorname{Var}(Y)\; = \;{\varphi}\; \cdot \;V({\mu}) Var ( Y ) = φ ⋅ V ( μ ) Quasi-scores solve exact same equations as MLE for the mean, but SEs use estimated dispersion φ. Quasi-Poisson: V(μ) = μ, φ estimated → same β as Poisson but corrected SEs. Cheap fix when the distribution isn't quite right — used extensively in industry counting problems.
What is a Generalized Additive Model (GAM)? hard g ( E [ Y ] ) = β 0 + f 1 ( x 1 ) + f 2 ( x 2 ) g(E[Y])\; = \;{\beta}_{0}\; + \;f_{1}(x_{1})\; + \;f_{2}(x_{2}) g ( E [ Y ]) = β 0 + f 1 ( x 1 ) + f 2 ( x 2 ) f j f_{j} f j Extends GLM by letting each feature enter through a non-parametric smooth. Interpretable (partial dependence plot per feature), captures non-linearity without explicit interactions. Fit via penalized splines (mgcv in R, pygam in Python). Modern uses: interpretable ML (competes with GBDT on tabular), forecasting (Prophet is a GAM).
Splines: knots, degrees of freedom, regularization. hard Spline = piecewise polynomial with continuity at knots. Cubic splines: 3rd-degree pieces, continuous through 2nd derivative. Natural spline: linear beyond boundary knots (avoids wild extrapolation). Knot placement: at quantiles of X (uniform coverage). Regularization: penalized splines add smoothing penalty λ * ∫ ( f ( x ) ) 2 (f(x))^{2} ( f ( x ) ) 2 Standard smoother in GAMs.
How do you include and interpret interactions in regression? medium Include x 1 ⋅ x 2 x_{1}\; \cdot \;x_{2} x 1 ⋅ x 2 Interpretation: coefficient of x 1 x_{1} x 1 x 2 x_{2} x 2 Center predictors before interaction to avoid multicollinearity between the main effect and the product term. For categorical × numeric interactions: separate slopes per category. Test with joint F-test or LRT. In production ML, tree-based methods learn interactions automatically — GLM interactions matter mostly for interpretation.
What is a mixed-effects model? hard y = Xβ + Zu + ε, where β = fixed effects (population-level slopes), u = random effects (subject / cluster deviations), u ~ N(0, G), ε ~ N ( 0 , σ 2 I ) N(0, \;{\sigma}^{2}I) N ( 0 , σ 2 I ) Uses: repeated measures on subjects, hierarchical data (students in schools, users in accounts), longitudinal panels. Handles within-cluster correlation → correct SEs. Fit by REML or Laplace-approximated MLE (lme4, statsmodels, brms). Foundation of hierarchical Bayesian modeling.
Random intercepts vs random slopes — the difference. hard Random intercept: each cluster has its own baseline level around the global β 0 {\beta}_{0} β 0 Random slope: each cluster's slope for a predictor varies around the global β. Test which you need with LRT or AIC. Rule: if the effect of X differs meaningfully across clusters (users respond differently to price changes), use random slopes. Random slopes typically require more clusters (~30+) to fit reliably.
Cluster-robust standard errors — when to use? hard Observations within clusters (schools, accounts, sessions) are correlated → default OLS SEs are too small → false positives. Cluster-robust ('Liang-Zeger', 'sandwich') SEs correct for within-cluster correlation. Rule: cluster at the level of treatment assignment or the highest level of correlation. Needs ≥ 30-50 clusters to be reliable — with few clusters, use wild cluster bootstrap. Standard in econometrics / A/B testing on clusters.
Mixed-effects vs cluster-robust SEs — which do you pick? hard Both handle cluster correlation, but differently. Mixed-effects: fully specifies covariance structure → efficient if the model is correct, but misspecification biases estimates. Cluster-robust SEs: model-agnostic → less efficient but robust to structure misspecification. Rule of thumb: use mixed effects when the hierarchical structure is real and you want cluster-specific predictions; use cluster-robust when you just want valid SEs on the fixed effects without committing to a full model.
In practice, how do you handle correlated features in regression? medium (1) Domain knowledge: keep the more meaningful one (age vs birthyear). (2) Combine: sum, difference, or PCA / PLS. (3) Ridge: L2 shrinks correlated coefficients toward each other without dropping. (4) Lasso: L1 arbitrarily picks one of a correlated pair. (5) Elastic net: middle ground — spreads coefficients across correlated groups. (6) Feature importance-based selection. Rule: if inference matters, ridge / elastic net + domain pruning; if pure prediction, tree ensembles usually don't care about collinearity.
Durbin-Watson test — what does it test? hard Tests for first-order autocorrelation in regression residuals: D W = Σ ( e t − e t − 1 ) 2 / Σ e t 2 \mathrm{DW}\; = \;{\Sigma}(e_{t}\; - \;e_{t - 1})^{2}\; / \;{\Sigma}e_{t}^{2} DW = Σ ( e t − e t − 1 ) 2 / Σ e t 2 Range ~ [0, 4]. DW ≈ 2: no autocorr. DW < 1.5: positive autocorr (common in time series). DW > 2.5: negative autocorr. Only detects lag-1 correlation — use Breusch-Godfrey for higher orders. When positive: fix with Cochrane-Orcutt, Prais-Winsten, GLS with AR(1) errors, or explicit ARIMA modeling.
Two-stage least squares (2SLS) — the recipe. hard Stage 1: regress T on instrument Z (and controls X) → predicted T̂. Stage 2: regress Y on T̂ (and X) → coefficient on T̂ is IV estimate. Equivalent to OLS-with-instrument formula β I V = Cov ( Z , Y ) / Cov ( Z , T ) {\beta}_{\mathrm{IV}}\; = \;\operatorname{Cov}(Z, \;Y) / \operatorname{Cov}(Z, \;T) β IV = Cov ( Z , Y ) / Cov ( Z , T ) Use robust or cluster SEs; standard 'ivreg2' / 'AER' / linearmodels packages compute correct SEs automatically. Never use naive 2-step OLS SEs from stage 2 — they're wrong. Your model reports R 2 = 0.92 R^{2}\; = \;0.92 R 2 = 0.92 medium Plenty. R 2 R^{2} R 2 R 2 R^{2} R 2 It is scale-free but not comparable across datasets, because it depends on the variance of the target: the same model achieves a high value on a heterogeneous population and a low one on a narrow slice. On time series with a trend, a high R 2 R^{2} R 2 It also says nothing about whether the residuals are well behaved, so a value of 0.92 is compatible with obvious structure the model missed. Report out-of-sample error in the units of the problem alongside it, and always look at the residual plot.
Two predictors correlate at 0.95. Does it matter? medium It depends entirely on what you want from the model. For prediction, collinearity is largely harmless: the fitted values and out-of-sample error are barely affected, because the pair jointly carries the information regardless of how the coefficient is split between them. For interpretation it matters a great deal, since the coefficients become unstable, their standard errors inflate, and the split between the two is nearly arbitrary, so a sign can flip when you add a few observations. Diagnose it with the variance inflation factor rather than pairwise correlation, which misses multi-way collinearity. Remedies are to drop one, combine them into a single index, or use ridge regularization, which trades a little bias for a large reduction in coefficient variance and is the standard answer when you must keep both.
You have 50,000 measurements from 200 patients. What is your sample size? hard Closer to 200 than to 50,000, because observations within a patient are correlated and each additional measurement from the same patient carries much less new information than a measurement from a new patient. Treating all 50,000 as independent shrinks the standard errors dramatically and produces confidence intervals far too narrow, which is how clustered data generates confident nonsense. The effective sample size depends on the intraclass correlation, and even a modest value collapses the benefit of many repeats per subject. The correct treatments are a mixed effects model with a random intercept per patient, cluster-robust standard errors, or aggregating to one summary per patient and analyzing those. Randomization must also happen at the patient level, otherwise the treatment is entangled with the cluster.