35 interview questions on linear & regularized regression, each answered in full. Free to read, no account needed.
L1 vs L2 regularization — what's the difference?
easy- L2 (ridge) adds sum(w2) to the loss and shrinks weights smoothly toward zero.
- L1 (lasso) adds sum(∣w∣) and pushes weights exactly to zero, giving sparse solutions and doing feature selection.
- Elastic Net combines both.
- Use L1 when you want a sparse, interpretable model; use L2 when you want smooth shrinkage and stable coefficients.
Why is logistic regression called a linear model if it uses a sigmoid?
medium- The decision boundary is linear in the input features because the model is linear in the log-odds: logit(p)=wT x + b.
- The sigmoid only maps that linear score into [0, 1] for a probability.
- So the model is linear in parameters and features; only the link function is nonlinear.
What are the classical assumptions of linear regression?
medium- (1) Linearity: the conditional mean E[y∣x] is linear in the parameters.
- (2) Independence of errors.
- (3) Homoscedasticity: constant error variance across x.
- (4) Normality of errors (needed for exact inference on coefficients, not for point estimates).
- (5) No perfect multicollinearity between features.
- Violations don't ruin prediction but do ruin confidence intervals, p-values, and standard errors.
How do you interpret a coefficient in a multiple linear regression?
easy- The coefficient βj is the expected change in y per one-unit increase in xj while holding all other features fixed — the partial or ceteris-paribus effect.
- Sign gives the direction; magnitude depends on the scale of xj.
- If features are standardized, |βj∣is comparable across features (standardized beta / beta weight).
- Careful: 'holding others fixed' only makes sense if features are not perfectly collinear.
What is VIF and when do you worry about multicollinearity?
medium- Variance Inflation Factor of feature xj=1/(1−Rj2), where Rj2 is the R2 of regressing xj on the other features.
- VIF = 1 means no collinearity; VIF > 5 or 10 is a common warning threshold.
- High VIF inflates coefficient standard errors, makes individual coefficients unstable and hard to interpret, but doesn't harm overall prediction.
- Fixes: drop redundant features, combine them (PCA), or use ridge regression which is stable under collinearity.
How do you handle strong multicollinearity in a linear model?
medium- (1) Drop one of the correlated features (highest VIF first).
- (2) Combine them via PCA or domain-driven aggregation.
- (3) Switch to Ridge regression — the L2 penalty stabilizes coefficients under collinearity by shrinking correlated features together.
- (4) Collect more data if the collinearity is sample-specific.
- Note: multicollinearity does not bias predictions, only inference on individual coefficients, so if you only need predictions you can leave it alone.
How do you detect and fix heteroscedasticity?
hard- Heteroscedasticity = error variance depends on x.
- Detect with a residuals-vs-fitted plot (funnel shape), Breusch-Pagan or White test.
- Consequences: OLS coefficients remain unbiased but standard errors are wrong, so t-tests and CIs are invalid.
- Fixes: (1) transform y (log, sqrt, Box-Cox) if variance increases with the mean; (2) use weighted least squares with weights proportional to 1/variance; (3) use heteroscedasticity-robust (sandwich / White) standard errors.
Adding a feature raised R2. Does that mean the model improved?
easy- R2=1−SStotSSres is the fraction of variance explained; it never decreases when you add a feature, even a useless one.
- Adjusted R2 penalizes for the number of features and can decrease when you add a feature that doesn't help.
- Use adjusted R2 for model comparison across different feature counts; use R2 to describe fit at a fixed feature set.
- Both can be misleading for non-linear models; prefer AIC/BIC or CV.
Can R2 be negative? What does that mean?
medium- Yes. R2=1−SStotSSres compares your model's residual sum of squares to that of a constant-mean baseline.
- A negative R2 on a test set means your model is worse than predicting the mean everywhere — usually a sign of severe overfitting or distribution shift.
- On the training set with intercept, R2 is always in [0,1]; the negative case only appears on held-out data or intercept-free fits.
When would you use robust regression (Huber, RANSAC) instead of OLS?
medium- Use robust regression when the data contains outliers that shouldn't dominate the fit.
- OLS minimizes squared error, so a single large-residual point can pull the line dramatically.
- Huber loss is quadratic near zero and linear in the tails, giving less weight to outliers.
- RANSAC fits on random subsets and picks the fit with the most inliers — great when up to ~50% of points are contaminated.
- Quantile regression is another robust option that models a specific quantile.
What's the main risk of polynomial regression?
easy- High-degree polynomial features overfit dramatically: they can hit training points exactly while oscillating wildly between them (Runge's phenomenon).
- Predictions extrapolate very poorly outside the training range.
- Alternatives: use splines (piecewise low-degree polynomials with continuity), local polynomials (LOESS), regularized polynomials, or non-parametric models like trees.
- If you must use polynomials, standardize inputs and combine with L2 regularization.
When should you add interaction features to a linear model?
medium- Add xi⋅xj when the effect of one feature on y depends on the value of another and your model class can't discover this on its own (linear models can't).
- Common in marketing (channel × segment), medicine (treatment × age), and pricing (product × geography).
- Signs: residual patterns when you slice by a second feature, domain knowledge of moderation, or when a tree model beats a linear model but you want linear interpretability.
When does log-transforming the target help a regression model?
medium- Log-transform y when (1) y is strictly positive and right-skewed (revenue, counts, biological measurements, house prices), (2) residuals show heteroscedasticity that shrinks after the transform, (3) you care about relative rather than absolute errors — modeling log(y) is equivalent to modeling percentage errors.
- Watch out: predictions in log space need exp() to be brought back, and the expectation is biased (E[exp(z)]=exp(E[z])).
What do Box-Cox and Yeo-Johnson transforms do?
medium- Both are power transforms that find a lambda making the data approximately Gaussian and stabilizing variance.
- Box-Cox requires strictly positive data.
- Yeo-Johnson generalizes to any real values (including zero and negatives).
- They're useful for the target of a regression or heavily skewed features feeding a linear/Gaussian-based model.
- Not needed for tree ensembles (monotone-invariant) or for deep nets (BatchNorm handles it).
Why does Ridge regression give more stable coefficients than OLS?
medium- Ridge solves (XTX+λI) w=XT y instead of XT X w=XT y.
- Adding lambda I to the diagonal keeps the matrix invertible even when features are collinear or when p > n, and shrinks coefficients toward zero proportionally to their scale.
- The shrinkage introduces bias but drastically reduces variance — often a favourable tradeoff for prediction.
- Ridge does not zero-out coefficients, it just makes them small and stable.
Why does L1 (Lasso) produce sparse coefficients but L2 (Ridge) does not?
medium- Geometrically, the L1 constraint region is a diamond with corners on the axes, so the loss contour tends to intersect the constraint at a corner where some coefficients are exactly zero.
- The L2 region is a sphere, so intersections are typically off-axis — coefficients become small but not zero.
- Algebraically, L1's subgradient at zero is a range containing zero, so a solver can 'stick' coefficients at exactly zero.
When is Elastic Net better than pure Lasso or Ridge?
medium- Elastic Net combines L1 and L2 penalties: α⋅(ρ⋅∣w∣1+(1−ρ)⋅0.5⋅∣w∣22).
- Use it when (1) features are highly correlated — Lasso arbitrarily picks one and drops the rest, Elastic Net groups correlated features together via the L2 term; (2) you have more features than samples (p >> n) — Lasso caps the number of selected features at n.
- Tune rho on validation data.
How do you choose the regularization strength (lambda / alpha)?
medium- Cross-validation over a log-spaced grid is the standard approach — pick alpha minimizing average validation loss.
- Coefficient path plots (coefficient vs log alpha) help understand which features enter/exit at what strength.
- For efficiency, use algorithms that compute the whole path (LARS, coordinate descent with warm starts): scikit-learn's LassoCV / RidgeCV / ElasticNetCV do this out of the box.
- Standardize features first; alpha's meaningful scale depends on it.
Why must you standardize features before applying L1 or L2 regularization?
easy- The penalty is applied uniformly to all coefficients, but the natural scale of a coefficient depends on the scale of its feature (weight * feature = signal).
- Without standardization, a feature measured in millimetres would get a huge coefficient and be over-penalized versus one in metres.
- Standardizing (mean 0, std 1) puts coefficients on comparable scales, so alpha has a consistent effect and coefficient magnitudes become interpretable.
In one sentence, what does the LARS algorithm compute?
hard- Least Angle Regression is a stagewise algorithm that computes the entire regularization path of Lasso coefficients (from full L1 penalty down to zero penalty) in about the same cost as one OLS fit, by adding features to the active set in the order of highest correlation with the current residual and moving in a direction equiangular to them.
When would you use Group Lasso instead of standard Lasso?
hard- Group Lasso applies an L2 penalty within groups of coefficients and an L1 penalty across groups: entire groups get selected or dropped together.
- Use it when features come in natural groups — one-hot dummies of a categorical variable, spline basis functions, features from the same sensor.
- Standard Lasso would drop some dummies of a category, which is semantically odd; Group Lasso keeps or drops the whole category.
When is quantile regression more useful than mean (OLS) regression?
hard- Quantile regression models a specific conditional quantile of y (e.g., the median, 90th percentile) instead of the mean.
- Use it when (1) you care about tails, not the average — delivery time SLA, inventory sizing, risk quantiles; (2) the target distribution is asymmetric or heavy-tailed; (3) you want a prediction interval by fitting several quantiles.
- It minimizes the pinball (quantile) loss instead of MSE.
What is 'perfect separation' in logistic regression and how do you fix it?
hard- Perfect separation happens when a feature (or combination) fully separates the classes: the MLE for the corresponding weight goes to +/- infinity and the solver never converges (or gives massive coefficients with huge standard errors).
- Fixes: (1) add L2 regularization (drives the weight to a finite value); (2) use Firth's penalized likelihood; (3) drop or bin the separating feature.
- Common on tiny or heavily imbalanced datasets and one-hot rare categories.
L1 vs L2 in logistic regression — practical differences.
easy- Same intuition as in linear regression: L2 (default) shrinks all coefficients smoothly and stabilizes training under multicollinearity; L1 produces sparse coefficients and performs feature selection during training.
- In scikit-learn, use penalty='l1' with the 'liblinear' or 'saga' solver.
- L1 is nice for interpretable, sparse models on wide (many-feature) data; L2 is the default for prediction accuracy.
- Elastic-Net penalty combines both.
What is Laplace (additive) smoothing in Naive Bayes?
easy- It adds a small constant alpha to every count when estimating P(xj∣c) so no probability is ever exactly zero.
- Without smoothing, a single word never seen with class c in training would give P(document∣c)=0, killing the posterior.
- Alpha=1 is Laplace smoothing; smaller alpha (e.g., 0.01) is Lidstone smoothing.
- Choose it by cross-validation — it plays the role of a regularizer for the likelihood.
Pre-pruning vs post-pruning in decision trees — what's the difference and when do you use each?
medium- Pre-pruning stops the tree from growing during construction, via maxdepth, minsamplessplit, minsamplesleaf, or minimpuritydecrease.
- Fast and simple but you may stop too early (horizon effect).
- Post-pruning (cost-complexity or reduced-error pruning) grows the tree fully then removes subtrees using a held-out set or a complexity penalty.
- Usually gives a slightly better bias-variance tradeoff but requires the extra pass.
- Modern ensembles (RF, GBM) rely on pre-pruning + averaging, which is why deep individual trees are OK in them.
How does cost-complexity (CCP) pruning work?
hard- Grow the tree fully.
- Define Rα(T)=R(T)+α * |T|, where R(T) is the training error, |T∣is the number of leaves, and alpha ≥ 0 penalizes complexity.
- For each alpha, the subtree minimizing Rα is unique and nested — you get a full sequence of subtrees indexed by alpha.
- Pick alpha via cross-validation on the sequence.
- Scikit-learn exposes it via costcomplexitypruning_path and the ccpα argument.
How does early stopping work in gradient boosting and why is it important?
easy- Monitor validation loss after each tree is added; stop when it hasn't improved for earlystoppingrounds boosting iterations.
- Returns the model at the best iteration (not the last).
- Effect: automatic selection of nestimators, protection against overfitting, faster training when convergence happens early.
- Set nestimators very large (2000+) and let early stopping pick the actual count.
- Requires an evalset/validationdata — never use the test set for this.
How does the C parameter in an SVM affect the fit?
medium- C weights the misclassification penalty in the soft-margin loss (opposite of a typical regularization strength — larger C means less regularization).
- Large C: try hard to classify every training point correctly → narrow margin, complex boundary, high variance.
- Small C: tolerate more margin violations → wider margin, simpler boundary, high bias.
- Tune C on a log-spaced grid with CV.
- With an RBF kernel, C interacts with gamma — tune them jointly.
How does SVM adapt to regression (SVR)?
medium- SVR uses an epsilon-insensitive loss: residuals with∣y−yhat| ≤ epsilon are free (zero penalty), residuals outside are penalized linearly (like L1).
- Combined with the kernel trick, it fits a flexible non-linear function using only a subset of training points as support vectors.
- Tune C (violation penalty), epsilon (tube width), and gamma (for RBF).
- Works well on small/medium tabular problems where linear methods underfit and boosted trees are overkill.
What is Huber loss and why is it a compromise between MAE and MSE?
medium- Huber loss is quadratic for∣r| ≤ delta (like MSE) and linear beyond delta (like MAE), joined smoothly at delta.
- Combines MSE's differentiability near zero (good gradients for optimization) with MAE's robustness to outliers (linear tail doesn't blow up on rare huge errors). delta acts as an outlier threshold — typical values are 1⋅σy or set via CV.
- Used in robust regression, Huber-loss neural nets, and reinforcement learning (Huber for value updates).
What does the quantile (pinball) loss measure?
hard- For quantile tau in (0, 1), pinball loss(r) = tau * r if r > 0 else (tau - 1) * r, where r=y−yhat.
- Minimizing it fits the conditional tau-quantile of y given x. τ=0.5 → median (equivalent to MAE). τ=0.9 → 90th percentile prediction — great for delivery-time SLAs, inventory sizing, risk quantiles.
- Fit multiple quantiles to construct prediction intervals: e.g., 0.05 and 0.95 give a 90% interval.
- Used by quantile regression, LightGBM's quantile objective, and neural nets with pinball loss.
What is stability selection?
hard- Run your feature-selection method (usually Lasso) on many random subsamples of the data.
- Record how often each feature is selected.
- Keep only features selected in ≥ πthr (typically 60-90%) of subsamples.
- Rationale: any single Lasso fit is unstable — features come and go with the sample.
- Stability selection isolates features that are robustly informative across data resamples, with theoretical false-discovery-rate guarantees.
What are monotonic constraints in gradient boosting and when are they worth using?
hard- They force the predicted output to move in one direction as a feature increases.
- Useful when domain knowledge is certain: risk should not fall as debt rises, and price should not fall as square footage rises.
- Benefits are regulatory defensibility, more stable behaviour in sparse regions of the feature space, and mild regularization that often helps on small data.
- Cost is bias if the true relationship is not monotone, so check that the constraint does not degrade validation performance.
- Supported directly in XGBoost, LightGBM and CatBoost.
When would you use quantile regression instead of predicting the mean?
medium- When the decision needs a range rather than a point.
- Inventory planning cares about the 90th percentile of demand, not the average, because stocking to the mean stocks out half the time.
- Delivery estimates quote a pessimistic quantile so most promises are kept.
- Quantile regression minimizes the pinball loss, which weights under- and over-prediction asymmetrically, and gives you an interval by fitting several quantiles.
- It is also robust to outliers, since the median is not pulled by extreme values the way the mean is.
- Available natively in LightGBM and scikit-learn.