39 interview questions on fundamentals & theory, each answered in full. Free to read, no account needed.
What is the bias-variance tradeoff?
easy- Bias is error from wrong assumptions (underfitting).
- Variance is error from sensitivity to training data (overfitting).
- Total generalization error decomposes as E[(y−f^(x))2]=Bias[f^(x)]2+Var[f^(x)]+σ2 where σ2 is the irreducible noise.
- You reduce bias with more complex models or better features, and reduce variance with more data, regularization, or ensembling.
- The tradeoff means you can rarely minimize both at once.
How do you detect and fix overfitting?
easy- Detect it when training loss keeps dropping but validation loss rises, or when train accuracy is much higher than validation.
- Fixes: more data or augmentation, regularization (L1/L2, dropout), simpler model, early stopping, cross-validation, and ensembling.
- In deep learning, add dropout, weight decay, and monitor the validation curve.
What is k-fold cross-validation and when do you use stratified or grouped folds?
medium- Split the data into k folds; train on k-1 and validate on the held-out fold, then average.
- Use stratified k-fold for classification (preserves class ratios per fold).
- Use grouped k-fold when rows share an identity (same user, same patient) so the group never appears in both train and validation, preventing leakage.
- Use time-series split for temporal data.
Generative vs discriminative classifier — what's the difference?
medium- A discriminative model learns P(y∣x) directly (logistic regression, SVM, neural nets).
- A generative model learns P(x, y) or P(x∣y) and applies Bayes' rule to get P(y∣x) (Naive Bayes, GDA, GANs, diffusion for x).
- Discriminative models usually classify better; generative models can sample new data and often work with less data or missing features.
What is the curse of dimensionality?
medium- As dimensions grow, data becomes sparse: distances between points concentrate, the volume needed to cover the space grows exponentially, and models that rely on locality (k-NN, kernel methods) degrade.
- You need exponentially more data to maintain density.
- Remedies: feature selection, dimensionality reduction (PCA, UMAP), regularization, and models that assume structure (trees, deep nets with inductive biases).
Parametric vs non-parametric models — what's the difference?
easy- Parametric models have a fixed number of parameters that doesn't grow with the dataset (linear regression, logistic regression, GLMs, Naive Bayes).
- They make strong assumptions about the functional form and are fast + data-efficient when those assumptions hold.
- Non-parametric models grow in complexity with the data (KNN, decision trees, kernel methods, Gaussian processes).
- They are more flexible but need more data and are prone to overfit.
Why do we prefer simpler models when performance is equal?
easy- Occam's razor: among models with similar validation performance, the simpler one usually generalizes better and is cheaper to serve, easier to debug and monitor, and less prone to overfit noise.
- Complexity should be justified by clear gains.
- In practice this drives choices like linear over polynomial when residuals allow, smaller trees over deeper ones, and shallow ensembles over stacks-of-stacks.
What does the no free lunch theorem say for ML?
medium- Averaged over all possible problems, no learning algorithm is better than random guessing.
- Any advantage of an algorithm on one class of problems is offset by worse performance elsewhere.
- Practically, this means model choice must match the structure of your data.
- Trees dominate tabular data; CNNs dominate images; transformers dominate sequences — because their inductive biases match the domain, not because they are universally better.
In one sentence, what is PAC learning?
hard- Probably Approximately Correct learning formalizes when a hypothesis class can be learned from a finite sample: for any small epsilon and delta, with enough samples the algorithm returns a hypothesis with error at most epsilon with probability at least 1-delta.
- The required sample size scales with the complexity of the hypothesis class (VC dimension) and 1/epsilon.
What is VC dimension and why should you care?
hard- The Vapnik-Chervonenkis dimension of a hypothesis class is the largest number of points it can shatter (label with any possible +/- assignment).
- Higher VC dimension = more expressive class = higher risk of overfitting on small samples.
- Linear classifiers in d dimensions have VC dim d+1.
- VC bounds justify why you need more data as model capacity grows, and why regularization limits effective VC dim.
What is Empirical Risk Minimization?
medium- ERM is the standard learning principle: pick the hypothesis in your class that minimizes the average loss on the training sample, as a proxy for the true (unknown) expected loss.
- It works when the class is not too flexible relative to the sample size — otherwise you overfit.
- Structural risk minimization adds a complexity penalty (regularization) to trade off training loss and capacity.
Why do we care whether the loss landscape is convex?
medium- A convex loss (linear regression MSE, logistic regression log loss, SVM hinge, LASSO) has a single global minimum, so any local optimizer finds the best solution and results are reproducible.
- Non-convex losses (neural networks, matrix factorization, deep boosting) have many local minima and saddle points — training is sensitive to initialization, learning rate, and stochasticity.
- Convex problems admit strong theory (convergence, duality); non-convex ones rely on empirical tricks.
What are the roles of the training, validation and test sets?
easy- Training set: fit the model parameters.
- Validation set: tune hyperparameters, choose between models, early stop.
- Test set: give one final unbiased estimate of generalization on unseen data.
- If you touch the test set during model selection you leak information and overestimate performance.
- Typical split for medium datasets: 70/15/15 or 60/20/20; for large data, ~1-2% is enough per set.
- For small data use cross-validation on train+val and hold out a small test.
What does the IID assumption mean and when is it violated?
medium- IID = samples are independent and identically distributed, drawn from the same population.
- Standard ML theory (CV, generalization bounds, resampling) assumes IID.
- It breaks with time series (temporal correlation), repeated measurements per subject, spatial data (autocorrelation), and streaming data where the distribution drifts.
- Violations lead to over-optimistic error estimates; the fix is to use time-series splits, grouped CV, or drift-aware evaluation.
Why is the base rate of the positive class critical to know?
medium- The base rate is the prior probability of the positive class in the population you'll deploy on.
- It sets the floor for any metric — 99% accuracy is trivial on a 1%-positive problem by always predicting negative.
- It affects PPV (precision) via Bayes rule even at fixed TPR/FPR.
- Always report the class balance with your metrics and, when the deployment base rate differs from training, calibrate probabilities or adjust the decision threshold.
What is inductive bias and why does every model have one?
medium- Inductive bias is the set of assumptions a learning algorithm uses to generalize from finite data to unseen examples.
- Without an inductive bias, any function consistent with the training data is equally likely — no learning is possible.
- Examples: linear regression assumes a linear conditional mean; CNNs assume locality and translation equivariance; trees assume axis-aligned splits; k-NN assumes locality in feature space.
- Choosing a model = choosing an inductive bias that matches your data.
What is the generalization gap and how do you shrink it?
medium- Generalization gap = expected test loss minus training loss.
- A large gap means overfitting.
- Shrink it with more data, stronger regularization (L2, dropout, weight decay), data augmentation, simpler models, ensembling, or early stopping.
- It relates to the model's effective capacity and the sample size — the classic result is that gap decreases like sqrt(capacity / n).
Why does OLS have a closed-form solution but logistic regression doesn't?
medium- OLS minimizes ||y - Xw||^2 which is quadratic in w — setting the gradient to zero yields the normal equations w=(XTX)−1 XT y, a closed form.
- Logistic regression minimizes the log-loss which is convex but not quadratic (log of a sigmoid); its gradient has no closed-form root, so we solve it iteratively (IRLS, Newton, or SGD).
- Both are convex, so the iterative solution reaches the unique optimum.
What's the Bayesian interpretation of Ridge regression?
hard- Ridge regression is the maximum a posteriori (MAP) estimate under a Gaussian prior on the weights: w ~ N(0,σ2/λ).
- Equivalently, adding an L2 penalty on w is equivalent to assuming the coefficients are a priori small.
- Lasso corresponds to a Laplace (double-exponential) prior on w, which has more probability mass at zero and heavier tails, explaining its sparsity behavior.
What is a Generalized Linear Model and when is it the right tool?
medium- A GLM has three ingredients: (1) a linear predictor η=w⊤x; (2) a link function g relating η to E[y∣x], so g(μ)=η; (3) a distribution from the exponential family for y (Gaussian → linear regression, Bernoulli → logistic, Poisson → Poisson regression, Gamma → Gamma regression).
- Use a GLM when your target has a known non-Gaussian distribution but the mean-predictor relationship is approximately linear on the right scale.
Why do we use cross-entropy (log loss) instead of MSE for classification?
medium- Cross-entropy matches the maximum-likelihood estimator for a Bernoulli / categorical target and has a well-shaped gradient when combined with sigmoid/softmax outputs — large errors give large gradients so the model learns fast.
- MSE on top of sigmoid gives a very small gradient when predictions are confidently wrong (the sigmoid saturates), so training is slow and can get stuck.
- Cross-entropy is also convex in the logits for logistic regression, MSE-with-sigmoid is not.
How is logistic regression fit in practice?
medium- Two main approaches: (1) Iteratively Reweighted Least Squares (IRLS) — an implementation of Newton-Raphson on the log-likelihood.
- Each iteration solves a weighted least squares problem with weights = p(1-p).
- Converges in ~10-20 iterations for well-conditioned problems.
- (2) First-order methods: L-BFGS, SAG, SAGA, SGD.
- Better for large datasets and L1 penalty. scikit-learn uses 'lbfgs' by default; 'liblinear'/'saga' for L1.
Your scikit-learn logistic regression warns 'lbfgs failed to converge'. What do you do?
medium- Common fixes: (1) standardize features — most convergence issues come from ill-scaled features; (2) increase maxiter (e.g., 1000); (3) increase regularization C down (stronger penalty smooths the loss); (4) switch solver ('saga' or 'liblinear' for L1, 'newton-cg' for very small data); (5) check for perfect / near-perfect separation and add L2 or drop the offending feature; (6) verify the target is not constant.
How does Bayes' theorem drive Naive Bayes classification?
easy- For class c and features x, P(c∣x) ∝ P(x∣c)⋅P(c).
- Naive Bayes assumes features are conditionally independent given the class: P(x∣c)=prodj P(xj∣c).
- This lets you estimate each P(xj∣c) separately (very fast, needs little data per parameter).
- At prediction time you pick the class with the largest posterior score.
- Priors P(c) are usually class frequencies; likelihoods P(xj∣c) come from a chosen distribution (Gaussian, Multinomial, Bernoulli).
The 'naive' independence assumption is almost always false. Why does Naive Bayes still work?
medium- Because classification only requires that the argmax over classes be correct — not that the posterior probabilities themselves be accurate.
- Even when the independence assumption is violated, the decision boundary from NB can be close to optimal on many problems.
- Its probabilities, however, are usually badly miscalibrated (too extreme, close to 0 or 1).
- If you need calibrated probabilities, apply Platt or isotonic calibration on top.
Why do we compute Naive Bayes scores in log space?
easy- The product prodj P(xj∣c) can be astronomically small (underflow) when there are many features (e.g., thousands of words in a document).
- Taking logs turns the product into a sum: log P(c)+sumj log P(xj∣c).
- Sums are numerically stable, and the argmax is preserved because log is monotonic.
- All practical NB implementations work in log space.
What are the core assumptions behind Linear Discriminant Analysis (LDA)?
medium- LDA assumes each class-conditional distribution is Gaussian with class-specific mean but a *shared* covariance matrix Σ.
- Under this assumption, the log-posterior is linear in x — the decision boundary between any two classes is a hyperplane.
- It also assumes features are continuous.
- When assumptions hold, LDA is close to Bayes-optimal and needs little data because it estimates fewer parameters than QDA (one Σ instead of one per class).
Why does KNN degrade badly in high dimensions?
medium- In high dimensions, pairwise distances between random points concentrate — the ratio (max - min) / min distance approaches 0.
- Every point becomes roughly the same distance from every other, so the concept of 'nearest' loses meaning and KNN votes become dominated by noise.
- Remedies: apply dimensionality reduction (PCA, autoencoder) first, use domain-specific distance metrics, or switch to a model with stronger inductive bias (tree ensembles).
How do KD-trees and ball-trees speed up KNN, and when do they stop helping?
medium- Both are spatial index structures that let you skip large portions of the training set during queries.
- KD-trees split axis-aligned; efficient for low-dimensional data (~≤ 20 dims) with continuous features.
- Ball-trees split with hyperspheres; better for non-Euclidean metrics and moderate dimensions.
- Both degrade to brute force as dimensions grow (~50+), because the pruning becomes ineffective — then approximate methods (HNSW, FAISS, LSH) are the practical way to scale.
Why not just fit a linear regression for a binary label (linear probability model)?
medium- You can — sometimes it even works reasonably as a quick baseline — but three problems bite in practice: (1) predicted probabilities aren't bounded to [0, 1]; (2) errors are heteroscedastic (variance = p(1-p) depends on x), so OLS standard errors are wrong; (3) it isn't the MLE for Bernoulli data.
- Logistic regression fixes all three via the sigmoid link and the Bernoulli likelihood — that's why it's the default.
What's the difference between probability, odds, and log-odds?
easy- Probability p in [0, 1].
- Odds = p / (1 - p) in [0, ∞): 'how many times more likely than not'.
- Log-odds (logit) = log(p / (1 - p)) in (-∞, ∞): a symmetric, unbounded scale, useful for modelling.
- Logistic regression models the log-odds linearly in x, exactly because log-odds live on the real line (matching a linear predictor) and the inverse (sigmoid) automatically bounds probabilities to [0, 1].
Why is a single decision tree considered a 'high-variance' model?
medium- A small change in the training data can produce a completely different tree: the first split cascades, so if a different feature is chosen at the top, everything below changes.
- This makes individual trees unstable — predictions vary a lot with resampling.
- Bagging (Random Forest) and boosting exploit this: averaging many independent (or corrected-residual) trees reduces variance dramatically without hurting bias.
Why do decision trees (and boosted trees) fail to extrapolate?
medium- A tree predicts by returning the leaf value; leaves are bounded by the observed training range.
- If a test feature is beyond the training range, the tree returns whatever leaf value corresponds to the closest region — no extrapolation.
- This is fine on tabular data with stable distributions, but disastrous for time series with trend, or physics-style problems where extrapolation is the point.
- Use linear models, splines, or neural nets when extrapolation matters.
What is the out-of-bag (OOB) score in Random Forests?
medium- Each bootstrap sample leaves out ~37% of the training data (untouched by that tree).
- For each training point x, aggregate predictions from the trees where x was NOT in-bag; compare to y.
- This gives an internal estimate of generalization error at zero extra cost, roughly equivalent to a k-fold CV score, and lets you tune nestimators without a separate held-out set.
- Enable with oobscore=True in scikit-learn's RandomForestClassifier/Regressor.
What is a support vector, and why does the SVM only depend on them?
medium- Support vectors are the training points that lie on the margin or inside it (or misclassified for soft margins) — points with non-zero Lagrange multipliers in the dual problem.
- The optimal hyperplane is a linear combination of these support vectors alone: adding or removing non-SV points doesn't change the solution.
- This makes SVMs memory-efficient at inference (store only SVs) and gives an implicit form of feature/data selection.
Why don't kernel SVMs scale well to millions of samples?
medium- The dual problem has O(n2) memory (kernel matrix) and O(n2) to O(n3) training complexity.
- For n = 1M, that's a trillion entries — impossible.
- Workarounds: (1) linear SVM (LinearSVC / liblinear) which is O(nd) via primal optimization; (2) approximate kernels via random Fourier features / Nyström before feeding to a linear SVM; (3) switch entirely to gradient-boosted trees or neural nets, which routinely handle millions of samples.
What is 'deviance' and why is it used to evaluate GLMs?
hard- Deviance = -2 * (log-likelihood of your model - log-likelihood of the saturated model).
- Analog of squared error for non-Gaussian likelihoods: Poisson deviance for count data, Tweedie deviance for insurance-style claims, binomial deviance for logistic regression.
- It's the natural loss for the corresponding GLM and gives coherent training + evaluation.
- Modern GBM libraries expose Poisson / Tweedie / gamma deviance losses directly — pick the one matching your target distribution.
You must pick ONE metric for your model. How do you decide?
medium- Start from the business objective, not the model: what does 'error' cost, and to whom?
- Classification with asymmetric costs → expected cost or F-beta with the appropriate beta.
- Rare-positive detection → PR-AUC / recall at fixed precision.
- Ranking → NDCG or MAP.
- Probabilistic forecasts → log loss or Brier + calibration.
- Regression with tail-critical decisions → quantile loss.
- Regression with symmetric costs and Gaussian errors → RMSE.
- Always report multiple metrics; the *primary* one is the one you optimize and stop-arg on.
An interviewer asks you to build a churn model. What do you do before touching any algorithm?
easy- Define the prediction problem precisely: what counts as churn, and over what horizon.
- Fix the prediction time, so every feature is something you would actually have at that moment.
- Then build the dumbest possible baseline: predict the base rate, or a single rule like 'no login in 30 days'.
- That baseline is what every later model must beat, and it takes an hour instead of a week.
- Only then pick a model family, and prefer gradient boosting on tabular data.
- Candidates who jump straight to model choice usually end up with leakage or a target that nobody agreed on.