EasyDeepLearn
Supervised Learning · section 4 of 18

Naive Bayes, LDA & KNN

8 interview questions on naive bayes, lda & knn, each answered in full. Free to read, no account needed.

Why is Naive Bayes still a solid baseline for text classification?

easy
  • For bag-of-words / TF-IDF features it is (1) very fast to train and predict — a few matrix operations; (2) memory-efficient — you only store per-class word probabilities; (3) works well even with tiny labeled datasets since parameters are estimated per feature; (4) linear in the number of features; (5) resilient to irrelevant features because they contribute little to the score.
  • It's the default first baseline before trying logistic regression or transformers.
#naive-bayes#textPermalink & quiz →

QDA vs LDA — how do you choose between them?

medium
  • QDA relaxes LDA's shared-covariance assumption: each class has its own Σc{\Sigma}_{c}, so the decision boundary is quadratic.
  • Prefer QDA when class covariances clearly differ (visible from ellipsoid plots or per-class covariance estimates) and you have enough data per class to estimate them reliably.
  • Prefer LDA when data per class is limited, features are noisy, or class covariances are similar — the pooled Σ regularizes the estimate.
#lda-qda#discriminant-analysisPermalink & quiz →

How is LDA used for supervised dimensionality reduction?

medium
  • Beyond classification, LDA finds up to K-1 linear directions (K = number of classes) that maximize between-class variance while minimizing within-class variance.
  • Project the data onto these axes and you get a low-dimensional embedding that keeps class separation.
  • Unlike PCA (unsupervised, maximizes total variance), LDA uses the labels — so it's the go-to supervised dim-reduction for classification, e.g., visualizing a K-class problem in K-1 dimensions.
#lda-qda#discriminant-analysis#featuresPermalink & quiz →

How do you choose k in k-Nearest Neighbours?

easy
  • Cross-validation over odd values (avoids ties in binary problems).
  • Small k (1-3) = low bias, high variance — sensitive to noise.
  • Large k = smooth boundary, higher bias, lower variance.
  • A common heuristic is k ≈ sqrt(n).
  • Weighted KNN (weights inversely proportional to distance) reduces sensitivity to k.
  • For classification with C classes, k should not be a multiple of C (to break ties).
#knn#hyperparameter-tuningPermalink & quiz →

How do you choose the distance metric for KNN?

medium
  • Euclidean is the default for continuous features that have been standardized.
  • Manhattan (L1) is more robust to outliers and often used with sparse or high-dimensional data.
  • Cosine distance for text / normalized feature vectors where direction matters more than magnitude.
  • Mahalanobis when features have different scales and correlations (uses the inverse covariance).
  • Hamming for binary/categorical features.
  • Domain-specific metrics (e.g., edit distance for strings) when relevant.
#knn#featuresPermalink & quiz →

What's the point of distance-weighted KNN?

easy
  • Instead of every neighbour voting equally, weight closer neighbours more (e.g., weight = 1/distance or a Gaussian kernel).
  • This makes KNN less sensitive to k, gives a smoother decision boundary near class overlaps, and gives the closest point the strongest vote — useful when the local geometry matters.
  • In scikit-learn, weights='distance'.
#knnPermalink & quiz →

You trained Naive Bayes on balanced data but deploy on data where positives are 1%. What happens and what do you do?

hard
  • The prior P(c) in Bayes' rule shifts, so probabilities and thresholds are off.
  • Fixes: (1) refit priors from the deployment distribution while keeping the likelihoods (P(x∣c))(P(x \mid c)) from training — this is exactly what you need since P(c) is easy to estimate from deployment counts; (2) equivalently, adjust logit predictions by log⁡(Pdeploy(c)  /  Ptrain(c))\operatorname{log}(P_{\mathrm{deploy}}(c)\; / \;P_{\mathrm{train}}(c)) per class; (3) recalibrate on a small deployment-labelled sample if you can get one.
#naive-bayes#imbalance#calibrationPermalink & quiz →

How does KNN imputation work and when is it a good choice?

medium
  • For each row with missing values, find the k nearest complete rows using the observed features, then fill the missing values with a weighted average (regression) or mode (categorical) of those neighbours.
  • Handles multivariate structure automatically — better than mean/median for correlated features.
  • Downsides: quadratic in n (slow on large data), sensitive to scale (standardize first), degrades in high dimensions.
  • Great default for medium tabular data with < 100k rows.
#missing-data#knnPermalink & quiz →

Practise Supervised Learning

All 18 sectionsFlashcardsQuiz