EasyDeepLearn
All categoriesStudy
Lesson

Unsupervised Learning

214 concepts~321 min read9 sections

Clustering, dimensionality reduction, density estimation and anomaly detection.

Interview flashcardsTake the quiz
Filter by difficulty
Filter by concept tag

Introduction

Unsupervised learning finds structure in data without labels. The three big families are clustering (grouping similar points), dimensionality reduction (compressing while preserving structure), and density / anomaly modeling (learning what "normal" looks like).

Modern representation learning — contrastive methods, autoencoders, self-supervised pretraining — sits on the same foundations and powers today's foundation models. Interviewers will probe how you'd choose k, why distances break in high dimensions, and when to reach for DBSCAN over k-means.

01

Clustering basics

69 concepts

How does k-means work and what are its main limitations?

easy
  • Initialize k centroids (k-means++ is best).
  • Assign each point to the nearest centroid, then recompute centroids as the mean of assigned points; repeat until convergence.
  • Limitations: you must choose k, it assumes spherical clusters of similar size, it's sensitive to scale and outliers, and it can get stuck in local minima (mitigated by multiple restarts).
Permalink

How do you choose the number of clusters k?

easy
  • Use multiple signals, not one: the elbow method on inertia (look for the bend), the silhouette score (higher is better, up to 1), the gap statistic, and domain knowledge.
  • For density-based clustering (DBSCAN, HDBSCAN) you don't set k at all — you set density parameters instead.
Permalink

When would you pick DBSCAN over k-means?

medium
  • Pick DBSCAN when clusters are non-convex or of very different densities, when you don't want to specify k, and when you want an explicit notion of noise/outliers. k-means struggles with irregular shapes and forces every point into a cluster.
  • DBSCAN needs two parameters: eps (neighborhood radius) and min⁡samples\operatorname{min}_{\mathrm{samples}}.
Permalink

What does the silhouette score measure?

medium
  • For each point, silhouette = (b - a) / max(a, b), where a is the mean distance to points in its own cluster and b is the mean distance to points in the nearest other cluster.
  • Values close to +1 mean well-separated clusters, near 0 means overlapping, negative means misassigned.
  • The overall score is the mean across all points.
Permalink

What is hierarchical clustering and when is it useful?

medium
  • It builds a tree (dendrogram) of nested clusters, either agglomerative (bottom-up: merge closest clusters) or divisive (top-down).
  • You cut the tree at a chosen height to get k clusters.
  • Useful when you want to explore cluster structure at multiple scales, or when you don't know k in advance and want a visual.
  • Downside: O(n2)O(n^{2}) memory, so it doesn't scale beyond ~10k points.
Permalink

Gaussian Mixture Model vs k-means — what's the difference?

medium
  • k-means gives hard assignments and assumes spherical, equal-variance clusters.
  • A GMM models each cluster as a multivariate Gaussian and returns soft (probabilistic) memberships; it handles ellipsoidal clusters and different covariances.
  • GMM is fit with the EM algorithm.
  • Use GMM when clusters overlap, have different shapes, or when you need probabilities.
Permalink

What is linkage in agglomerative clustering?

hard
  • Linkage defines the distance between two clusters.
  • Single linkage = min pairwise distance (chains, elongated clusters).
  • Complete linkage = max pairwise distance (compact clusters, sensitive to outliers).
  • Average linkage = mean pairwise distance.
  • Ward linkage minimizes within-cluster variance (produces balanced, compact clusters — most common default).
Permalink

Why does clustering degrade in high dimensions?

medium
  • In high dimensions, pairwise distances concentrate — points look almost equidistant — so distance-based clustering loses discriminative power.
  • Also, irrelevant features add noise.
  • Fixes: dimensionality reduction (PCA/UMAP) before clustering, feature selection, or use models less reliant on Euclidean distance (e.g., subspace clustering).
Permalink

How does k-means++ initialization work?

medium
  • Pick the first centroid uniformly at random.
  • For each subsequent centroid, sample point x with probability proportional to D(x)2D(x)^{2} where D(x) is the distance to the nearest already-chosen centroid.
  • Result: centroids spread out over the data → far better starting point than random.
  • Expected O(log k) approximation to optimal SSE.
  • Default in scikit-learn and Spark MLlib.
Permalink

When is mini-batch k-means preferred?

medium
  • Large datasets (millions of points).
  • Sample small random mini-batches (~1000 points), update centroids incrementally toward batch means with a learning rate.
  • Slightly worse SSE than full k-means, but 10-100x faster and streaming-friendly.
  • Standard for big-data clustering (Spark, scikit-learn's MiniBatchKMeans).
  • For tabular data > 100k rows, prefer it over standard k-means.
Permalink

k-medoids vs k-means — the difference.

medium
  • k-medoids uses actual data points as cluster representatives (medoids) instead of arithmetic means.
  • Robust to outliers (the median is more robust than the mean).
  • Works with any distance/dissimilarity — not just Euclidean (categorical, custom).
  • Cost: O(k(n−k)2)O(k(n - k)^{2}) per iteration (PAM algorithm) — slower than k-means.
  • Use for outlier-heavy data or non-Euclidean distances.
Permalink

What is fuzzy c-means?

hard
  • Each point has a membership degree uiju_{\mathrm{ij}} ∈ [0,1] for every cluster (rows sum to 1).
  • Centroids weighted by uijmu_{\mathrm{ij}}^{m} for a fuzzifier m > 1 (m=2 typical).
  • Update alternates between soft assignments and weighted means.
  • Use when clusters overlap naturally and hard assignment loses information.
  • Sits between k-means (m→1: hard) and uniform (m→∞).
Permalink

HDBSCAN — how does it improve on DBSCAN?

hard
  • Runs DBSCAN over all density levels and extracts the most stable clusters — avoids picking a single eps.
  • Handles clusters of varying density (DBSCAN's biggest failure mode).
  • Output: cluster labels + outlier scores + soft memberships.
  • Standard practice for real-world clustering when densities are heterogeneous (customer segments, sensor data).
  • Slightly slower but no eps tuning needed.
Permalink

OPTICS — what does it produce and how do you use it?

hard
  • Computes a reachability plot ordering all points by density-reachability.
  • Reveals cluster hierarchy visually as valleys in the plot.
  • Extract clusters at any density threshold from a single run.
  • Slower than DBSCAN (O(n log n) with index) but more flexible.
  • Used as an exploratory tool when you don't know cluster densities in advance.
Permalink

Spectral clustering — the core idea.

hard
  • Build similarity graph on data (k-NN or ε-neighborhood).
  • Compute graph Laplacian L.
  • Find eigenvectors of the k smallest eigenvalues → embed points into RkR^{k}.
  • Run k-means in this embedding.
  • Captures non-convex, manifold-shaped clusters that k-means can't handle.
  • Cost: O(n3)O(n^{3}) eigendecomposition or O(n2)O(n^{2}) with Nyström — scales to ~50k.
  • Modern uses: community detection in graphs, image segmentation.
Permalink

Affinity propagation — how does it work?

hard
  • Message-passing algorithm exchanging 'responsibility' and 'availability' messages between all pairs of points until exemplars emerge.
  • No k needed — 'preference' parameter influences the number of clusters.
  • Handles non-convex clusters, but O(n2)O(n^{2}) memory + slow.
  • Use for small-to-medium exploratory clustering when k is unknown and interpretable exemplars matter (e.g., customer archetypes).
Permalink

Mean shift clustering — mechanism and use case.

hard
  • Non-parametric mode-seeking.
  • For each point, repeatedly shift it toward the local mean within a bandwidth-radius kernel until convergence.
  • Points converging to the same mode form a cluster.
  • No k needed — clusters = number of density modes.
  • Bandwidth is the key parameter.
  • Uses: image segmentation, mode tracking.
  • Slow (O(n2)  per  iteration)(O(n^{2})\;\mathrm{per}\;\mathrm{iteration}) but robust to cluster shape.
Permalink

BIRCH — when to use it?

hard
  • Streaming / very large dataset clustering.
  • Builds a CF-Tree (Cluster Feature summaries: N, LS, SS) in a single pass over data — leaves become micro-clusters.
  • Then apply hierarchical or k-means clustering on the leaves.
  • Memory-efficient, near-linear in n, streaming-friendly.
  • Weakness: assumes spherical clusters (built-in radius threshold).
  • Standard for terabyte-scale exploratory clustering.
Permalink

EM for a GMM — one iteration explicitly.

hard
  • E-step: for each point xix_{i} and cluster j, γij  =  πj{\gamma}_{\mathrm{ij}}\; = \;{\pi}_{j} N(xi  ∣  μj,  Σj)  /  ΣkN(x_{i}\; \mid \;{\mu}_{j}, \;{\Sigma}_{j})\; / \;{\Sigma}_{k} πk{\pi}_{k} N(xi  ∣  μk,  Σk)N(x_{i}\; \mid \;{\mu}_{k}, \;{\Sigma}_{k}) (responsibilities).
  • M-step: update πj  =  Σi{\pi}_{j}\; = \;{\Sigma}_{i} γij  /  n{\gamma}_{\mathrm{ij}}\; / \;n, μj  =  Σi{\mu}_{j}\; = \;{\Sigma}_{i} γij{\gamma}_{\mathrm{ij}} xi  /  Σix_{i}\; / \;{\Sigma}_{i} γij{\gamma}_{\mathrm{ij}}, Σj  =  Σi{\Sigma}_{j}\; = \;{\Sigma}_{i} γij{\gamma}_{\mathrm{ij}} (xi  −  μj)(x_{i}\; - \;{\mu}_{j})(xi  −  μj)(x_{i}\; - \;{\mu}_{j})' / Σi{\Sigma}_{i} γij{\gamma}_{\mathrm{ij}}.
  • Converges to local max of log-likelihood.
  • Init sensitive → use k-means for starting points.
Permalink

GMM covariance types — which do you pick?

hard
  • Full: each cluster has its own general Σj{\Sigma}_{j} — most flexible, most parameters (p2)(p^{2}).
  • Tied: all clusters share one Σ.
  • Diagonal: axis-aligned ellipsoids (p params per cluster).
  • Spherical: Σ  =  σ2{\Sigma}\; = \;{\sigma}^{2} I (1 param) — closest to k-means.
  • Rule: use full for small p, tied or diagonal for high p to avoid overfitting.
  • Cross-validate log-likelihood or BIC to choose.
Permalink

How does BIC choose k in GMM?

medium
  • BIC = -2 log L + p log n where p = number of free parameters, n = dataset size.
  • Lower is better.
  • Balances fit (log L) against complexity (p log n).
  • For GMM, p grows with k and covariance type.
  • Standard practice: fit GMM at k = 1..KmaxK_{\mathrm{max}}, plot BIC vs k, pick the minimum (or the elbow).
  • More reliable than AIC for clustering.
Permalink

How does the gap statistic work?

hard
  • Compare observed within-cluster dispersion WkW_{k} to expected WkW_{k} under a null reference distribution (uniform in the bounding box).
  • Gap(k)  =  E⋅[log⁡  Wk⋅]  −  log⁡\mathrm{Gap}(k)\; = \;E \cdot [\operatorname{log}\;W_{k} \cdot ]\; - \;\operatorname{log} WkW_{k}.
  • Pick smallest k such that Gap(k) ≥ Gap(k+1)  −  sk+1\mathrm{Gap}(k + 1)\; - \;s_{k + 1}.
  • More principled than the elbow; handles cluster-vs-no-cluster case (returns k=1 when data is unstructured).
  • Costly (needs B ~ 20 reference simulations).
Permalink

Why can the elbow method fail?

medium
  • (1) No clear elbow — smooth curve, subjective.
  • (2) Different features scales dominate SSE → misleading k.
  • (3) Sees the biggest cluster count as always better (SSE monotonically decreases).
  • Use with silhouette or gap statistic as tiebreakers.
  • In practice, teams combine 3 metrics + domain knowledge; blindly trusting elbow SSE is a common bug in production clustering pipelines.
Permalink

Calinski-Harabasz index — what is it?

medium
  • CH(k)  =  [Tr(Bk)  /  (k−1)]  /  [Tr(Wk)  /  (n−k)]\mathrm{CH}(k)\; = \;[\mathrm{Tr}(B_{k})\; / \;(k - 1)]\; / \;[\mathrm{Tr}(W_{k})\; / \;(n - k)] — ratio of between-cluster dispersion to within-cluster dispersion (higher = better).
  • Analogue to an F-statistic for clustering.
  • Fast, no ground truth needed.
  • Bias: tends to favor higher k.
  • Complements silhouette.
  • Combined with domain, one of the standard internal indices.
Permalink

Davies-Bouldin index — how is it computed?

hard
  • DB = (1/k) Σi{\Sigma}_{i} max⁡j≠i\operatorname{max}_{j \ne i} [(si  +  sj)  /  dij][(s_{i}\; + \;s_{j})\; / \;d_{\mathrm{ij}}] where si  =  within−clusters_{i}\; = \;\mathrm{within} - \mathrm{cluster} scatter, dij  =  distanced_{\mathrm{ij}}\; = \;\mathrm{distance} between centroids.
  • Lower is better.
  • Rewards compact + well-separated clusters.
  • Fast, ground-truth-free.
  • Weak on non-convex clusters (assumes centroid representation).
  • Standard internal metric alongside silhouette.
Permalink

Adjusted Rand Index vs NMI — external cluster metrics.

medium
  • Both compare a clustering to ground-truth labels (when available).
  • ARI = (RI - E[RI]) / (max RI - E[RI]) → 0 for random, 1 for perfect match, can be negative.
  • Corrects for chance.
  • NMI = mutual information normalized to [0,1] — measures how much info the clustering carries about labels.
  • AMI = adjusted NMI (chance-corrected).
  • Rule: report both — they disagree in edge cases.
Permalink

How do you cluster mixed numeric + categorical data?

hard
  • (1) Gower distance — averages appropriate distances per feature type (Manhattan for numeric, matching for categorical).
  • (2) k-prototypes — extends k-means with categorical dissimilarity.
  • (3) Embed categorical as one-hot / target encoding, scale numerics, then k-means (loses interpretability).
  • (4) UMAP with metric='gower' → then cluster in embedding.
  • Most production tabular clustering uses Gower + PAM or k-prototypes.
Permalink

Why do you scale features before k-means?

easy
  • Euclidean distance is scale-sensitive: a feature with 1000-range dominates one with 0-1 range.
  • Standardize (z-score) so features contribute equally.
  • Exception: if scale carries meaning (spatial coordinates in meters), don't scale.
  • Also: outliers stretch scale → robust scaling (median / IQR) can be better.
  • Missing this is one of the most common bugs in first-time clustering pipelines.
Permalink

Cosine similarity vs Euclidean — when do you use cosine?

medium
  • Cosine similarity = x · y  /  (∣∣x∣∣  ∣∣y∣∣)y\; / \;( \mid \mid x \mid \mid \; \mid \mid y \mid \mid ) — measures angle, ignores magnitude.
  • Use when magnitude is irrelevant: TF-IDF vectors (doc length varies), sentence embeddings, user-item preferences.
  • Use Euclidean when magnitude matters (physical measurements, absolute values).
  • In practice: L2-normalize embeddings then use Euclidean → equivalent to cosine but works in standard k-means / FAISS indexes.
Permalink

How do outliers affect k-means and how do you handle them?

medium
  • Outliers pull centroids toward themselves (mean is non-robust) → distort cluster boundaries.
  • Handle: (1) pre-cluster outlier removal via IsolationForest / z-score, (2) use k-medoids (median-like), (3) use DBSCAN which explicitly flags outliers as noise, (4) trim tails / winsorize before clustering.
  • Always plot the largest / smallest per-cluster distances after fitting — extreme values are usually outliers or mistakes in the pipeline.
Permalink

How do you assess cluster stability?

hard
  • (1) Bootstrap: resample data, re-cluster, measure Jaccard overlap between cluster memberships.
  • (2) Random-init variability: many k-means restarts, measure ARI between runs.
  • (3) Perturbation: add small noise, check whether cluster boundaries shift.
  • Stable clusters survive resampling / perturbation.
  • Standard sanity check before shipping a customer segmentation.
Permalink

What is consensus clustering?

hard
  • Run clustering many times (varying algorithm, k, subsample, init).
  • Build co-clustering matrix Mij  =  fractionM_{\mathrm{ij}}\; = \;\mathrm{fraction} of runs where i and j were in the same cluster.
  • Cluster M (usually hierarchical) → consensus assignment.
  • Standard in bioinformatics for gene expression subtype discovery.
  • More robust than any single run; helps identify natural k when peaks in M are stable across k.
Permalink

Online k-means — how does it work?

hard
  • For each incoming point, find nearest centroid and update it: μj{\mu}_{j} ← μj  +  η{\mu}_{j}\; + \;{\eta} (x  −  μj)(x\; - \;{\mu}_{j}) with decaying learning rate.
  • Streaming-friendly, constant memory.
  • Sensitive to arrival order and cluster drift.
  • Modern alternative: mini-batch k-means; production streaming systems (Kafka + Flink) prefer mini-batch for robustness.
  • Concept drift → reset centroids periodically.
Permalink

'Cluster then classify' — when does it help?

medium
  • Add cluster label as an extra feature to a supervised model.
  • Sometimes lifts performance on data with strong latent groups (customer segments, region clusters) — the label encodes a non-linear grouping the model wouldn't easily learn.
  • But: usually replaces with target-encoded categoricals and interaction features that a GBDT can learn directly.
  • Test with CV — often marginal or negative for modern tree models.
Permalink

How do you cluster time series?

hard
  • (1) Feature extraction: extract statistics (mean, variance, autocorrelation, FFT bands) and cluster the feature vectors.
  • (2) Dynamic Time Warping (DTW) distance + k-medoids / hierarchical.
  • (3) Shape-based: k-Shape (normalized cross-correlation).
  • (4) Embed with sequence autoencoder / TS2Vec then cluster embeddings.
  • Rule: DTW for shape similarity, feature-based for interpretability, embedding-based for large heterogeneous series.
Permalink

What is Dynamic Time Warping?

hard
  • Similarity between two sequences allowing non-linear alignment (stretch/compress).
  • Finds the best matching via dynamic programming through a cost matrix.
  • Robust to phase shifts and different lengths.
  • Cost O(mn) per pair — expensive for large series (mitigated  by  lower−bounds  like  LBKeogh,  sakoe−chiba  constraint  window)(\mathrm{mitigated}\;\mathrm{by}\;\mathrm{lower} - \mathrm{bounds}\;\mathrm{like}\;\mathrm{LB}_{\mathrm{Keogh}}, \;\mathrm{sakoe} - \mathrm{chiba}\;\mathrm{constraint}\;\mathrm{window}).
  • Standard for shape-based time-series similarity in speech, ECG, gesture recognition.
Permalink

Community detection in graphs — main approaches.

hard
  • (1) Modularity optimization: Louvain, Leiden (greedy hierarchical) — scale to millions of nodes, standard default.
  • (2) Spectral: eigenvectors of the graph Laplacian → k-means.
  • (3) Label propagation (fast, non-deterministic).
  • (4) Stochastic Block Models (probabilistic).
  • Modern: GNN-based (DeepWalk / Node2Vec + clustering).
  • Leiden fixes Louvain's resolution-limit + connected-community issues — usually the default now.
Permalink

How do you interpret / visualize clusters?

medium
  • (1) 2D projection with UMAP / t-SNE colored by cluster.
  • (2) Feature-importance-per-cluster: mean of each feature per cluster vs global mean → describes each group.
  • (3) Decision-tree surrogate: fit a small tree predicting cluster label → rule-based description.
  • (4) Representative sample per cluster (medoid + top / bottom feature scores).
  • Standard deliverable for a customer segmentation report.
Permalink

How do you find nested or hierarchical structure?

medium
  • (1) Agglomerative clustering with a dendrogram → cut at different heights for different granularity.
  • (2) HDBSCAN's condensed cluster tree — natural hierarchy.
  • (3) Recursively apply k-means with k=2.
  • (4) Bayesian nonparametric methods (Dirichlet Process) let hierarchy emerge from data.
  • Useful for taxonomies, org charts, biological classifications, topic hierarchies.
Permalink

Dirichlet Process — how does it help clustering?

hard
  • Bayesian nonparametric: number of clusters is inferred rather than pre-specified.
  • New points can start new clusters with a probability depending on concentration parameter α (Chinese Restaurant Process metaphor: 'rich get richer' + occasional new table).
  • Fit via Gibbs / variational inference.
  • Automatic model complexity selection.
  • Foundation of DPGMM (Dirichlet Process GMM) in scikit-learn.
Permalink

One cluster dominates in k-means — what do you do?

medium
  • (1) Try higher k — the 'big' cluster may naturally split.
  • (2) Log-scale skewed features.
  • (3) Try k-medoids (robust to imbalanced density).
  • (4) Use HDBSCAN which handles varying densities.
  • (5) Balanced k-means variants that constrain cluster size.
  • (6) Sanity-check with silhouette per cluster — often the dominant cluster is a garbage-collector for outliers or non-standardized features.
Permalink

Interview: 'you're asked to segment 10M customers — how?'

hard
  • (1) Sample down to 1M for prototype.
  • (2) Feature engineering: RFM (recency, frequency, monetary), behavioral counts, categorical embeddings.
  • (3) Scale (log-transform skewed features, z-score).
  • (4) PCA/UMAP to 20-50 dim for compute.
  • (5) Mini-batch k-means or HDBSCAN across candidate k (silhouette + business interpretability).
  • (6) Profile clusters with per-feature means + decision-tree surrogate.
  • (7) Stability check via bootstrap.
  • (8) Deliver actionable segments with names + business hypotheses.
Permalink

How do you monitor cluster drift over time?

hard
  • Track: (1) fraction of new points assigned to each cluster (should be stable).
  • (2) Distribution of intra-cluster distances (blow-up = drift).
  • (3) Silhouette on rolling window.
  • (4) Population Stability Index of cluster proportions.
  • (5) Re-cluster periodically, measure ARI/NMI vs previous partition (< 0.7 → cluster meaning shifted).
  • When drift detected, retrain segmentation and rename / merge / split clusters with business.
Permalink

Mixture models for density estimation — beyond GMM.

hard
  • GMM assumes Gaussian components.
  • Alternatives: mixture of t-distributions (heavier tails, robust), mixture of Dirichlet (categorical), mixture density networks (NN outputs mixture parameters — good for heteroscedastic regression), Bayesian nonparametric DP-mixtures (infer k).
  • Choose component distribution based on data support (bounded → beta; positive → gamma; heavy tails → t).
Permalink

LDA vs NMF for topic modeling — which do you pick?

medium
  • LDA: probabilistic — documents = Dirichlet mixture of topics, topics = Dirichlet mixture over words.
  • Fit via variational Bayes or Gibbs.
  • Interpretable but sensitive to hyperparameters (α, β).
  • NMF: matrix factorization of TF-IDF, faster and often more coherent on short text.
  • LDA better for longer documents / traditional corpora; NMF better for short noisy text (tweets, reviews).
  • Both superseded by embedding-based topic models (BERTopic) on modern short text.
Permalink

How does BERTopic work?

medium
  • (1) Embed documents with sentence-transformers.
  • (2) Reduce dimensionality with UMAP.
  • (3) Cluster with HDBSCAN.
  • (4) Extract topic keywords via class-based TF-IDF (c-TF-IDF): TF-IDF where each 'document' is the concatenation of cluster texts.
  • Interpretable, handles short noisy text, supports topic modeling over time and hierarchical topics.
  • Modern default for exploratory topic modeling on real-world corpora.
Permalink

Applications of clustering in NLP.

medium
  • (1) Topic modeling / document grouping (LDA, NMF, BERTopic).
  • (2) Word sense discovery: cluster contextualized embeddings of ambiguous words.
  • (3) Sentence deduplication in training data (embed + cluster + keep centroids).
  • (4) User intent discovery in chatbot logs.
  • (5) Named entity clustering across languages.
  • (6) Query intent clustering for search.
  • All exploit unsupervised structure discovery.
Permalink

How do you cluster images at scale?

medium
  • (1) Feature extraction: pass through pretrained CNN or ViT (DINOv2, CLIP) → embeddings.
  • (2) UMAP / PCA to ~50 dim for compute.
  • (3) HDBSCAN or mini-batch k-means.
  • (4) For huge scale (billions): FAISS + BIRCH or streaming k-means.
  • (5) Semi-supervised: use CLIP embeddings + text queries as prompts for prototype clustering.
  • Modern: DINOv2 embeddings + k-means or HDBSCAN is production default.
Permalink

User behavior segmentation — feature engineering.

medium
  • (1) RFM: Recency (days since last activity), Frequency (activities per period), Monetary (spend).
  • (2) Behavioral counts: events per day/week.
  • (3) Categorical: preferred category, region, device.
  • (4) Sequences: transformer / GRU embed of event sequence.
  • (5) Time-decay weighted (recent behavior matters more).
  • (6) Log-transform skewed monetary.
  • Then z-score, PCA/UMAP → k-means or HDBSCAN.
  • Standard for marketing / product analytics segments.
Permalink

Clustering in single-cell RNA-seq — the standard pipeline.

hard
  • (1) QC + normalize (log-CPM, SCTransform).
  • (2) HVG selection.
  • (3) PCA to 50 dim.
  • (4) Batch correction (Harmony, scVI).
  • (5) k-NN graph in PCA space.
  • (6) Leiden community detection.
  • (7) UMAP for visualization.
  • (8) Marker gene analysis per cluster to identify cell types.
  • Modern: scVI / scANVI for VAE-based normalization + integration; totalVI for multimodal (CITE-seq).
Permalink

Interview: cluster 5M e-commerce customers for a marketing campaign.

hard
  • (1) Define business goal (targeting, retention, cross-sell?).
  • (2) Sample 500k for prototyping.
  • (3) Build RFM + behavioral features + demographic.
  • (4) Log-transform skewed features + z-score.
  • (5) PCA/UMAP to 30 dim.
  • (6) Try mini-batch k-means (k=5-10) and HDBSCAN.
  • (7) Metrics: silhouette + business KPI (per-cluster conversion / LTV).
  • (8) Profile clusters: mean features + tree surrogate + medoid examples + name each.
  • (9) Stability check.
  • (10) Deploy to full 5M via saved pipeline; retrain quarterly.
Permalink

Interview: how would you group log messages from a large distributed system?

hard
  • (1) Parse log: extract template (Drain algorithm, LogPunk) → replace variables (timestamps, IPs, numbers) with placeholders.
  • (2) Vectorize templates via TF-IDF or sentence-transformers on remaining text.
  • (3) Cluster with HDBSCAN → template categories.
  • (4) Anomaly detection: rare template = novel event.
  • (5) Interactive drill-down: click cluster → see representative logs + timeline.
  • Standard in Splunk, DataDog logs, Elasticsearch's ML.
  • Log2Vec / LogBERT are modern DL variants.
Permalink

Interview: business asks 'why is customer X in Cluster 3?' — how do you explain?

medium
  • (1) Show customer X's features + Cluster 3 mean/std for each feature → highlight top-3 features driving assignment (largest z-score contribution).
  • (2) Show representative medoid / centroid of Cluster 3 alongside X.
  • (3) Decision-tree surrogate: rule-based description of Cluster 3 (e.g. 'high-frequency + high-monetary + monthly recency').
  • (4) SHAP values on the cluster assignment probability.
  • (5) Contrast: how would X have to change to belong to Cluster 2 instead?
  • Standard for stakeholder-friendly cluster interpretability.
Permalink

Interview: 'you have to pick k for the executive team — walk me through it.'

medium
  • (1) Business first: how many actionable groups can marketing / product operate?
  • Usually 3-8 in practice.
  • (2) Compute silhouette, gap statistic, BIC (if GMM) across k = 2-15.
  • (3) Plot metrics — find plateau / knee.
  • (4) Choose k that satisfies business AND metrics (e.g. silhouette > 0.3 acceptable, gap not too noisy).
  • (5) Stability check: does the top-k survive resampling?
  • (6) Present 2-3 candidates with pros/cons; let the exec team choose the interpretation.
Permalink

Interview: after segmentation, one segment has 60% women but each region shows 40%. Why?

hard
  • Simpson's paradox — different regions have different sizes / proportions in that cluster.
  • Verify with contingency table: within each region, the segment may be 40% women, but the sample-size-weighted overall proportion is 60% because certain regions dominate the segment.
  • Fix by post-stratifying — report within-region proportions if that's what stakeholders care about.
  • Also check for sampling bias / SRM in segmentation.
Permalink

Interview: how would you cluster 1B rows on a budget?

hard
  • (1) Sample 1M for prototype clustering.
  • (2) Feature-engineer + reduce (PCA / hashed features).
  • (3) Mini-batch k-means on the sample → get k centroids.
  • (4) Extend to 1B via 'assign nearest centroid' in a single map-reduce (Spark, Dask, Ray).
  • (5) Or: BIRCH streaming to build micro-clusters, then hierarchical on summaries.
  • (6) HDBSCAN via subsample-then-extend heuristic.
  • (7) Store cluster IDs alongside data.
  • (8) Iterate on 1M until quality is right; only re-run map-reduce on 1B when necessary.
Permalink

Interview: you cluster and 3 clusters look 'right', but 2 mix categories — what next?

hard
  • (1) Feature analysis: what's shared in each mixed cluster (maybe a real behavior spans your labels).
  • (2) Try higher k → often 'mixed' clusters split cleanly.
  • (3) Try alternate algorithm (HDBSCAN, GMM) — mixed clusters may be low-density boundary.
  • (4) Feature engineering: add features that separate the mixed classes.
  • (5) Semi-supervised: use partial labels to nudge clustering via constrained k-means / seeded k-means.
  • (6) Accept that categories don't fully align with unsupervised structure and communicate to stakeholders.
Permalink

Interview: 'no labels — how do you know your clustering is any good?'

hard
  • (1) Internal indices: silhouette + Calinski-Harabasz + Davies-Bouldin across k.
  • (2) Stability: bootstrap Jaccard, ARI between init runs.
  • (3) Downstream utility: does adding cluster label improve any adjacent supervised task?
  • (4) Domain sanity: do clusters agree with known heuristics?
  • (5) Visual: 2D projection colored by cluster — coherent groups?
  • (6) Business sanity: are cluster profiles actionable + interpretable?
  • Reality: unsupervised eval is always partial — combine 3-4 signals + stakeholder buy-in.
Permalink

Interview: 'when should you NOT cluster?'

medium
  • (1) Data is truly unimodal / uniform — no structure to find.
  • (2) Very high dimensionality without reduction → distance concentration ruins similarity.
  • (3) The business question is regression / classification, not grouping.
  • (4) You need explanations, not groups (use decision trees or PDP).
  • (5) Sample size too small (< 100).
  • (6) You need causal answers → clustering can produce spurious segments.
  • Always sanity-check whether clustering is the right tool before running one.
Permalink

Interview: 'how would you tune HDBSCAN's min⁡clustersize\operatorname{min}_{\mathrm{cluster}}\mathrm{size}?'

hard
  • (1) Business constraint: what's the smallest meaningful group in the domain?
  • (2) Sensitivity analysis: sweep min⁡clustersize\operatorname{min}_{\mathrm{cluster}}\mathrm{size} ∈ [5, 10, 20, 50, 100] → measure silhouette + noise-fraction + number-of-clusters.
  • (3) Stability: bootstrap Jaccard between runs.
  • (4) Choose the min⁡clustersize\operatorname{min}_{\mathrm{cluster}}\mathrm{size} that gives stable + interpretable + right number of clusters.
  • (5) Also tune min⁡samples\operatorname{min}_{\mathrm{samples}} (density conservativeness) and clusterselectionmethod\mathrm{cluster}_{\mathrm{selection}}\mathrm{method} ('leaf' vs 'eom').
  • Iterate.
Permalink

Interview: 'discuss the tradeoffs between k-means, DBSCAN, and HDBSCAN.'

medium
  • (1) k-means: fast, scales to millions (mini-batch), but assumes spherical + equal-size clusters + fixed k + non-robust to outliers.
  • (2) DBSCAN: handles non-convex + detects noise, but needs eps tuning + fails with varying densities + O(n log n) with index.
  • (3) HDBSCAN: no eps tuning + handles varying densities + hierarchy + stability, but slower + memory-heavy for big data + hyperparameter min⁡clustersize\operatorname{min}_{\mathrm{cluster}}\mathrm{size} still matters.
  • Rule: k-means → baseline / large-scale; DBSCAN → known density; HDBSCAN → real-world exploration.
Permalink

Interview: 'what's the biggest mistake you've seen in a real clustering project?'

medium
  • (Answering as a candidate) '(1) Not scaling features — one feature dominated distances → nonsense clusters.
  • (2) Trusting elbow / silhouette alone without stability check → non-reproducible clusters.
  • (3) Skipping outlier removal for k-means.
  • (4) Presenting clusters without profiling — stakeholders couldn't act on them.
  • (5) Not monitoring drift — clusters silently degraded over 6 months.
  • Lesson: cluster is 30% algorithm, 70% feature engineering + evaluation + stakeholder communication.'
Permalink

Without labels, how do you convince a stakeholder your clustering is any good?

medium
  • Combine three kinds of evidence, because no single one is sufficient.
  • Internal measures such as silhouette or Davies-Bouldin say whether clusters are compact and separated, which is necessary but not meaningful on its own.
  • Stability says whether the structure is real: re-run on bootstrap samples or different seeds and measure how often pairs of points stay together, since a partition that changes every run is describing noise.
  • Then external usefulness, which is what actually persuades anyone: profile each cluster on variables you did not cluster on, and show that the groups differ in ways the business recognizes, or that using the cluster as a feature improves a downstream model.
  • Present the clusters as hypotheses to be validated, never as ground truth.
Permalink

You run k-means on customer data with age, income and number of purchases. What breaks?

easy
  • The scales.
  • Income in dollars ranges over tens of thousands while age spans a few decades, so squared Euclidean distance is essentially income distance and the other two variables contribute nothing.
  • Standardize or use robust scaling first, and understand that scaling is a modelling decision, because it declares each feature equally important.
  • Skew is the second problem: income is heavy-tailed, so a handful of high earners dominate the centroids, and a log transform usually helps.
  • If some variables are categorical, k-means is the wrong algorithm entirely, since a mean of a category is meaningless; use k-prototypes or a Gower distance with a method that accepts arbitrary distances.
Permalink

The elbow plot has no elbow. How do you pick k?

medium
  • Accept that a smooth curve is telling you the data has no crisp cluster count, which is common and not a failure of method.
  • Look at the silhouette across a range of k and prefer a value that is locally best rather than globally optimal.
  • Use the gap statistic, which compares within-cluster dispersion against a uniform reference and gives a principled comparison.
  • Fit a Gaussian mixture and compare the Bayesian information criterion, which penalizes complexity explicitly.
  • Then apply the constraint that usually decides it in practice: how many groups the organization can actually act on differently.
  • Six segments a marketing team can staff beats a statistically optimal thirty-one nobody can use.
Permalink

DBSCAN labels almost everything as noise. What do you change?

medium
  • The neighbourhood radius is too small for the density of your data, or the minimum points is too large.
  • Set the minimum points from domain reasoning, often around twice the dimensionality, then choose the radius from the k-distance plot: sort every point's distance to its k-th nearest neighbour and pick the value at the knee, which is where density drops off.
  • Check scaling first, since the radius is a single global distance and meaningless if features have different units.
  • If clusters genuinely have very different densities, no single radius works and that is the algorithm's real limitation, so switch to HDBSCAN, which varies the density threshold and returns a hierarchy instead of forcing one global choice.
Permalink

Is it a good idea to run k-means on raw text embeddings?

hard
  • It works, with caveats worth stating.
  • Embeddings are high-dimensional and roughly isotropic, so Euclidean distances concentrate and everything looks similarly far apart, which weakens the contrast k-means depends on.
  • Use cosine similarity instead, which for normalized vectors is equivalent to Euclidean distance on the sphere, so normalizing your vectors and then running k-means is the usual fix.
  • Reducing to a few dozen dimensions with PCA or UMAP before clustering often improves both quality and speed.
  • Also remember that embeddings cluster by whatever the encoder was trained to emphasize, frequently topic and register, so if you need a different notion of similarity no clustering algorithm will recover it.
Permalink

LDA or embedding-based topic modelling for a corpus of support tickets?

medium
  • Embedding-based clustering is usually the better modern default.
  • LDA treats a document as a bag of words with no notion of synonymy, which hurts badly on short texts like tickets, where there are too few words per document to estimate a topic mixture reliably.
  • Embedding the tickets, reducing dimensionality and clustering, then labelling each cluster by its most distinctive terms, handles paraphrase and short text far better.
  • LDA still has advantages: it gives each document a genuine mixture over topics rather than one assignment, it is cheap, and its probabilistic form is easier to defend statistically.
  • Whichever you pick, topic quality is judged by whether humans can name the topics, so run a coherence measure and then read samples.
Permalink

Your customer segments change completely when you re-run the pipeline monthly. Is that acceptable?

hard
  • No, because a segmentation the business acts on has to be stable enough to plan around, and instability means you are describing sampling noise rather than structure.
  • First check whether the instability is algorithmic: k-means with random initialization on data with no strong separation gives different partitions every run, which a fixed seed hides without fixing.
  • Measure it properly with the adjusted Rand index between consecutive runs, and if agreement is low the structure is weak.
  • Options are to reduce k until the solution is stable, initialize from the previous month's centroids so segments evolve rather than being reinvented, or move to a model with an explicit assignment probability so you can see which customers are genuinely borderline.
  • Report the stability number alongside the segments.
Permalink
02

Advanced clustering & density

8 concepts

Local Outlier Factor (LOF) — how does it work?

medium
  • For each point, compare its local density (via k-NN distances) to that of its neighbors.
  • LOF > 1: point is in a lower-density region than its neighbors → outlier.
  • Handles clusters of varying density — a big advantage over global thresholds.
  • Cost O(n2)O(n^{2}) or O(n log n) with indexes.
  • Weakness: sensitive to k.
  • Standard in tabular anomaly detection alongside Isolation Forest.
Permalink

Elliptic envelope / robust Mahalanobis distance — when to use?

medium
  • Fit a robust multivariate Gaussian (Minimum Covariance Determinant, Rousseeuw-Van Driessen) → flag points with high Mahalanobis distance.
  • Assumes elliptical / Gaussian normal cloud; robust to outliers via MCD.
  • Fast and interpretable.
  • Best for tabular data with roughly Gaussian normals (financial time series, sensor readings).
  • Fails for multimodal / non-Gaussian normals — use GMM or Isolation Forest instead.
Permalink

Kernel Density Estimation — mechanism.

medium
  • Non-parametric density estimator: p̂(x)  =  (1/(nhd))(x)\; = \;(1 / (\mathrm{nh}^{d})) Σ K((x  −  xi)/h)K((x\; - \;x_{i}) / h).
  • K = kernel (Gaussian typical). h = bandwidth (critical hyperparameter).
  • Uses: visualization (smoothed histogram), density-based anomaly detection, generative sampling.
  • Scales poorly to high d (curse of dimensionality — need exponentially more data).
  • Use cross-validation or Silverman's rule for h.
Permalink

How do you choose the KDE bandwidth?

hard
  • (1) Silverman's rule: h = (4/(d+2))^(1/(d+4)) * σ * n^(-1/(d+4)) — closed form, assumes Gaussian data.
  • (2) Scott's rule: similar but different exponent.
  • (3) Cross-validation of log-likelihood → most principled, more expensive.
  • (4) Adaptive bandwidth: h varies with local density.
  • Small h → overfit / spiky; large h → over-smooth.
  • Rule: start with Silverman, refine with CV if it matters.
Permalink

What is a copula and why use one?

hard
  • Copula separates marginal distributions from dependence structure: F(x,  y)  =  C(FX(x),  FY(y))F(x, \;y)\; = \;C(F_{X}(x), \;F_{Y}(y)).
  • Sklar's theorem: unique C for continuous margins.
  • Uses: multivariate density estimation when marginals are heavy-tailed but dependence is simpler (finance risk, joint failure modeling).
  • Gaussian copula, t-copula, Archimedean copulas (Clayton, Gumbel) — different tail dependence structures.
  • Foundational in quantitative finance.
Permalink

Normalizing flows for density estimation — the idea.

hard
  • Learn invertible neural network f_θ that maps simple base distribution (Gaussian) to complex data distribution.
  • Change of variables: log p(x) = log p(z)  +  log⁡∣det⁡p(z)\; + \;\operatorname{log} \mid \operatorname{det} JfJ_{f}^(-1)|.
  • Requires efficient Jacobian determinants → architectures: RealNVP, Glow, NICE (coupling layers), MAF, NSF.
  • Uses: exact density estimation, sampling, anomaly detection.
  • Modern: replaced by diffusion models for sampling but still competitive for exact-density needs.
Permalink

Score matching — what does it estimate?

hard
  • Rather than p(x), estimate the score s(x) = ∇_x log p(x).
  • Loss (Hyvärinen): E[∣∣sθ(x)  −  ∇  log⁡  p(x)∣∣2]E[ \mid \mid s{\theta}(x)\; - \; \nabla \;\operatorname{log}\;p(x) \mid \mid ^{2}] → doesn't need normalization constant → tractable for unnormalized density models.
  • Foundation of energy-based models and diffusion models (score-based generative modeling, Song & Ermon; Ho et al. DDPM).
  • Modern generative modeling paradigm shift.
Permalink

KL divergence — what it measures and pitfalls.

medium
  • KL⁡(P  ∣∣  Q)  =  Σ\operatorname{KL}(P\; \mid \mid \;Q)\; = \;{\Sigma} P(x) log⁡(P(x)/Q(x))\operatorname{log}(P(x) / Q(x)) = -H(P) + H(P, Q).
  • Not symmetric, not a metric.
  • Zero iff P = Q, infinite if Q(x) = 0 where P(x) > 0.
  • Two directions: forward KL⁡(P∣∣Q)\operatorname{KL}(P \mid \mid Q) 'mode-covering' (VAE default → q spreads over p); reverse KL⁡(Q∣∣P)\operatorname{KL}(Q \mid \mid P) 'mode-seeking' (VI → q concentrates on one mode of p).
  • Standard divergence in ML but rarely a metric; prefer Wasserstein / MMD when metric properties matter.
Permalink
03

Distance & scaling considerations

6 concepts

HNSW — how does it work?

hard
  • Hierarchical multi-layer graph.
  • Top layer: sparse long-range edges.
  • Lower layers: denser short-range.
  • Search: start at top, greedy traversal toward query, descend one layer, refine.
  • Insertion: probabilistically assign layer (exponential decay), connect to M nearest neighbors per layer.
  • Log-time search on average.
  • Trade-off knobs: M (neighbors per node), efconstruction  /  efsearch\mathrm{ef}_{\mathrm{construction}}\; / \;\mathrm{ef}_{\mathrm{search}} (search-time exploration).
Permalink

Product Quantization (PQ) — how does it compress vectors?

hard
  • Split d-dim vector into m sub-vectors of d/m dims.
  • Cluster each sub-vector space separately (k centroids per subspace, k typically 256).
  • Encode a vector as m codes (log⁡2(k)  bits  each)(\operatorname{log}_{2}(k)\;\mathrm{bits}\;\mathrm{each}).
  • Distance computed via precomputed lookup tables between query and centroids → very fast. 8-32× compression vs float32 with modest recall loss.
  • Foundation of FAISS's scale to billions of vectors.
Permalink

Locality-Sensitive Hashing (LSH) — the core trick.

hard
  • Hash function h such that P(h(x) = h(y)) is high for close x, y and low for far ones (opposite of crypto hash).
  • Different metric → different LSH family: MinHash for Jaccard (near-duplicate documents), SimHash for cosine, random hyperplane for cosine, Euclidean via random projections.
  • Multiple hash tables → high recall.
  • Standard for near-duplicate detection in web crawls, genomic sequence search.
Permalink

MinHash — how does it estimate Jaccard similarity?

hard
  • Given sets A, B: pick many random permutations of the universe.
  • For each permutation, MinHash(A) = smallest element of A under that permutation.
  • Fraction of permutations where MinHash(A) = MinHash(B) is an unbiased estimator of Jaccard(A, B) = |A ∩ B| / |A ∪ B|.
  • Compact signature (128-256 hashes) enables billion-set comparison.
  • Standard for near-duplicate detection in web crawlers (Bing, Google historically).
Permalink

Interview: your image dataset (10M) has near-duplicates — how do you dedup at scale?

hard
  • (1) Perceptual hashing (pHash / dHash / wHash): 64-bit hash → Hamming distance threshold catches near-duplicates and small edits.
  • Fast, cheap.
  • (2) Deep embedding-based (CLIP / DINOv2) + LSH or FAISS ANN for semantic near-duplicates (same subject, different pose).
  • (3) Combine both — pHash for pixel-level, embeddings for content.
  • Standard in training data prep for image generation (Stable Diffusion 3, Ideogram).
  • Also matters for benchmarking to avoid train-test leak.
Permalink

Concretely, what goes wrong with distance-based methods in high dimensions?

hard
  • Distances concentrate: as dimensionality grows, the ratio between the nearest and farthest neighbour of a point tends towards one, so the very notion of a nearest neighbour loses discriminative power.
  • Volume also grows so fast that any realistic sample is sparse, meaning every point is effectively an outlier and density estimates have almost no support.
  • Irrelevant dimensions make this worse, since each contributes noise to the distance and dilutes the informative ones, which is why feature selection helps distance methods disproportionately.
  • The practical responses are to reduce dimensionality first, use a metric learned or chosen for the domain, use cosine distance where magnitude is uninformative, or switch to methods such as tree ensembles that select dimensions rather than aggregating over all of them.
Permalink
04

Cluster evaluation

5 concepts

How do you evaluate anomaly detection?

medium
  • Usually severely imbalanced → don't use accuracy.
  • Standard metrics: (1) ROC AUC (rank-based, threshold-free), (2) PR AUC (better under extreme imbalance), (3) precision@k and recall@k at operating threshold, (4) F1 for a chosen threshold.
  • Cost-sensitive: business cost of false positive / false negative.
  • Also: alert precision and MTTR (time to detect) in production ops contexts.
Permalink

How do you evaluate the quality of a self-supervised representation?

medium
  • (1) Linear probing: freeze encoder, train linear classifier on downstream labels → measures 'linearly separable info'.
  • (2) Fine-tuning: unfreeze, full downstream training → measures 'total info'.
  • (3) k-NN classification in embedding space → non-parametric.
  • (4) Transfer to many tasks (VTAB, ELEVATER).
  • (5) Robustness under distribution shift.
  • Rule: report linear probe + k-NN + fine-tune; don't over-index on one downstream task.
Permalink

How do you evaluate topic quality?

medium
  • (1) Coherence metrics: CVC_{V}, CUMassC_{\mathrm{UMass}}, CNPMIC_{\mathrm{NPMI}} (Röder et al.) — measure semantic similarity of top-k topic words.
  • (2) Topic diversity: fraction of unique top-k words across topics.
  • (3) Downstream utility: classification accuracy using topic features.
  • (4) Human evaluation: 'word intrusion' task.
  • (5) Perplexity on held-out — surprisingly poor predictor of human-judged quality.
  • Rule: report coherence + diversity + qualitative inspection.
Permalink

Your anomaly detector has no labels. How do you set the decision threshold?

medium
  • Set it from operational capacity rather than from statistics.
  • Decide how many alerts a human can genuinely review per day, then take that quantile of the score distribution, which turns an unanswerable question into a staffing one.
  • Get a handful of confirmed cases, even a dozen from historical incidents, and check that they land above the threshold, because a threshold nothing known can pass is not a threshold.
  • Track the score distribution over time, since drift shifts scores and a fixed absolute cutoff silently changes the alert volume.
  • Then close the loop: every reviewed alert becomes a label, and after a few weeks you have enough to evaluate precision honestly and eventually train a supervised model.
Permalink

What makes a good self-supervised pretext task?

hard
  • It must be solvable only by learning something you actually want.
  • The failure mode is a shortcut: a task the network can solve with a superficial cue, such as detecting a rotation from a border artefact or matching two crops by their shared colour histogram, which yields a high pretext score and useless representations.
  • Good tasks force invariance to nuisance factors while preserving the semantics you need downstream, which is why contrastive augmentation design matters more than the loss function.
  • The task also has to be hard enough to require capacity but not so ambiguous that the target is unpredictable in principle.
  • The only real test is transfer: freeze the encoder and measure linear-probe performance on the downstream task, because pretext loss is not the objective you care about.
Permalink
05

Linear dimensionality reduction

38 concepts

PCA vs t-SNE vs UMAP — when do you use each?

medium
  • PCA is a linear, deterministic projection that preserves global variance — use it for compression, denoising, or as input to another model. t-SNE and UMAP are nonlinear and preserve local neighborhoods — use them for 2D/3D visualization, not for downstream modeling.
  • UMAP is faster than t-SNE and preserves more global structure.
  • Never interpret t-SNE distances between clusters as meaningful.
Permalink

How do you decide how many PCA components to keep?

easy
  • Look at the cumulative explained-variance ratio and keep enough components to reach a target (e.g., 90% or 95%).
  • You can also inspect the scree plot for an elbow, or pick components based on downstream cross-validation performance.
  • Standardize features before PCA when they are on different scales.
Permalink

Derive PCA — what does it optimize?

hard
  • Given centered X, find orthonormal w that maximizes Var⁡(Xw)  =  w\operatorname{Var}(\mathrm{Xw})\; = \;w' Σ w subject to ||w||=1.
  • Solution: eigenvector of Σ = X'X/n corresponding to the largest eigenvalue.
  • Subsequent components: next largest eigenvalue, orthogonal.
  • Equivalently: minimizes squared reconstruction error over rank-k subspaces (Eckart-Young).
  • Two views, same answer.
Permalink

PCA via SVD — the connection.

medium
  • For centered X (n × p), SVD gives X = UΣV'.
  • Principal directions = columns of V (right singular vectors).
  • Principal scores = UΣ.
  • Explained variance ratios ∝ diag⁡(Σ)2\operatorname{diag}({\Sigma})^{2}.
  • More numerically stable than eigendecomposing X'X (which squares condition number).
  • Standard scikit-learn PCA uses SVD internally.
  • Randomized SVD (Halko-Tropp) approximates top-k in O(np log k) — big-data default.
Permalink

Why standardize before PCA (usually)?

easy
  • PCA maximizes variance in the original units.
  • If features have wildly different scales (income in $ vs age in years), the high-variance one dominates → useless components.
  • Standardize (z-score) so each contributes equally, unless: features are on the same natural scale (pixel intensities, log-returns) where scale carries meaning.
  • Alternative: correlation-matrix PCA is equivalent to z-scored PCA.
Permalink

When does PCA fail?

medium
  • (1) Non-linear structure (rolls, spirals) — PCA sees only linear correlations.
  • (2) Discrete / categorical features (one-hot creates artificial variance).
  • (3) Skewed features — one heavy-tail feature dominates; log-transform first.
  • (4) Interpretability priority — PC1 mixes many features, hard to name.
  • (5) Anomaly-heavy data — outliers distort covariance; use robust PCA.
  • Alternatives: KernelPCA, autoencoders, ICA depending on the failure mode.
Permalink

Kernel PCA — when and how?

hard
  • Apply the kernel trick to PCA: implicitly map data to a feature space Φ(x) via kernel K(x, y) = ⟨Φ(x), Φ(y)⟩, then do PCA there.
  • Captures non-linear structure without explicit feature construction.
  • Common kernels: RBF, polynomial, cosine.
  • Cost O(n2)O(n^{2}) memory (kernel matrix) — doesn't scale beyond ~10k.
  • Modern replacement: autoencoders / UMAP for larger data.
Permalink

ICA vs PCA — the key difference.

hard
  • PCA: finds uncorrelated components maximizing variance.
  • ICA: finds statistically independent components — stronger requirement.
  • Uses non-Gaussianity (kurtosis, negentropy) to find directions that look non-Gaussian.
  • Classic use: blind source separation (cocktail party — separate mixed audio signals).
  • Requires nsourcesn_{\mathrm{sources}} ≤ nsensorsn_{\mathrm{sensors}} and at most one Gaussian source.
  • FastICA is the standard implementation.
Permalink

Non-negative Matrix Factorization (NMF) — when to use?

medium
  • Factor X ≈ WH with W, H ≥ 0.
  • Enforces additive parts-based representation.
  • Uses: (1) topic modeling on TF-IDF (topics as sparse non-negative bases), (2) audio spectrogram decomposition, (3) image parts (Lee & Seung's faces).
  • More interpretable than PCA when non-negativity is natural.
  • Fit by multiplicative updates or ADMM.
  • Rank must be chosen (CV / stability).
Permalink

Sparse coding — what is it?

hard
  • Learn overcomplete dictionary D such that x ≈ Dα with sparse α (few non-zero entries).
  • Uses: image compression, denoising, feature learning.
  • Fit alternates between (1) LASSO for α given D, (2) dictionary update for D given α.
  • Foundation of Olshausen & Field's V1 modeling and later dictionary learning.
  • Related to compressed sensing.
Permalink

Random projection — how does it work?

medium
  • Johnson-Lindenstrauss: for ε > 0, a random Gaussian matrix R (m × p) with m  =  O(log⁡  n  /  ε2)m\; = \;O(\operatorname{log}\;n\; / \;{\varepsilon}^{2}) preserves pairwise distances up to (1±ε) with high probability.
  • Extremely cheap (no data-dependent fit).
  • Uses: dimensionality reduction for huge sparse data, approximate nearest neighbors (LSH), streaming.
  • Less informative than PCA but scales linearly.
Permalink

Linear Discriminant Analysis (LDA) vs PCA — the difference.

medium
  • PCA is unsupervised — max variance.
  • LDA is supervised — projects to maximize class separability: max Tr(SB)  /  Tr(SW)\mathrm{Tr}(S_{B})\; / \;\mathrm{Tr}(S_{W}), where SB  =  between−classS_{B}\; = \;\mathrm{between} - \mathrm{class} scatter, SW  =  within−classS_{W}\; = \;\mathrm{within} - \mathrm{class} scatter.
  • Yields at most (k-1) components for k classes.
  • Uses: dimensionality reduction as preprocessing for classifiers, face recognition (Fisherfaces).
  • Assumes normally distributed classes with equal covariance.
Permalink

Canonical Correlation Analysis (CCA) — use case.

hard
  • Finds pairs of linear projections (u, v) of two multivariate datasets X, Y that maximize their correlation.
  • Uses: (1) multi-view learning — combine images and text embeddings, (2) genomics (SNPs vs gene expression), (3) recommendation with side information.
  • Deep CCA extends with neural nets on each side.
  • Foundational for cross-modal alignment.
Permalink

t-SNE perplexity — what does it control?

medium
  • Perplexity ≈ effective number of neighbors each point 'considers' when building the conditional similarity distribution.
  • Typical range: 5-50.
  • Low perplexity → local, may fragment groups.
  • High → global, may merge distinct clusters.
  • Rule of thumb: try 5, 30, 50 and inspect.
  • Don't over-tune to make plots look better — the map should be robust across a range.
Permalink

Top t-SNE pitfalls to avoid.

hard
  • (1) Cluster sizes in the map are not meaningful — t-SNE inflates dense clusters.
  • (2) Inter-cluster distances are NOT preserved — 'far' clusters aren't necessarily far in data space.
  • (3) Random-seed sensitivity → run multiple times.
  • (4) Only meaningful in 2/3D.
  • (5) NEVER feed t-SNE embedding into a downstream model — use PCA or UMAP instead.
  • Wattenberg's 'How to Use t-SNE Effectively' is required reading.
Permalink

UMAP vs t-SNE — practical differences.

medium
  • UMAP: faster (O(n114)  vs  O(n  log⁡  n)  with  Barnes−Hut  but  higher  constant)(O(n^{1}14)\;\mathrm{vs}\;O(n\;\operatorname{log}\;n)\;\mathrm{with}\;\mathrm{Barnes} - \mathrm{Hut}\;\mathrm{but}\;\mathrm{higher}\;\mathrm{constant}), preserves more global structure (inter-cluster distances less-meaningful but somewhat interpretable), supports supervised / semi-supervised variants, can transform new points (t-SNE cannot).
  • Both are for visualization primarily.
  • UMAP has become the default for 2D projection in scRNA-seq, embeddings inspection, exploratory analysis.
Permalink

UMAP key hyperparameters.

medium
  • (1) nneighborsn_{\mathrm{neighbors}}: local vs global (5-15 local, 50-100 global; default 15).
  • (2) min⁡dist\operatorname{min}_{\mathrm{dist}}: minimum spacing in embedding — small (0.1) preserves clusters, large (0.5) spreads points.
  • (3) ncomponentsn_{\mathrm{components}}: usually 2 for viz, 10-50 for downstream.
  • (4) metric: euclidean, cosine, correlation, hamming, custom callable.
  • (5) randomstate\mathrm{random}_{\mathrm{state}} for reproducibility.
  • Tune nneighbors  +  min⁡distn_{\mathrm{neighbors}}\; + \;\operatorname{min}_{\mathrm{dist}} together; try 3-4 combinations.
Permalink

Isomap — what does it do?

hard
  • Non-linear DR that preserves geodesic (manifold) distances.
  • Steps: (1) k-NN graph, (2) all-pairs shortest paths (Dijkstra / Floyd-Warshall) → geodesic distance matrix, (3) MDS on geodesic distances → low-dim embedding.
  • Discovers curved manifolds where Euclidean distance fails (roll, sphere).
  • Precursor to UMAP; still useful for interpretability and small datasets.
Permalink

Locally Linear Embedding (LLE) — how does it work?

hard
  • (1) Find k nearest neighbors for each point.
  • (2) Reconstruct each point as linear combination of neighbors: minimize ||xi  −  Σx_{i}\; - \;{\Sigma} wijw_{\mathrm{ij}} xjx_{j}||² with Σ wij  =  1w_{\mathrm{ij}}\; = \;1.
  • (3) Find embedding yiy_{i} minimizing ||yi  −  Σy_{i}\; - \;{\Sigma} wijw_{\mathrm{ij}} yjy_{j}||² preserving the same weights.
  • Captures non-linear structure with a linear reconstruction locally.
  • Sensitive to k.
  • Modern: superseded by UMAP but still cited in manifold-learning theory.
Permalink

Multidimensional Scaling (MDS) — variants.

medium
  • Given pairwise distance matrix D, find low-dim embedding preserving distances.
  • Classical MDS: closed form via eigendecomposition of doubly-centered D2D^{2} — equivalent to PCA on distance data.
  • Metric MDS: minimize stress = Σ (∣∣yi  −  yj∣∣  −  dij)2( \mid \mid y_{i}\; - \;y_{j} \mid \mid \; - \;d_{\mathrm{ij}})^{2}.
  • Non-metric MDS: preserves only rank order of distances.
  • Uses: psychometrics, marketing perception maps, small-n visualization.
Permalink

Autoencoder for dimensionality reduction — pros and cons.

medium
  • Pros: non-linear, scalable (SGD), transformable (new data → embedding), can be regularized (denoising, sparse, contractive).
  • Cons: no orthogonality (hard to interpret), no explained-variance ratios, sensitive to hyperparameters, harder than PCA.
  • Rule: for tabular data < 50 dim, PCA is enough.
  • For images / audio / very high dim, autoencoders (especially convolutional) dominate.
Permalink

What is the manifold hypothesis?

medium
  • Real high-dimensional data (images, audio, text) lies on a low-dimensional manifold embedded in the high-dim space — e.g. natural images inhabit ~O(hundreds)-dim manifold within millions of pixels.
  • Motivates non-linear DR (Isomap, UMAP, autoencoders): find that low-dim coordinate system.
  • Foundation of modern representation learning: features are more useful in manifold coordinates than raw pixels.
Permalink

How do you estimate the intrinsic dimension of a dataset?

hard
  • (1) Correlation dimension (Grassberger-Procaccia): slope of log⁡(number  of  pairs  within  ε)\operatorname{log}(\mathrm{number}\;\mathrm{of}\;\mathrm{pairs}\;\mathrm{within}\;{\varepsilon}) vs log ε.
  • (2) MLE-based (Levina-Bickel): from distances to k nearest neighbors.
  • (3) TwoNN (Facco et al., 2017): ratio of first two neighbor distances.
  • (4) PCA scree-plot elbow.
  • Real datasets typically have intrinsic dim much smaller than ambient dim (MNIST ambient=784, intrinsic ≈ 12-14).
Permalink

Truncated SVD vs PCA on sparse data.

medium
  • PCA subtracts the mean → densifies sparse matrices (bad for TF-IDF, one-hot).
  • Truncated SVD (scikit-learn's TruncatedSVD, also known as LSA in text) does SVD without centering → preserves sparsity, scales to millions of features.
  • Standard for topic modeling (LSA), sparse recommender matrices.
  • Coordinates and variance interpretations same as PCA, just without the mean shift.
Permalink

Incremental PCA — when do you need it?

medium
  • Dataset doesn't fit in RAM.
  • Process mini-batches, update running estimate of principal components (block Lanczos or Ross et al.'s method). scikit-learn: IncrementalPCA.
  • Constant memory O(dk) rather than O(np).
  • Slight accuracy tradeoff vs full SVD.
  • Standard for streaming / out-of-core PCA on terabyte-scale tabular data.
Permalink

Robust PCA — what problem does it solve?

hard
  • Standard PCA is sensitive to outliers (single bad row can rotate components).
  • Robust PCA (Candès et al.): decompose X = L + S where L is low-rank (clean data) and S is sparse (outliers).
  • Solved via convex relaxation: min ||L||_* + λ ||S||_1 (Principal Component Pursuit).
  • Uses: video background subtraction, corrupt entry recovery, anomaly detection.
Permalink

Why does truncated SVD denoise?

hard
  • Signal typically lies in low-rank subspace; noise spreads across all singular directions.
  • Discarding small singular values throws away noise-dominant components while keeping signal.
  • Foundation of PCA-based denoising, spectral clustering, and matrix completion (Netflix Prize).
  • Also called optimal shrinkage — Gavish-Donoho gives an explicit optimal threshold for Gaussian noise.
Permalink

Word embeddings as unsupervised DR of text.

medium
  • Word2Vec (skip-gram, CBOW), GloVe: predict context words → learn dense word vectors capturing co-occurrence structure.
  • Word2Vec is implicit PMI matrix factorization (Levy & Goldberg 2014).
  • GloVe: explicit log-count factorization.
  • FastText: adds sub-word info for OOV.
  • Modern replacement: contextual embeddings (BERT, sentence-transformers) — the dominant text representation since 2018.
Permalink

How do modern sentence / doc embeddings work?

medium
  • Sentence-BERT (Reimers-Gurevych): fine-tune BERT with siamese contrastive loss on paraphrase / NLI pairs → semantically-meaningful vectors, cosine similarity ≈ semantic similarity.
  • Modern: E5, BGE, GTE, OpenAI text-embedding-3, Cohere embed-v3 — trained on contrastive tasks over huge multilingual corpora.
  • Used everywhere in RAG, retrieval, semantic search, clustering.
Permalink

Matrix completion — how does it relate to unsupervised learning?

hard
  • Recover missing entries in a matrix by assuming low-rank structure.
  • Solve min rank⁡(M)\operatorname{rank}(M) s.t. observed entries match — NP-hard, relaxed to min ||M||_* (nuclear norm).
  • Foundation of collaborative filtering (Netflix Prize), missing-data imputation.
  • Modern: Alternating Least Squares (Spark ALS), deep matrix factorization, or neural collaborative filtering.
Permalink

Principal Components Regression (PCR) — what does it do?

hard
  • Regress on PCA components instead of raw features: (1) PCA → keep top k components, (2) OLS on those.
  • Reduces multicollinearity + acts as regularization.
  • Downside: PCA is unsupervised → top components may not correlate with Y.
  • Alternative: PLS (Partial Least Squares) finds components maximizing covariance with Y — supervised DR + regression.
  • PLS usually better for prediction with many correlated features.
Permalink

Partial Least Squares (PLS) — how is it different from PCR?

hard
  • PCR: PCA components chosen by max variance in X (Y-ignorant).
  • PLS: components maximize covariance with Y (supervised DR).
  • Better prediction for high-dim, multicollinear data (spectroscopy, genomics, chemometrics).
  • Fit iteratively: extract latent variable, deflate X and Y, repeat.
  • Bridge between PCA and OLS.
  • Standard in chemometrics, increasingly used in bioinformatics.
Permalink

LSA (Latent Semantic Analysis) — how does it relate to modern retrieval?

medium
  • Truncated SVD on TF-IDF document-term matrix.
  • Rows → topic vectors for docs, columns for terms.
  • Uses: information retrieval (query as bag-of-words → SVD-project → cosine-retrieve).
  • Precursor to modern dense retrieval (sentence-transformers, dual encoders).
  • Still useful as baseline; interpretable in low-resource languages where fine-tuned encoders aren't available.
Permalink

Matrix factorization for recsys — objective.

medium
  • Model rating ruir_{\mathrm{ui}} ≈ pup_{u}' * qiq_{i} where pup_{u} ∈ RkR^{k} is user vector, qiq_{i} is item vector.
  • Minimize Σ (rui  −  puqi)2  +  λ(r_{\mathrm{ui}}\; - \;p_{u}q_{i})^{2}\; + \;{\lambda} (∣∣pu∣∣2  +  ∣∣qi∣∣2)( \mid \mid p_{u} \mid \mid ^{2}\; + \; \mid \mid q_{i} \mid \mid ^{2}) over observed entries.
  • Solved with SGD or ALS.
  • Handles missing entries naturally (only sum over observed).
  • Extensions: biases (puqi  +  bu  +  bi  +  μ)(p_{u}q_{i}\; + \;b_{u}\; + \;b_{i}\; + \;{\mu}), implicit feedback (BPR loss, weighted ALS in Spark).
Permalink

Interview: 'you have 500 features, most correlated — how do you preprocess?'

medium
  • (1) Correlation heatmap → drop obviously redundant.
  • (2) Domain-driven grouping (aggregate related features first).
  • (3) Missing-value pattern check.
  • (4) Standardize.
  • (5) PCA / Truncated SVD → keep 95% variance (usually 30-80 components).
  • (6) If non-linear structure suspected: UMAP (for viz), autoencoder (for downstream).
  • (7) For interpretable model: keep top-loadings; for pure prediction: use the full transformation.
  • (8) Consider L1-regularized model as alternative that learns sparsity end-to-end.
Permalink

Interview: 'when should you use PCA vs autoencoder for dim reduction?'

medium
  • PCA: (1) linear structure, (2) < 200 features, (3) want interpretability (loadings), (4) fast + closed form, (5) baseline.
  • Autoencoder: (1) non-linear structure (images, sequences), (2) very high dim (thousands+), (3) enough data to train, (4) willing to accept less interpretability, (5) transformable to new points, (6) regularizable (denoising, sparsity).
  • Rule: start with PCA — if downstream metric plateaus, try autoencoder to catch non-linear info.
Permalink

How many principal components do you keep, and what does 95% variance actually guarantee?

medium
  • Common choices are a cumulative variance threshold, the knee in the scree plot, or the number that maximizes downstream performance, and the last is the only one tied to your goal.
  • What a 95% variance threshold guarantees is only that reconstruction error in the least-squares sense is small; it says nothing about whether the discarded 5% contained the signal you care about.
  • This matters because variance is not relevance: a low-variance direction can carry the entire class distinction, which is exactly why PCA is unsupervised and can discard the label-bearing axis.
  • If your aim is prediction, treat the component count as a hyperparameter tuned inside cross-validation, and remember to fit PCA on the training fold only.
Permalink

What conclusions can you not draw from a t-SNE plot?

hard
  • Cluster sizes are meaningless, because t-SNE expands sparse regions and compresses dense ones to fit everything into two dimensions, so a visually large blob is not a numerous group.
  • Distances between clusters are also unreliable: the method preserves local neighbourhoods and deliberately sacrifices global geometry, so two well-separated blobs may be closer in the original space than they look.
  • Apparent gaps can be artefacts of the perplexity setting, and running it twice with different seeds gives different layouts.
  • What you can read is which points are neighbours of which.
  • Treat it as a qualitative sanity check, never as evidence for the number of clusters, and use UMAP if you need somewhat better global structure.
Permalink
06

Anomaly & outlier detection

26 concepts

What are the main approaches to anomaly detection?

medium
  • (1) Statistical: fit a distribution and flag low-density points (z-score, Gaussian mixtures).
  • (2) Distance/density-based: LOF, DBSCAN.
  • (3) Isolation-based: Isolation Forest — random splits, anomalies are isolated with few splits.
  • (4) One-class SVM.
  • (5) Reconstruction-based: autoencoders — high reconstruction error means anomaly.
  • Choose based on data size, dimensionality, and whether you have any labels.
Permalink

Isolation Forest — how does it detect anomalies?

medium
  • Build many random trees splitting on random features and thresholds.
  • Anomalies are 'isolated' with fewer splits than inliers (shorter path length).
  • Score s(x, n) = 2^(-E[h(x)] / c(n)) where E[h] = average path length, c(n) = normalization.
  • Fast O(n log n), works well in high dims, no density estimation.
  • Standard first choice for tabular anomaly detection.
Permalink

One-class SVM — mechanism and pitfalls.

hard
  • Learns a decision boundary enclosing 'normal' data by maximizing margin from origin in RKHS (with RBF kernel typically).
  • Parameter ν upper-bounds fraction of outliers.
  • Sensitive to hyperparameters (ν, γ).
  • Doesn't scale beyond ~10k.
  • Modern alternative: Isolation Forest (faster, more robust).
  • Still useful when a semantic 'normal' class is well-defined and features are dense.
Permalink

Autoencoder for anomaly detection — how?

medium
  • Train AE on 'normal' data — it learns to reconstruct normal patterns well.
  • At inference, compute reconstruction error ||x - x̂||²; high error = anomaly.
  • Works well when normal data has low-dimensional manifold structure (images, time series).
  • Pitfalls: (1) if AE is too powerful, it reconstructs anomalies too — use small bottleneck; (2) contamination in training data → semi-supervised or filter first.
Permalink

VAE-based anomaly detection — advantages.

hard
  • VAE learns probabilistic latent space with p(x).
  • Anomaly score: (1) reconstruction error, (2) negative ELBO, (3) KL between q(z∣x)q(z \mid x) and prior.
  • Advantages over AE: probabilistic, calibrated uncertainty, generative — can synthesize plausible normals.
  • Weakness: still needs threshold tuning; failure mode: 'likelihood > uniform' issue (in-distribution mismatch on OOD data — Nalisnick et al. 2019).
Permalink

Deep SVDD — the core idea.

hard
  • Deep Support Vector Data Description (Ruff et al. 2018): train a neural encoder φ_θ to map all normal training data into a small hypersphere in feature space (center c, minimum radius).
  • Anomaly score: distance ||φ_θ(x) - c||.
  • End-to-end representation-plus-boundary learning.
  • Modern default for deep anomaly detection on images / time series.
Permalink

Semi-supervised vs unsupervised anomaly detection.

medium
  • Unsupervised: no labels; assumes normals dominate.
  • Semi-supervised: only normal data labeled — train on clean normals, detect deviations (one-class SVM, deep SVDD, AE).
  • Supervised: rare — a few labeled anomalies (PU learning, imbalanced classification).
  • Rule: whenever you have even 20-50 confirmed anomaly examples, do semi-supervised evaluation or PU learning — it's a huge lift over pure unsupervised.
Permalink

Positive-Unlabeled (PU) learning — when useful?

hard
  • You have some confirmed positives (frauds, anomalies) and a large unlabeled pool that mixes positives + negatives.
  • Standard classification is biased.
  • PU methods: (1) treat unlabeled as noisy negatives and correct via propensity, (2) two-step: identify reliable negatives then train classifier, (3) unbiased PU risk estimator (Kiryo et al. 2017).
  • Uses: fraud (few labeled frauds, most 'clean' actually clean), rare-disease diagnosis.
Permalink

Anomaly detection in time series — what changes?

hard
  • Temporal context matters.
  • Options: (1) statistical: rolling mean/std + z-score, ARIMA residuals, STL decomposition + IQR on residuals.
  • (2) Prophet / SARIMA residual monitoring.
  • (3) LSTM-AE / Transformer-AE — reconstruction error on windows.
  • (4) Anomaly Transformer / TimesNet.
  • (5) Change-point detection (Bayesian, CUSUM).
  • Consider: point anomalies, contextual anomalies, collective anomalies.
  • Pipe with alerting + human review.
Permalink

How is drift detection an unsupervised problem?

medium
  • Compare current feature distributions to training / baseline distributions without labels.
  • Methods: (1) KS test per feature (univariate).
  • (2) Population Stability Index (PSI).
  • (3) Multivariate: MMD (Maximum Mean Discrepancy), classifier drift test (train binary classifier reference vs current — AUC > 0.7 = drift).
  • (4) Wasserstein / Jensen-Shannon divergence.
  • Standard in production ML monitoring — Datadog, Arize, WhyLabs, Fiddler all implement variants.
Permalink

Maximum Mean Discrepancy (MMD) — what is it?

hard
  • Distance between two distributions in RKHS: MMD(P, Q) = ||μP  −  μQ{\mu}_{P}\; - \;{\mu}_{Q}||_H where μP{\mu}_{P} is the mean embedding.
  • Estimated from samples using kernel k (Gaussian typical): MMD2  =  E[k(x,  x)]  +  E[k(y,  y)]  −  2\mathrm{MMD}^{2}\; = \;E[k(x, \;x)]\; + \;E[k(y, \;y)]\; - \;2 E[k(x, y)].
  • Uses: two-sample tests (do P and Q differ?), drift detection, generative model evaluation.
  • More powerful than univariate tests in multivariate settings.
Permalink

Wasserstein distance — intuition.

hard
  • 'Earth mover's distance': minimum cost of transporting mass from P to Q where cost = distance × mass.
  • W1(P,  Q)  =  inf⁡W_{1}(P, \;Q)\; = \;\operatorname{inf}_γ E_γ[∣∣x  −  y∣∣][ \mid \mid x\; - \;y \mid \mid ] over couplings γ.
  • Metrically meaningful even for disjoint distributions (unlike KL, which is infinite).
  • Uses: WGAN training stability, drift detection, optimal transport for domain adaptation, distribution comparison in imaging.
Permalink

Covariate drift vs concept drift vs label drift — the differences.

medium
  • Covariate drift: P(X) changes, P(Y  ∣  X)P(Y\; \mid \;X) same — model still valid, just used on different inputs (retrain if severe).
  • Concept drift: P(Y  ∣  X)P(Y\; \mid \;X) changes — model relationship broken, must retrain.
  • Label drift: P(Y) shifts (imbalance changes) — matters most for calibration.
  • Rule: monitor all three.
  • Concept drift is the hardest to detect without labels (needs delayed ground truth) → proxy via prediction confidence drift.
Permalink

Out-of-distribution (OOD) detection — approaches.

hard
  • Softmax confidence is unreliable (over-confident on OOD).
  • Better: (1) MSP with temperature scaling, (2) energy score (LeCun et al.), (3) ODIN (temperature + input perturbation), (4) Mahalanobis distance in feature space, (5) deep generative likelihood (with Nalisnick's caveat), (6) contrastive OOD detectors.
  • Modern default: energy score or Mahalanobis on penultimate features.
  • Critical in safety-critical deployment (medical, autonomous).
Permalink

Conformal prediction for anomaly detection.

hard
  • Given calibration set of normal points, use non-conformity score to produce p-values with guaranteed (1-α) coverage on inliers under exchangeability. p-value < α → outlier at level α.
  • Distribution-free, model-agnostic → wraps any anomaly detector for calibrated alarms.
  • Standard in modern high-assurance production monitoring; foundational in the CP framework (Vovk, Shafer).
Permalink

Scan statistics — when do you use them?

hard
  • Detect unusual clusters in space-time (disease outbreaks, crime hotspots, network intrusion).
  • Slide a window over spatial / temporal regions, compare observed vs expected counts (usually Poisson).
  • Kulldorff's spatial scan is standard.
  • Uses: public health surveillance (BioSense, Prodrome), retail hotspot detection.
  • Corrects for multiple testing across regions via Monte Carlo simulation of the null.
Permalink

CUSUM change-point detection — how does it work?

hard
  • Cumulative sum of deviations from a reference mean: St  =  max⁡(0,  St−1  +  (xt  −  μ)  −  k)S_{t}\; = \;\operatorname{max}(0, \;S_{t - 1}\; + \;(x_{t}\; - \;{\mu})\; - \;k).
  • Alarm when St  >  thresholdS_{t}\; > \;\mathrm{threshold} h.
  • Quickly detects small persistent shifts.
  • Bidirectional variant.
  • Tune k, h to control false-alarm rate + detection latency.
  • Standard in manufacturing SPC, network monitoring, sensor drift.
  • Bayesian alternatives (BOCPD) provide posterior over change points.
Permalink

How do you handle 99.9% normal / 0.1% anomaly training data?

hard
  • (1) Semi-supervised: train only on normal (assume clean).
  • (2) Isolation Forest / LOF: designed for exactly this ratio.
  • (3) If you have SOME labeled anomalies (>50), oversample them + SMOTE + supervised (imbalanced XGBoost).
  • (4) PU learning if unlabeled is 'mostly normal'.
  • (5) Cost-sensitive: assign high FN cost.
  • Avoid: naive balanced accuracy, single-threshold optimization — use PR AUC.
Permalink

How do you explain why a point is anomalous?

medium
  • (1) Per-feature contribution: which feature's z-score / reconstruction error / SHAP value drove the anomaly.
  • (2) Nearest-normals: 'this looks unusual because normally X, Y differ by …'.
  • (3) Rule-based post-hoc: fit decision tree to Isolation Forest scores.
  • (4) Counterfactual: what minimal change makes it normal?
  • Standard in production ops so on-call engineers can validate alerts.
Permalink

Interview: 'design an unsupervised fraud detection system'.

hard
  • (1) Feature engineering: transaction amount, velocity, device / IP fingerprints, merchant risk, time-of-day, historical account patterns.
  • (2) Stack detectors: (a) Isolation Forest on tabular features, (b) autoencoder on account behavior sequences, (c) graph-based (transaction network) for money-laundering rings.
  • (3) Ensemble scores.
  • (4) Human-in-the-loop: score → analyst → labeled anomalies → semi-supervised uplift.
  • (5) Monitor drift + retrain weekly.
  • (6) Guardrails: false-positive rate SLO.
Permalink

How would you detect anomalies in multi-modal data (image + tabular)?

hard
  • (1) Fuse: (a) tabular through MLP encoder, (b) image through CNN encoder → concat embeddings → joint autoencoder or joint density model.
  • (2) Score per modality then combine (max or weighted sum).
  • (3) Cross-modal consistency: does the image match tabular metadata?
  • (Contrastive score.) Standard in medical (image + EHR), manufacturing (sensor + camera), autonomous vehicles (camera + LiDAR).
Permalink

How would you detect fraud rings (colluding accounts)?

hard
  • (1) Build interaction graph: accounts as nodes, shared devices/IPs/transactions as edges.
  • (2) Community detection (Louvain, Leiden, HDBSCAN on node embeddings).
  • (3) Node2Vec / GraphSAGE embeddings → cluster.
  • (4) Anomalous dense subgraphs / motifs.
  • (5) Combine with per-account anomaly scores.
  • Standard in banking / crypto / marketplace fraud (Uber, Airbnb).
  • GNN-based methods (GraphSAGE + supervised head where labels exist) common.
Permalink

Interview: production fraud detector — daily volume 10M, current FN too high.

hard
  • (1) Investigate: what patterns are FNs?
  • Feature gaps?
  • Concept drift?
  • (2) Enrich features: velocity ratios, device / IP fingerprints, graph features.
  • (3) Ensemble models: existing rules + Isolation Forest + supervised XGBoost on labeled + AE on sequences.
  • (4) Add HITL: analyst-labeled cases feed back weekly.
  • (5) Score threshold tuning by cost-sensitive PR curve.
  • (6) Monitor precision + recall + latency + drift dashboards.
  • (7) Guardrail: never lower FP-rate SLA.
  • (8) Iterate.
Permalink

Interview: model trained in region A must now serve region B — approach?

hard
  • (1) Diagnose: KS + PSI feature-by-feature; MMD/classifier-drift for multivariate.
  • (2) If severe drift, retrain on B data (labeled ideally).
  • (3) Domain adaptation: (a) importance weighting via density ratio, (b) adversarial (DANN), (c) CORAL alignment of covariances, (d) fine-tune on small B-labeled set.
  • (4) Semi-supervised: use B unlabeled + A labeled.
  • (5) Test-time adaptation (BN statistics update on B).
  • (6) Monitor performance in B before full rollout.
Permalink

Interview: 'design anomaly detection for a factory sensor with 200 signals.'

hard
  • (1) EDA: distribution + autocorrelation per signal; note skew / seasonality.
  • (2) Feature engineering: rolling stats + FFT / wavelet features + inter-sensor correlations.
  • (3) Ensemble: (a) univariate rolling z-score / STL, (b) Isolation Forest on features, (c) multivariate autoencoder on signal windows.
  • (4) Score → alerts with severity levels.
  • (5) HITL: operator confirms anomalies → label + retrain.
  • (6) Explainability: feature contribution per alert.
  • (7) Monitor: false alert rate + MTTR.
Permalink

Isolation forest or autoencoder for anomaly detection?

medium
  • Isolation forest for tabular data, as a first attempt in nearly every case: it is fast, needs almost no tuning, handles mixed scales tolerably, and its notion of an anomaly as a point that is easy to isolate is easy to explain to a stakeholder.
  • Autoencoders earn their complexity when anomalies are defined by structure a tree cannot see, such as images, spectrograms, or long sequences where the relationship between dimensions is the signal.
  • They need enough clean normal data, careful capacity control since an over-large autoencoder reconstructs anomalies too, and real tuning effort.
  • The honest default is to run isolation forest, measure it against reviewed alerts, and only reach for reconstruction error when the data is high-dimensional and structured.
Permalink
07

Autoencoders & VAEs

34 concepts

What is self-supervised learning and why does it matter?

medium
  • Self-supervised learning creates labels from the data itself via pretext tasks (masked language modeling, next-token prediction, contrastive views of images) — no manual annotation.
  • It matters because it lets us pretrain huge models on unlabeled data and then transfer to downstream tasks with little labeled data.
  • It's the engine behind BERT, GPT, SimCLR, CLIP, MAE, DINO.
Permalink

How does contrastive learning work?

hard
  • Create two augmented views of the same example (positive pair) and treat other examples as negatives.
  • Train an encoder so positives are close in embedding space and negatives are far, using a loss like InfoNCE.
  • Examples: SimCLR, MoCo, CLIP (contrasts image with text).
  • Produces general-purpose embeddings useful for many downstream tasks.
Permalink

What is an autoencoder and what are the common variants?

medium
  • An autoencoder learns to compress input x through a bottleneck and reconstruct it.
  • Variants: denoising AE (train to reconstruct clean from noisy input), sparse AE (bottleneck via sparsity penalty), variational AE (probabilistic latent, generative), masked AE (mask patches, reconstruct — used in vision pretraining).
  • Uses: dimensionality reduction, denoising, anomaly detection, pretraining.
Permalink

Autoencoder variants — quick tour.

medium
  • (1) Undercomplete AE: bottleneck < input dim → compression.
  • (2) Denoising AE: reconstruct clean from noised input → robustness / representation.
  • (3) Sparse AE: L1 or KL penalty on activations → sparse code.
  • (4) Contractive AE: penalize ||∂h/∂x||_F → local invariance.
  • (5) VAE: probabilistic latent, generative.
  • (6) Masked AE (MAE, He et al. 2022): mask 75% of patches, reconstruct — SOTA vision SSL.
Permalink

Derive the VAE ELBO.

hard
  • log p(x) = log ∫ p(x∣z)p(x \mid z) p(z) dz ≥ Eq(z∣x)[log⁡  p(x∣z)]  −  KL⁡(q(z∣x)  ∣∣  p(z))E_{q(z \mid x)}[\operatorname{log}\;p(x \mid z)]\; - \;\operatorname{KL}(q(z \mid x)\; \mid \mid \;p(z)) (Jensen).
  • ELBO decomposes into reconstruction (log-likelihood of x given z) minus KL to prior.
  • Encoder q_φ(z∣x)(z \mid x) parameterized as Gaussian (μ,  σ2)({\mu}, \;{\sigma}^{2}); decoder p_θ(x∣z)(x \mid z).
  • Reparameterization trick: z = μ + σ ⊙ ε with ε ~ N(0, I) → backprop through sample.
  • Optimize both φ, θ jointly.
Permalink

VAE vs plain autoencoder — the key advantages.

medium
  • (1) Probabilistic latent → generative (sample z ~ p(z), decode → new x).
  • (2) Regularized latent space by KL to prior → smooth interpolation, meaningful arithmetic.
  • (3) Calibrated uncertainty.
  • Weaknesses: VAE samples are typically blurrier than GAN samples (mean-of-modes issue), latent may collapse (posterior collapse), harder to train.
  • Modern hybrid: VQ-VAE, β-VAE, hierarchical VAE (NVAE) address these.
Permalink

β-VAE — what does the β hyperparameter do?

hard
  • Modifies ELBO to Eq[log⁡  p(x∣z)]  −  β  ⋅  KL⁡(q  ∣∣  p)E_{q}[\operatorname{log}\;p(x \mid z)]\; - \;{\beta}\; \cdot \;\operatorname{KL}(q\; \mid \mid \;p). β > 1 pushes KL harder → more independent latent factors → disentangled representations (each latent dim captures one factor: pose, color, size).
  • Cost: worse reconstruction.
  • Trade-off between disentanglement (large β) and fidelity (small β).
  • Foundation of disentangled representation learning; extended by FactorVAE, β-TCVAE.
Permalink

VQ-VAE — the core idea.

hard
  • Discrete latent codes: encoder output quantized to nearest code in a learned codebook (like k-means).
  • Straight-through gradient estimator for backprop.
  • Uses: (1) discrete-latent generative modeling (audio in WaveNet-style, video in Sora / VideoPoet), (2) tokenizer for text-image (DALL-E), (3) speech (Wav2Vec 2).
  • Enables autoregressive priors on discrete latents.
  • Foundation of many modern multi-modal generative models.
Permalink

SimCLR — recipe.

medium
  • Contrastive framework for visual SSL: (1) two random augmentations of each image → positive pair, (2) encoder + projection head, (3) NT-Xent loss (InfoNCE variant): pull positives together, push all other batch items apart.
  • Needs LARGE batch size (~4096) for enough negatives.
  • Removes projection head at downstream time.
  • Started the modern SSL for vision revolution; superseded by SimSiam / DINO / MAE.
Permalink

MoCo (Momentum Contrast) — how does it enable smaller batches?

hard
  • Maintain a queue of negatives from previous batches instead of relying on current batch only.
  • Momentum encoder: EMA of the main encoder → consistent keys in queue as encoder updates slowly.
  • Enables SimCLR-level performance with batches of 256 instead of 4096.
  • MoCo v2 / v3 improved augmentations, projection head, ViT backbone.
  • Foundation of large-scale efficient SSL.
Permalink

BYOL — how does it avoid the need for negatives?

hard
  • Two networks: online (learnable) and target (EMA of online).
  • Feed different augmentations to each; online must predict target's representation via a predictor head.
  • No negatives, no contrastive loss — momentum + asymmetric predictor prevents collapse.
  • Surprising and empirical: BYOL matches SimCLR without negatives.
  • Successor SimSiam removes even the momentum target (relies only on stop-gradient).
Permalink

SimSiam — what makes it minimal?

hard
  • Removes EMA target: both branches share weights.
  • Uses only stop-gradient on one branch and an asymmetric predictor head.
  • Still matches BYOL / SimCLR.
  • Chen & He 2021 showed non-collapse is achieved purely by stop-gradient asymmetry — huge simplification.
  • Foundation of understanding why SSL works: asymmetric optimization dynamics, not contrast, is the key.
Permalink

DINO — self-distillation with no labels.

hard
  • Student network learns to match teacher (EMA of student) on different augmentations of same image.
  • Uses ViT backbone → produces amazing self-attention maps (unsupervised object segmentation emerges!).
  • Centering + sharpening on teacher output prevents mode collapse.
  • DINOv2 (Meta 2023) scaled it to 1B parameters + 142M images → foundation vision encoder rivaling CLIP.
Permalink

MAE (Masked Autoencoder) — He et al. 2022.

medium
  • Mask 75% of image patches, encode only visible patches, decode all patches (mask tokens).
  • Loss: MSE on masked pixel patches.
  • Encoder never sees mask tokens → fast pretraining.
  • Simple, scalable, competitive with contrastive methods.
  • Same idea as BERT for text.
  • Vision SOTA circa 2022-2023 alongside DINO.
  • Foundation of many modern vision transformers (SAM's encoder uses MAE pretraining).
Permalink

CLIP — how does contrastive image-text training work?

medium
  • Train two encoders (image ViT + text Transformer) so that matching (image, caption) pairs are close in shared embedding space, mismatched pairs far.
  • InfoNCE loss over batch.
  • Trained on 400M image-caption pairs from web.
  • Result: zero-shot image classification via 'a photo of a {class}' prompt, cross-modal retrieval, foundation for DALL-E / Stable Diffusion.
  • Rewrote vision-language modeling; superseded by SigLIP, EVA-CLIP, SAM 2.
Permalink

JEPA (I-JEPA, V-JEPA) — LeCun's alternative to generative SSL.

hard
  • Predict abstract features (not pixels) of masked regions in embedding space.
  • Sidesteps pixel-level detail (irrelevant + wasteful) and models context predictively.
  • I-JEPA (images 2023): predicts feature representation of masked target block from context.
  • V-JEPA (video 2024): same for video.
  • Faster + more semantic representations than MAE.
  • LeCun's flagship 'world model' architecture.
Permalink

InfoNCE loss — formula and intuition.

hard
  • L = -log [exp⁡(s(x,  x+)/τ)  /  Σj  exp⁡(s(x,  xj)/τ)][\operatorname{exp}(s(x, \;x^{+}) / {\tau})\; / \;{\Sigma}_{j}\;\operatorname{exp}(s(x, \;x_{j}) / {\tau})] — softmax over one positive and many negatives, temperature τ.
  • Cross-entropy that treats classification 'which of N is the positive?'.
  • Lower bound on mutual information between x and x+x^{+}.
  • Standard loss for contrastive SSL (SimCLR, CLIP, MoCo, etc.).
  • Temperature τ controls concentration: low τ → sharp, hard-negative-focused.
Permalink

Why are augmentations so important in contrastive SSL?

medium
  • Augmentations define the invariances the encoder learns — 'same-object under transformation' becomes the training signal.
  • Strong compositions (crop + color jitter + Gaussian blur) work best (Chen et al. 2020).
  • Too weak → trivial invariances; too strong → destroys semantic content.
  • Rule: task-relevant invariances.
  • Cross-modal alignment (CLIP) uses text as 'super-augmentation'.
Permalink

Word2Vec skip-gram — objective and training.

medium
  • For each target word, predict surrounding context words (window size ~5).
  • Softmax over full vocabulary too expensive → negative sampling: for each positive (target, context) pair, sample k random 'negative' words and train binary logistic.
  • Learns dense word vectors capturing distributional semantics.
  • Levy & Goldberg 2014: mathematically equivalent to implicit factorization of shifted PMI matrix.
Permalink

GloVe vs Word2Vec.

medium
  • GloVe (Pennington 2014): explicit weighted matrix factorization of log co-occurrence counts.
  • Objective: log⁡(Xij)\operatorname{log}(X_{\mathrm{ij}}) ≈ wiw_{i}' wj  +  bi  +  bjw_{j}\; + \;b_{i}\; + \;b_{j}.
  • Batch training on global statistics (vs Word2Vec's per-window streaming).
  • Similar quality to Word2Vec; GloVe often better on analogy tasks, Word2Vec on similarity.
  • Modern: both superseded by contextual embeddings (BERT / sentence-transformers).
  • Still cited in interview questions.
Permalink

Word embedding analogies — why do they work?

hard
  • king - man + woman ≈ queen.
  • Emerges because embeddings capture distributional differences.
  • Actually more fragile than the meme suggests: only works for common frequent analogies, and typical evaluations use cosine similarity + exclusion of query words.
  • Modern contextual embeddings capture analogy implicitly through generation.
  • Interview trap: don't overstate this as evidence of 'reasoning'.
Permalink

Sentence-BERT — how does it improve on BERT for retrieval?

medium
  • BERT's [CLS] token is not a great semantic sentence vector out of the box.
  • SBERT fine-tunes BERT with a siamese architecture on NLI + STS: same encoder on two sentences, minimize distance for paraphrases, maximize for contradictions.
  • Cosine similarity ≈ semantic similarity.
  • Enables efficient sentence retrieval (encode once, cosine at query time), replaced by newer E5 / BGE / OpenAI text-embedding-3 for scale, but SBERT is still baseline.
Permalink

Node2Vec — how does it learn graph node embeddings?

medium
  • Random walks starting from each node → treat walks as 'sentences', apply Word2Vec skip-gram → embed nodes.
  • Biased walks (p, q) control breadth (BFS-like) vs depth (DFS-like): controls whether embeddings capture structural equivalence or community proximity.
  • Uses: link prediction, node classification, community detection preprocessing.
  • Foundational shallow graph embedding; modern replacement: GNNs (GraphSAGE, GAT).
Permalink

Modern graph representation learning — GNNs.

hard
  • Graph Neural Networks aggregate neighbor features iteratively: hvl+1  =  UPDATEh_{v}^{l + 1}\; = \;\mathrm{UPDATE}(hvlh_{v}^{l}, AGG(hul    u  ∈  N(v))\mathrm{AGG}(h_{u}^{l}\;\;u\; \in \;N(v))).
  • Variants: GraphSAGE (sampled aggregation), GAT (attention weights), GCN (spectral), MPNN (message passing).
  • Trained supervised, semi-sup, or self-supervised (GraphCL, BGRL).
  • Uses: recommendation, drug discovery, molecule property prediction, fraud rings.
Permalink

Self-supervised graph learning — approaches.

hard
  • (1) Node-level: contrastive on augmented views (drop edges/nodes/features), GraphCL.
  • (2) Predict masked node attributes (MaskGAE).
  • (3) Motif / structure prediction.
  • (4) Bootstrap approaches (BGRL, analog of BYOL for graphs).
  • Used to pretrain GNNs on unlabeled graphs before fine-tuning on labels.
  • Standard in drug discovery / molecular property pipelines (huge unlabeled compound libraries).
Permalink

Self-supervised audio — wav2vec 2 and HuBERT.

hard
  • Wav2Vec 2 (Facebook 2020): quantize audio features → mask spans → predict quantized targets via contrastive loss.
  • HuBERT: similar but uses clustered representations as pseudo-labels (BERT-style masked prediction).
  • Both dominated pretraining for ASR — supervised fine-tuning with ~10 hours of labeled speech reaches WER competitive with 1000+ hours of pure supervised training.
  • Foundation of Whisper's competitors and many production speech systems.
Permalink

How are multimodal embeddings unified across text / image / audio?

hard
  • Approaches: (1) CLIP-style paired training (image ↔ text).
  • (2) ImageBind (Meta 2023): 6 modalities aligned through image as the 'bridge' — no need for all-pair data.
  • (3) Whisper + CLIP for text-audio alignment.
  • (4) LLaVA-style: project image encoder into LLM embedding space.
  • Common thread: contrastive alignment + a bridge modality.
  • Foundation of multimodal LLMs.
Permalink

Scaling laws for self-supervised pretraining.

hard
  • Downstream performance improves as a power law in (data, params, compute).
  • MAE / CLIP / DINOv2 / vision transformers all show emergent capabilities at scale (rare classes, zero-shot).
  • Diminishing returns: 10× more compute typically gives ~linear gain in loss.
  • Data quality matters more than quantity at very large scale (DINOv2 uses curated 142M images vs LAION's 5B).
  • Modern trend: careful data selection + scale.
Permalink

Mode / representation collapse in SSL — what and why?

hard
  • Encoder outputs the same (or trivially degenerate) representation for all inputs → useless embeddings.
  • Root cause: shortcut solutions minimizing loss without capturing content.
  • Prevention: (1) contrastive negatives, (2) asymmetry (stop-gradient + predictor, BYOL / SimSiam), (3) centering + sharpening (DINO), (4) variance-covariance regularization (VICReg, Barlow Twins), (5) predictor bottleneck.
  • Understanding when/why collapse is avoided is an open research direction.
Permalink

VICReg / Barlow Twins — non-contrastive SSL via covariance regularization.

hard
  • Barlow Twins: cross-correlation of two augmented view embeddings should be identity (diagonal = 1, off-diag = 0 → invariance + de-correlation).
  • VICReg extends with three terms: invariance (MSE between views), variance (each dim has SD > threshold), covariance (off-diag = 0).
  • No negatives, no momentum encoder — just clean regularization.
  • Elegant, competitive with contrastive methods.
Permalink

Fine-tuning vs linear probe vs prompt-tuning — when do you use each?

medium
  • (1) Linear probe: small labeled data + strong SSL encoder → freeze encoder, add linear head.
  • Fast, tiny compute.
  • (2) Fine-tuning: enough labeled data + task shift → unfreeze last k layers or full network.
  • (3) LoRA / adapters: parameter-efficient, when full fine-tune too expensive.
  • (4) Prompt-tuning: LLM-specific — learn soft prompts instead of weights.
  • Rule: start with linear probe as baseline; escalate to fine-tune only if needed.
Permalink

What is a 'foundation model' in the unsupervised sense?

medium
  • Large model pretrained on massive unlabeled data via SSL → generalizes to many downstream tasks with little or no fine-tuning.
  • Examples: GPT / Llama (text), CLIP / DINOv2 (vision), SAM (segmentation), Whisper (speech), ImageBind (multimodal).
  • Bommasani et al. 2021 coined the term.
  • Enables the modern 'pretrain once, adapt many' paradigm — economically transformative in industry.
Permalink

Interview: 'you have unlabeled images — which SSL method should you use?'

hard
  • Ask: (1) Compute budget?
  • MAE is fastest to train.
  • Contrastive (SimCLR) needs huge batches / MoCo needs momentum.
  • (2) Downstream task type?
  • Classification → most methods work; segmentation → DINO or MAE better (spatial features); retrieval → contrastive (SimCLR, CLIP if paired text).
  • (3) Data volume?
  • Small → use pretrained + fine-tune, don't retrain SSL.
  • Large → DINOv2 or MAE.
  • (4) Domain?
  • Medical / satellite / etc — start from ImageNet-pretrained + continue SSL.
Permalink

Interview: 'when should you use a VAE vs GAN vs diffusion for generation?'

hard
  • (1) VAE: fast, latent space useful for interpolation + representation, but blurry samples.
  • Best when interpretable latent matters (drug design, molecule generation).
  • (2) GAN: sharp samples, fast inference, but training unstable + mode collapse.
  • Best for face generation, image editing (StyleGAN).
  • (3) Diffusion: SOTA quality, controllable, but slow sampling (50-1000 steps → mitigated by DDIM, distillation, Flow Matching).
  • Best default for images / video / audio generation in 2024+.
Permalink
08

Topic modeling & text

10 concepts

What is topic modeling and when do you use LDA?

medium
  • Topic modeling discovers latent themes in a document corpus.
  • LDA (Latent Dirichlet Allocation) models each document as a mixture of topics and each topic as a distribution over words.
  • Use it for exploratory analysis of text collections without labels.
  • Modern alternatives: embedding-based topic modeling (BERTopic) — usually more coherent on short/noisy text.
Permalink

TF-IDF weighting — formula and rationale.

easy
  • TF: term frequency in document (raw, log, or sublinear).
  • IDF: log⁡(N  /  dft)\operatorname{log}(N\; / \;\mathrm{df}_{t}) — downweights common words appearing in many documents.
  • Product TF-IDF highlights terms specific to a document.
  • Uses: bag-of-words features, BM25 (probabilistic IDF variant, dominant in classical IR).
  • Modern hybrid: BM25 + dense embeddings for retrieval → Elasticsearch does this natively (kNN + BM25 rrf).
Permalink

BM25 — why is it still competitive with modern retrievers?

medium
  • Okapi BM25 = probabilistic TF-IDF with length normalization + tunable k1k_{1} (TF saturation) + b (length norm).
  • Very fast (inverted index), well-understood, competitive on exact-match / keyword-heavy queries where dense embeddings miss.
  • Modern practice: hybrid retrieval BM25 + dense embeddings + RRF (reciprocal rank fusion) = best-of-both.
  • Standard in Elasticsearch, Vespa, Qdrant hybrid mode, all modern RAG pipelines.
Permalink

Vector search / approximate nearest neighbors — main algorithms.

medium
  • (1) HNSW (Hierarchical Navigable Small World): graph-based, dominant in industry (Qdrant, Weaviate, pgvector).
  • (2) IVF-PQ (inverted file + product quantization): FAISS default for very large indexes.
  • (3) ScaNN (Google): partition + reranking.
  • (4) LSH (locality-sensitive hashing): older, still used for near-duplicate detection.
  • Trade-off: recall vs latency vs memory.
  • HNSW default for < 10M vectors; IVF-PQ for > 100M.
Permalink

How is RAG retrieval an unsupervised problem?

medium
  • RAG (retrieval-augmented generation) retrieves passages from a corpus based on unsupervised embeddings (no labels of 'relevant' pairs).
  • Uses: dense retrieval (sentence embeddings + cosine + ANN), often hybrid with BM25.
  • Fine-tuning the retriever is supervised (via query-doc labels), but out-of-the-box embeddings + cosine already give strong retrieval unsupervised.
  • Foundation of modern LLM-augmented systems.
Permalink

PMI and PPMI — what they measure.

hard
  • PMI(x, y) = log[P(x, y) / (P(x) P(y))].
  • Positive → co-occur more than random.
  • Negative → less than random.
  • PPMI = max(0, PMI) — drops negative values, more stable.
  • Uses: word association (Church & Hanks), building sparse word co-occurrence matrices before SVD (LSA); implicit target of Word2Vec (Levy & Goldberg).
  • Foundation of distributional semantics.
Permalink

How do you cluster / retrieve code snippets?

hard
  • (1) Code-specific embeddings: CodeBERT, StarCoder, GTE-code, OpenAI text-embedding-3 handle code decently.
  • (2) Combine with AST features / API signatures.
  • (3) Cluster with HDBSCAN or embed + LSH for near-duplicate detection (deduplicate training data of code models).
  • (4) Retrieval-augmented: embed function bodies, retrieve similar for autocomplete.
  • Standard in GitHub Copilot's underlying retrieval, code deduplication for training.
Permalink

Interview: users complain search returns irrelevant results — how do you fix?

hard
  • (1) Analyze query logs: are they specific / broad / navigational / typos?
  • (2) Baseline: BM25 lexical.
  • Add: (3) Dense retrieval (sentence-transformers).
  • (4) Hybrid: BM25 + dense + RRF fusion.
  • (5) Cross-encoder reranker on top 100 → top 10.
  • (6) Query rewriting (HyDE / expansion for underspecified queries).
  • (7) Feedback logging (clicks) → LTR fine-tuning.
  • (8) A/B measure: click-through, dwell time, session success.
  • (9) Evaluate with human relevance judgments quarterly.
Permalink

Interview: choose a sentence embedding model for a startup RAG.

medium
  • Constraints: quality vs cost vs latency vs vector dimension.
  • (1) OpenAI text-embedding-3-small: cheap ($0.02/1M), 1536 dim, hosted.
  • (2) Cohere embed-v3: multilingual, 1024 dim.
  • (3) Open-source: bge-small-en-v1.5 (384 dim, fast, high MTEB score), E5-large-v2 (1024 dim, top-tier).
  • (4) DIY fine-tuning: contrastive on domain pairs.
  • Rule: start with bge-small / OpenAI-3-small for speed; benchmark on your data (MTEB score alone is misleading).
Permalink

Interview: 'you have 1M support tickets — how do you categorize them?'

hard
  • (1) Deduplicate near-identical tickets (LSH).
  • (2) Sentence-transformer embed (bge-small or multilingual if needed).
  • (3) UMAP to 20 dim.
  • (4) HDBSCAN → topic clusters.
  • (5) c-TF-IDF for cluster keywords → auto-label.
  • (6) Manual review of top 20 clusters + rename/merge/split with domain team.
  • (7) Train classifier on labeled clusters for future ticket routing.
  • (8) Monitor cluster drift monthly + retrain HDBSCAN if new topics emerge (novelty detection).
Permalink
09

Recommender & association

18 concepts

Collaborative filtering — how does it use unsupervised methods?

medium
  • Learn user + item embeddings from interaction matrix (implicit or explicit ratings).
  • No content features required — 'unsupervised' in the sense that only interactions are used.
  • Methods: (1) SVD / matrix factorization (Netflix Prize), (2) ALS (Spark), (3) Neural CF, (4) Two-tower / dual-encoder models, (5) Graph-based (LightGCN).
  • Cold-start problem (new user / item) needs content features → hybrid.
Permalink

Implicit feedback vs explicit ratings in recsys.

hard
  • Explicit: users give a star rating.
  • Rare in practice.
  • Implicit: clicks, views, time-on-page, purchases → 'positive' signal only, no negatives.
  • Weight positive (confidence) by frequency; treat unobserved as low-confidence negatives (Hu-Koren-Volinsky weighted ALS).
  • BPR: pairwise ranking loss on 'clicked > not-clicked'.
  • Standard in industry: Spark ALS (implicit), Amazon DSSTNE, YouTube's dual-tower.
Permalink

How do you handle the cold-start problem?

medium
  • New user or item with no interactions.
  • Solutions: (1) Content-based features (user profile, item text/image → embed) → hybrid recsys.
  • (2) Popularity-based fallback.
  • (3) Bandits: explore new items early (Thompson sampling on Beta posteriors).
  • (4) Meta-learning to warm-start embeddings.
  • (5) Ask onboarding questions.
  • Every real production recsys hybridizes CF + content to handle cold-start.
Permalink

Two-tower recsys — architecture and use.

hard
  • User tower: NN encoding user features / history → user embedding.
  • Item tower: NN encoding item features → item embedding.
  • Score = cosine or dot product.
  • Trained with softmax over batch (in-batch negatives).
  • Deployment: precompute item embeddings, index with ANN; user embedding computed at query time → real-time retrieval.
  • Standard in YouTube, TikTok, Instagram, LinkedIn retrieval-stage recsys.
Permalink

Association rule mining — Apriori & FP-Growth.

medium
  • Find frequent itemsets in transactions (e.g. 'diapers → beer').
  • Apriori: level-wise search, prune non-frequent supersets — slow, many DB scans.
  • FP-Growth: build FP-tree in one scan, mine recursively — much faster.
  • Metrics: support (frequency), confidence (P(consequent  ∣  antecedent))(P(\mathrm{consequent}\; \mid \;\mathrm{antecedent})), lift (association strength vs random).
  • Classic market-basket analysis; modern replaced by embedding-based recsys but still asked in interviews.
Permalink

How does image similarity search work in production?

medium
  • (1) Extract embeddings: CLIP / DINOv2 / SigLIP → 512-1024 dim vectors.
  • (2) L2-normalize + index in FAISS / Qdrant / pgvector (HNSW / IVF-PQ).
  • (3) Query: embed uploaded image, ANN search, rerank with cross-encoder if precision needed.
  • (4) Filter by metadata (category, brand).
  • Production examples: Pinterest visual search, Google Images similar, Bing 'search by image'.
Permalink

Market basket analysis — modern approach.

hard
  • Classical: association rules (Apriori/FP-Growth).
  • Modern: (1) product embeddings via word2vec-on-baskets ('prod2vec'), (2) session-based recsys (GRU4Rec, SASRec) on sequential purchases, (3) graph embeddings on co-purchase network, (4) contextual bandits for personalization.
  • Association rules still useful for interpretable insights + regulatory reporting; embeddings win on prediction / recall metrics.
Permalink

Unsupervised image segmentation — approaches.

hard
  • (1) Classical: k-means or mean-shift on pixel color + position (SLIC superpixels).
  • (2) Spectral clustering on affinity graph.
  • (3) Modern DL: DINO's self-attention maps segment objects without any supervision.
  • (4) SAM (Segment Anything Model, Meta 2023): promptable segmentation trained on 1B masks — used with 'automatic mask' mode for full-image unsupervised segmentation.
  • (5) STEGO (Hamilton et al. 2022): contrastive semantic segmentation.
Permalink

Zero-shot image classification via CLIP — mechanism.

medium
  • (1) Encode candidate class names as text ('a photo of a dog', 'a photo of a cat') → text embeddings.
  • (2) Encode query image → image embedding.
  • (3) Cosine similarity → class with max similarity is prediction.
  • No training on class labels.
  • Weakness: sensitive to prompt engineering; competitive with fine-tuned models on many benchmarks.
  • Foundation of open-vocabulary classification and detection (OWL-ViT, GLIP).
Permalink

How does unsupervised learning help with noisy labels?

hard
  • (1) Cluster the data unsupervisedly; check consistency of labels within clusters — outliers likely mislabeled.
  • (2) Confidence learning (Northcutt et al., cleanlab): use self-consistency + probabilistic label pruning.
  • (3) Small-loss trick during training (Co-teaching): assume noisy samples have larger loss; ignore top-loss fraction.
  • (4) Semi-supervised: leverage unlabeled or auto-labeled data.
  • Foundation of modern label cleaning at scale.
Permalink

Interview: design a recommender for a new streaming service.

hard
  • (1) Cold-start heavy: no history.
  • Start with content-based recsys (title/description embeddings, genre, actors → cosine similarity to what user selects).
  • (2) Log implicit signals from day 1 (plays, completes).
  • (3) After N interactions, hybrid: two-tower CF + content features.
  • (4) Diversity: MMR reranking + calibration.
  • (5) Bandits for exploration of new content.
  • (6) Evaluate: retention, watch time (long-term), CTR/CVR (short-term), diversity, novelty.
  • (7) A/B test each change.
Permalink

Interview: your model's accuracy dropped 15% overnight — how do you diagnose?

hard
  • (1) Rule out incidents first: pipeline break, feature-source outage, label logging bug.
  • (2) Check input distributions: KS + PSI per feature vs baseline.
  • (3) Check label distribution + prediction distribution shift.
  • (4) Segment: is drop uniform or in specific segment?
  • (5) Retrain on last 7 days as sanity check.
  • (6) Post-mortem: if drift, decide short-term (rollback / hotfix threshold) vs long-term (retrain cadence + monitoring).
  • (7) Add automated alert for the specific drift metric caught.
Permalink

Interview: you must ship an embedding service serving 100M vectors, 10ms p95 latency.

hard
  • (1) Choose embedding model by quality + latency (E5-small / bge-small / OpenAI text-embedding-3-small).
  • (2) Batch-embed the corpus offline.
  • (3) Index in HNSW (Qdrant, pgvector, Weaviate) — M=32, efconstruction=200\mathrm{ef}_{\mathrm{construction}} = 200.
  • (4) Serve queries: encode + ANN.
  • (5) Cache popular queries.
  • (6) Sharding by partition key + horizontal scale.
  • (7) Cold-index rebuild + hot-swap for updates.
  • (8) Monitor: recall@k, latency percentiles, index memory.
  • (9) Consider IVF-PQ if memory-constrained.
  • (10) Backup index.
Permalink

Interview: how do you monitor drift in an embedding-based retrieval system?

hard
  • (1) Query distribution drift: KS/PSI on query embedding norms, cluster distribution of queries over time.
  • (2) Corpus drift: same on doc embeddings.
  • (3) Retrieval quality: NDCG@k on golden query set + regression on click-through rates.
  • (4) Semantic drift: cluster queries weekly, compare centroids vs baseline (ARI between weekly clusterings).
  • (5) Model drift: recompute embeddings with new version, measure alignment (Procrustes / linear-CKA) to old version.
  • Alert on all four.
Permalink

Interview: 'when should you use graph-based methods over tabular?'

medium
  • (1) Data has natural relational structure (users↔items, molecules, transactions, social nets).
  • (2) Prediction depends on neighborhood (link prediction, fraud rings, drug interactions).
  • (3) Rich metadata on nodes + edges.
  • (4) You want to explain via relational features (visible connections).
  • Don't use graphs when: (1) data is purely tabular independent samples, (2) no clear notion of relation, (3) added complexity outweighs marginal gains.
Permalink

Interview: 'you have to embed 100M documents monthly — cost strategy?'

hard
  • (1) Only re-embed changed / new documents (change-data-capture).
  • (2) Choose model by cost/quality tradeoff: OpenAI-3-small ($0.02/1M tokens) vs OSS bge-small hosted on your infra (initial GPU spend, then free).
  • (3) Batch requests → economies of scale in API cost.
  • (4) Truncate long docs after N tokens or chunk + aggregate.
  • (5) Cache embeddings by content hash.
  • (6) Consider Matryoshka embeddings (OpenAI-3 supports variable dim) → smaller dim for retrieval, full dim for reranking.
Permalink

Interview: 'summarize when unsupervised learning wins in production.'

medium
  • Unsupervised wins when: (1) labels are expensive / impossible (fraud rings, anomalies, cold-start recsys).
  • (2) Data has natural structure worth exposing (customer segments, log templates, image duplicates).
  • (3) You need representations for downstream tasks (embeddings for RAG, features for classifiers).
  • (4) You need to detect novelty / drift (production monitoring).
  • (5) You want to enable exploratory analysis at scale (topic discovery, log clustering).
  • Rule: it's rarely the final answer — usually feeds a supervised or business-decision layer.
Permalink

How do you make recommendations for a brand-new user?

medium
  • Fall back through a ladder of decreasing personalization.
  • With no history, serve popularity, ideally popularity within whatever context you do know, such as country, device, or referral source, which is far better than global top items.
  • Use any content signal available: the first item viewed, a stated interest at signup, or the search query, and recommend by content similarity rather than collaborative signal.
  • Deliberately explore in the first sessions, since early diverse impressions are how you acquire the data personalization needs, and a purely greedy policy starves itself.
  • Then switch to the collaborative model once the user crosses an interaction threshold.
  • A hybrid model that consumes content features handles this natively instead of needing a separate rule.
Permalink
You finished the lesson
Now test what stuck.
Interview flashcardsTake the quizBrowse other topics