EasyDeepLearn
Unsupervised Learning · section 5 of 9

Linear dimensionality reduction

38 interview questions on linear dimensionality reduction, each answered in full. Free to read, no account needed.

PCA vs t-SNE vs UMAP — when do you use each?

medium
  • PCA is a linear, deterministic projection that preserves global variance — use it for compression, denoising, or as input to another model. t-SNE and UMAP are nonlinear and preserve local neighborhoods — use them for 2D/3D visualization, not for downstream modeling.
  • UMAP is faster than t-SNE and preserves more global structure.
  • Never interpret t-SNE distances between clusters as meaningful.
#dimensionality-reduction#visualizationPermalink & quiz →

How do you decide how many PCA components to keep?

easy
  • Look at the cumulative explained-variance ratio and keep enough components to reach a target (e.g., 90% or 95%).
  • You can also inspect the scree plot for an elbow, or pick components based on downstream cross-validation performance.
  • Standardize features before PCA when they are on different scales.
#dimensionality-reductionPermalink & quiz →

Derive PCA — what does it optimize?

hard
  • Given centered X, find orthonormal w that maximizes Var⁡(Xw)  =  w\operatorname{Var}(\mathrm{Xw})\; = \;w' Σ w subject to ||w||=1.
  • Solution: eigenvector of Σ = X'X/n corresponding to the largest eigenvalue.
  • Subsequent components: next largest eigenvalue, orthogonal.
  • Equivalently: minimizes squared reconstruction error over rank-k subspaces (Eckart-Young).
  • Two views, same answer.
#dimensionality-reduction#pcaPermalink & quiz →

PCA via SVD — the connection.

medium
  • For centered X (n × p), SVD gives X = UΣV'.
  • Principal directions = columns of V (right singular vectors).
  • Principal scores = UΣ.
  • Explained variance ratios ∝ diag⁡(Σ)2\operatorname{diag}({\Sigma})^{2}.
  • More numerically stable than eigendecomposing X'X (which squares condition number).
  • Standard scikit-learn PCA uses SVD internally.
  • Randomized SVD (Halko-Tropp) approximates top-k in O(np log k) — big-data default.
#dimensionality-reduction#pcaPermalink & quiz →

Why standardize before PCA (usually)?

easy
  • PCA maximizes variance in the original units.
  • If features have wildly different scales (income in $ vs age in years), the high-variance one dominates → useless components.
  • Standardize (z-score) so each contributes equally, unless: features are on the same natural scale (pixel intensities, log-returns) where scale carries meaning.
  • Alternative: correlation-matrix PCA is equivalent to z-scored PCA.
#dimensionality-reduction#pcaPermalink & quiz →

When does PCA fail?

medium
  • (1) Non-linear structure (rolls, spirals) — PCA sees only linear correlations.
  • (2) Discrete / categorical features (one-hot creates artificial variance).
  • (3) Skewed features — one heavy-tail feature dominates; log-transform first.
  • (4) Interpretability priority — PC1 mixes many features, hard to name.
  • (5) Anomaly-heavy data — outliers distort covariance; use robust PCA.
  • Alternatives: KernelPCA, autoencoders, ICA depending on the failure mode.
#dimensionality-reduction#pcaPermalink & quiz →

Kernel PCA — when and how?

hard
  • Apply the kernel trick to PCA: implicitly map data to a feature space Φ(x) via kernel K(x, y) = ⟨Φ(x), Φ(y)⟩, then do PCA there.
  • Captures non-linear structure without explicit feature construction.
  • Common kernels: RBF, polynomial, cosine.
  • Cost O(n2)O(n^{2}) memory (kernel matrix) — doesn't scale beyond ~10k.
  • Modern replacement: autoencoders / UMAP for larger data.
#dimensionality-reduction#pcaPermalink & quiz →

ICA vs PCA — the key difference.

hard
  • PCA: finds uncorrelated components maximizing variance.
  • ICA: finds statistically independent components — stronger requirement.
  • Uses non-Gaussianity (kurtosis, negentropy) to find directions that look non-Gaussian.
  • Classic use: blind source separation (cocktail party — separate mixed audio signals).
  • Requires nsourcesn_{\mathrm{sources}} ≤ nsensorsn_{\mathrm{sensors}} and at most one Gaussian source.
  • FastICA is the standard implementation.
#dimensionality-reductionPermalink & quiz →

Non-negative Matrix Factorization (NMF) — when to use?

medium
  • Factor X ≈ WH with W, H ≥ 0.
  • Enforces additive parts-based representation.
  • Uses: (1) topic modeling on TF-IDF (topics as sparse non-negative bases), (2) audio spectrogram decomposition, (3) image parts (Lee & Seung's faces).
  • More interpretable than PCA when non-negativity is natural.
  • Fit by multiplicative updates or ADMM.
  • Rank must be chosen (CV / stability).
#dimensionality-reductionPermalink & quiz →

Sparse coding — what is it?

hard
  • Learn overcomplete dictionary D such that x ≈ Dα with sparse α (few non-zero entries).
  • Uses: image compression, denoising, feature learning.
  • Fit alternates between (1) LASSO for α given D, (2) dictionary update for D given α.
  • Foundation of Olshausen & Field's V1 modeling and later dictionary learning.
  • Related to compressed sensing.
#dimensionality-reduction#representation-learningPermalink & quiz →

Random projection — how does it work?

medium
  • Johnson-Lindenstrauss: for ε > 0, a random Gaussian matrix R (m × p) with m  =  O(log⁡  n  /  ε2)m\; = \;O(\operatorname{log}\;n\; / \;{\varepsilon}^{2}) preserves pairwise distances up to (1±ε) with high probability.
  • Extremely cheap (no data-dependent fit).
  • Uses: dimensionality reduction for huge sparse data, approximate nearest neighbors (LSH), streaming.
  • Less informative than PCA but scales linearly.
#dimensionality-reductionPermalink & quiz →

Linear Discriminant Analysis (LDA) vs PCA — the difference.

medium
  • PCA is unsupervised — max variance.
  • LDA is supervised — projects to maximize class separability: max Tr(SB)  /  Tr(SW)\mathrm{Tr}(S_{B})\; / \;\mathrm{Tr}(S_{W}), where SB  =  between−classS_{B}\; = \;\mathrm{between} - \mathrm{class} scatter, SW  =  within−classS_{W}\; = \;\mathrm{within} - \mathrm{class} scatter.
  • Yields at most (k-1) components for k classes.
  • Uses: dimensionality reduction as preprocessing for classifiers, face recognition (Fisherfaces).
  • Assumes normally distributed classes with equal covariance.
#dimensionality-reductionPermalink & quiz →

Canonical Correlation Analysis (CCA) — use case.

hard
  • Finds pairs of linear projections (u, v) of two multivariate datasets X, Y that maximize their correlation.
  • Uses: (1) multi-view learning — combine images and text embeddings, (2) genomics (SNPs vs gene expression), (3) recommendation with side information.
  • Deep CCA extends with neural nets on each side.
  • Foundational for cross-modal alignment.
#dimensionality-reductionPermalink & quiz →

t-SNE perplexity — what does it control?

medium
  • Perplexity ≈ effective number of neighbors each point 'considers' when building the conditional similarity distribution.
  • Typical range: 5-50.
  • Low perplexity → local, may fragment groups.
  • High → global, may merge distinct clusters.
  • Rule of thumb: try 5, 30, 50 and inspect.
  • Don't over-tune to make plots look better — the map should be robust across a range.
#dimensionality-reduction#visualizationPermalink & quiz →

Top t-SNE pitfalls to avoid.

hard
  • (1) Cluster sizes in the map are not meaningful — t-SNE inflates dense clusters.
  • (2) Inter-cluster distances are NOT preserved — 'far' clusters aren't necessarily far in data space.
  • (3) Random-seed sensitivity → run multiple times.
  • (4) Only meaningful in 2/3D.
  • (5) NEVER feed t-SNE embedding into a downstream model — use PCA or UMAP instead.
  • Wattenberg's 'How to Use t-SNE Effectively' is required reading.
#dimensionality-reduction#visualizationPermalink & quiz →

UMAP vs t-SNE — practical differences.

medium
  • UMAP: faster (O(n114)  vs  O(n  log⁡  n)  with  Barnes−Hut  but  higher  constant)(O(n^{1}14)\;\mathrm{vs}\;O(n\;\operatorname{log}\;n)\;\mathrm{with}\;\mathrm{Barnes} - \mathrm{Hut}\;\mathrm{but}\;\mathrm{higher}\;\mathrm{constant}), preserves more global structure (inter-cluster distances less-meaningful but somewhat interpretable), supports supervised / semi-supervised variants, can transform new points (t-SNE cannot).
  • Both are for visualization primarily.
  • UMAP has become the default for 2D projection in scRNA-seq, embeddings inspection, exploratory analysis.
#dimensionality-reduction#visualizationPermalink & quiz →

UMAP key hyperparameters.

medium
  • (1) nneighborsn_{\mathrm{neighbors}}: local vs global (5-15 local, 50-100 global; default 15).
  • (2) min⁡dist\operatorname{min}_{\mathrm{dist}}: minimum spacing in embedding — small (0.1) preserves clusters, large (0.5) spreads points.
  • (3) ncomponentsn_{\mathrm{components}}: usually 2 for viz, 10-50 for downstream.
  • (4) metric: euclidean, cosine, correlation, hamming, custom callable.
  • (5) randomstate\mathrm{random}_{\mathrm{state}} for reproducibility.
  • Tune nneighbors  +  min⁡distn_{\mathrm{neighbors}}\; + \;\operatorname{min}_{\mathrm{dist}} together; try 3-4 combinations.
#dimensionality-reduction#visualizationPermalink & quiz →

Isomap — what does it do?

hard
  • Non-linear DR that preserves geodesic (manifold) distances.
  • Steps: (1) k-NN graph, (2) all-pairs shortest paths (Dijkstra / Floyd-Warshall) → geodesic distance matrix, (3) MDS on geodesic distances → low-dim embedding.
  • Discovers curved manifolds where Euclidean distance fails (roll, sphere).
  • Precursor to UMAP; still useful for interpretability and small datasets.
#dimensionality-reductionPermalink & quiz →

Locally Linear Embedding (LLE) — how does it work?

hard
  • (1) Find k nearest neighbors for each point.
  • (2) Reconstruct each point as linear combination of neighbors: minimize ||xi  −  Σx_{i}\; - \;{\Sigma} wijw_{\mathrm{ij}} xjx_{j}||² with Σ wij  =  1w_{\mathrm{ij}}\; = \;1.
  • (3) Find embedding yiy_{i} minimizing ||yi  −  Σy_{i}\; - \;{\Sigma} wijw_{\mathrm{ij}} yjy_{j}||² preserving the same weights.
  • Captures non-linear structure with a linear reconstruction locally.
  • Sensitive to k.
  • Modern: superseded by UMAP but still cited in manifold-learning theory.
#dimensionality-reductionPermalink & quiz →

Multidimensional Scaling (MDS) — variants.

medium
  • Given pairwise distance matrix D, find low-dim embedding preserving distances.
  • Classical MDS: closed form via eigendecomposition of doubly-centered D2D^{2} — equivalent to PCA on distance data.
  • Metric MDS: minimize stress = Σ (∣∣yi  −  yj∣∣  −  dij)2( \mid \mid y_{i}\; - \;y_{j} \mid \mid \; - \;d_{\mathrm{ij}})^{2}.
  • Non-metric MDS: preserves only rank order of distances.
  • Uses: psychometrics, marketing perception maps, small-n visualization.
#dimensionality-reductionPermalink & quiz →

Autoencoder for dimensionality reduction — pros and cons.

medium
  • Pros: non-linear, scalable (SGD), transformable (new data → embedding), can be regularized (denoising, sparse, contractive).
  • Cons: no orthogonality (hard to interpret), no explained-variance ratios, sensitive to hyperparameters, harder than PCA.
  • Rule: for tabular data < 50 dim, PCA is enough.
  • For images / audio / very high dim, autoencoders (especially convolutional) dominate.
#dimensionality-reduction#representation-learningPermalink & quiz →

What is the manifold hypothesis?

medium
  • Real high-dimensional data (images, audio, text) lies on a low-dimensional manifold embedded in the high-dim space — e.g. natural images inhabit ~O(hundreds)-dim manifold within millions of pixels.
  • Motivates non-linear DR (Isomap, UMAP, autoencoders): find that low-dim coordinate system.
  • Foundation of modern representation learning: features are more useful in manifold coordinates than raw pixels.
#dimensionality-reduction#theoryPermalink & quiz →

How do you estimate the intrinsic dimension of a dataset?

hard
  • (1) Correlation dimension (Grassberger-Procaccia): slope of log⁡(number  of  pairs  within  ε)\operatorname{log}(\mathrm{number}\;\mathrm{of}\;\mathrm{pairs}\;\mathrm{within}\;{\varepsilon}) vs log ε.
  • (2) MLE-based (Levina-Bickel): from distances to k nearest neighbors.
  • (3) TwoNN (Facco et al., 2017): ratio of first two neighbor distances.
  • (4) PCA scree-plot elbow.
  • Real datasets typically have intrinsic dim much smaller than ambient dim (MNIST ambient=784, intrinsic ≈ 12-14).
#dimensionality-reduction#theoryPermalink & quiz →

Truncated SVD vs PCA on sparse data.

medium
  • PCA subtracts the mean → densifies sparse matrices (bad for TF-IDF, one-hot).
  • Truncated SVD (scikit-learn's TruncatedSVD, also known as LSA in text) does SVD without centering → preserves sparsity, scales to millions of features.
  • Standard for topic modeling (LSA), sparse recommender matrices.
  • Coordinates and variance interpretations same as PCA, just without the mean shift.
#dimensionality-reduction#pcaPermalink & quiz →

Incremental PCA — when do you need it?

medium
  • Dataset doesn't fit in RAM.
  • Process mini-batches, update running estimate of principal components (block Lanczos or Ross et al.'s method). scikit-learn: IncrementalPCA.
  • Constant memory O(dk) rather than O(np).
  • Slight accuracy tradeoff vs full SVD.
  • Standard for streaming / out-of-core PCA on terabyte-scale tabular data.
#dimensionality-reduction#pcaPermalink & quiz →

Robust PCA — what problem does it solve?

hard
  • Standard PCA is sensitive to outliers (single bad row can rotate components).
  • Robust PCA (Candès et al.): decompose X = L + S where L is low-rank (clean data) and S is sparse (outliers).
  • Solved via convex relaxation: min ||L||_* + λ ||S||_1 (Principal Component Pursuit).
  • Uses: video background subtraction, corrupt entry recovery, anomaly detection.
#dimensionality-reduction#pcaPermalink & quiz →

Why does truncated SVD denoise?

hard
  • Signal typically lies in low-rank subspace; noise spreads across all singular directions.
  • Discarding small singular values throws away noise-dominant components while keeping signal.
  • Foundation of PCA-based denoising, spectral clustering, and matrix completion (Netflix Prize).
  • Also called optimal shrinkage — Gavish-Donoho gives an explicit optimal threshold for Gaussian noise.
#dimensionality-reductionPermalink & quiz →

Word embeddings as unsupervised DR of text.

medium
  • Word2Vec (skip-gram, CBOW), GloVe: predict context words → learn dense word vectors capturing co-occurrence structure.
  • Word2Vec is implicit PMI matrix factorization (Levy & Goldberg 2014).
  • GloVe: explicit log-count factorization.
  • FastText: adds sub-word info for OOV.
  • Modern replacement: contextual embeddings (BERT, sentence-transformers) — the dominant text representation since 2018.
#dimensionality-reduction#nlp#representation-learningPermalink & quiz →

How do modern sentence / doc embeddings work?

medium
  • Sentence-BERT (Reimers-Gurevych): fine-tune BERT with siamese contrastive loss on paraphrase / NLI pairs → semantically-meaningful vectors, cosine similarity ≈ semantic similarity.
  • Modern: E5, BGE, GTE, OpenAI text-embedding-3, Cohere embed-v3 — trained on contrastive tasks over huge multilingual corpora.
  • Used everywhere in RAG, retrieval, semantic search, clustering.
#dimensionality-reduction#nlp#representation-learningPermalink & quiz →

Matrix completion — how does it relate to unsupervised learning?

hard
  • Recover missing entries in a matrix by assuming low-rank structure.
  • Solve min rank⁡(M)\operatorname{rank}(M) s.t. observed entries match — NP-hard, relaxed to min ||M||_* (nuclear norm).
  • Foundation of collaborative filtering (Netflix Prize), missing-data imputation.
  • Modern: Alternating Least Squares (Spark ALS), deep matrix factorization, or neural collaborative filtering.
#dimensionality-reduction#applicationsPermalink & quiz →

Principal Components Regression (PCR) — what does it do?

hard
  • Regress on PCA components instead of raw features: (1) PCA → keep top k components, (2) OLS on those.
  • Reduces multicollinearity + acts as regularization.
  • Downside: PCA is unsupervised → top components may not correlate with Y.
  • Alternative: PLS (Partial Least Squares) finds components maximizing covariance with Y — supervised DR + regression.
  • PLS usually better for prediction with many correlated features.
#dimensionality-reduction#regressionPermalink & quiz →

Partial Least Squares (PLS) — how is it different from PCR?

hard
  • PCR: PCA components chosen by max variance in X (Y-ignorant).
  • PLS: components maximize covariance with Y (supervised DR).
  • Better prediction for high-dim, multicollinear data (spectroscopy, genomics, chemometrics).
  • Fit iteratively: extract latent variable, deflate X and Y, repeat.
  • Bridge between PCA and OLS.
  • Standard in chemometrics, increasingly used in bioinformatics.
#dimensionality-reduction#regressionPermalink & quiz →

LSA (Latent Semantic Analysis) — how does it relate to modern retrieval?

medium
  • Truncated SVD on TF-IDF document-term matrix.
  • Rows → topic vectors for docs, columns for terms.
  • Uses: information retrieval (query as bag-of-words → SVD-project → cosine-retrieve).
  • Precursor to modern dense retrieval (sentence-transformers, dual encoders).
  • Still useful as baseline; interpretable in low-resource languages where fine-tuned encoders aren't available.
#nlp#text#dimensionality-reductionPermalink & quiz →

Matrix factorization for recsys — objective.

medium
  • Model rating ruir_{\mathrm{ui}} ≈ pup_{u}' * qiq_{i} where pup_{u} ∈ RkR^{k} is user vector, qiq_{i} is item vector.
  • Minimize Σ (rui  −  puqi)2  +  λ(r_{\mathrm{ui}}\; - \;p_{u}q_{i})^{2}\; + \;{\lambda} (∣∣pu∣∣2  +  ∣∣qi∣∣2)( \mid \mid p_{u} \mid \mid ^{2}\; + \; \mid \mid q_{i} \mid \mid ^{2}) over observed entries.
  • Solved with SGD or ALS.
  • Handles missing entries naturally (only sum over observed).
  • Extensions: biases (puqi  +  bu  +  bi  +  μ)(p_{u}q_{i}\; + \;b_{u}\; + \;b_{i}\; + \;{\mu}), implicit feedback (BPR loss, weighted ALS in Spark).
#applications#dimensionality-reductionPermalink & quiz →

Interview: 'you have 500 features, most correlated — how do you preprocess?'

medium
  • (1) Correlation heatmap → drop obviously redundant.
  • (2) Domain-driven grouping (aggregate related features first).
  • (3) Missing-value pattern check.
  • (4) Standardize.
  • (5) PCA / Truncated SVD → keep 95% variance (usually 30-80 components).
  • (6) If non-linear structure suspected: UMAP (for viz), autoencoder (for downstream).
  • (7) For interpretable model: keep top-loadings; for pure prediction: use the full transformation.
  • (8) Consider L1-regularized model as alternative that learns sparsity end-to-end.
#interview#dimensionality-reductionPermalink & quiz →

Interview: 'when should you use PCA vs autoencoder for dim reduction?'

medium
  • PCA: (1) linear structure, (2) < 200 features, (3) want interpretability (loadings), (4) fast + closed form, (5) baseline.
  • Autoencoder: (1) non-linear structure (images, sequences), (2) very high dim (thousands+), (3) enough data to train, (4) willing to accept less interpretability, (5) transformable to new points, (6) regularizable (denoising, sparsity).
  • Rule: start with PCA — if downstream metric plateaus, try autoencoder to catch non-linear info.
#interview#dimensionality-reductionPermalink & quiz →

How many principal components do you keep, and what does 95% variance actually guarantee?

medium
  • Common choices are a cumulative variance threshold, the knee in the scree plot, or the number that maximizes downstream performance, and the last is the only one tied to your goal.
  • What a 95% variance threshold guarantees is only that reconstruction error in the least-squares sense is small; it says nothing about whether the discarded 5% contained the signal you care about.
  • This matters because variance is not relevance: a low-variance direction can carry the entire class distinction, which is exactly why PCA is unsupervised and can discard the label-bearing axis.
  • If your aim is prediction, treat the component count as a hyperparameter tuned inside cross-validation, and remember to fit PCA on the training fold only.
#dimensionality-reduction#pcaPermalink & quiz →

What conclusions can you not draw from a t-SNE plot?

hard
  • Cluster sizes are meaningless, because t-SNE expands sparse regions and compresses dense ones to fit everything into two dimensions, so a visually large blob is not a numerous group.
  • Distances between clusters are also unreliable: the method preserves local neighbourhoods and deliberately sacrifices global geometry, so two well-separated blobs may be closer in the original space than they look.
  • Apparent gaps can be artefacts of the perplexity setting, and running it twice with different seeds gives different layouts.
  • What you can read is which points are neighbours of which.
  • Treat it as a qualitative sanity check, never as evidence for the number of clusters, and use UMAP if you need somewhat better global structure.
#dimensionality-reduction#visualizationPermalink & quiz →

Practise Unsupervised Learning

All 9 sectionsFlashcardsQuiz