Derive PCA — what does it optimize?
hardAnswer
- Given centered X, find orthonormal w that maximizes ' Σ w subject to ||w||=1.
- Solution: eigenvector of Σ = X'X/n corresponding to the largest eigenvalue.
- Subsequent components: next largest eigenvalue, orthogonal.
- Equivalently: minimizes squared reconstruction error over rank-k subspaces (Eckart-Young).
- Two views, same answer.
Check yourself — multiple choice
- Random
- Max with ||w||=1 → top eigenvector of Σ; equivalently minimizes rank-k reconstruction error (Eckart-Young)
- Same as clustering
- Not derivable
PCA: max variance ≡ min reconstruction error; top eigenvectors of Σ.
#dimensionality-reduction#pca
Practise Unsupervised Learning
214 interview questions in this topic.