How many principal components do you keep, and what does 95% variance actually guarantee?

medium

Answer

$Common choices are a cumulative variance threshold, the knee in the scree plot, or the number that maximizes downstream performance, and the last is the only one tied to your goal.$
$What a 95% variance threshold guarantees is only that reconstruction error in the least-squares sense is small; it says nothing about whether the discarded 5% contained the signal you care about.$
$This matters because variance is not relevance: a low-variance direction can carry the entire class distinction, which is exactly why PCA is unsupervised and can discard the label-bearing axis.$
$If your aim is prediction, treat the component count as a hyperparameter tuned inside cross-validation, and remember to fit PCA on the training fold only.$

Check yourself — multiple choice

95% variance guarantees no information loss
It only bounds reconstruction error; variance is not relevance, so a discarded low-variance direction can hold the signal — tune the count against the downstream task inside CV
Always keep two components
PCA never loses information

Explained variance is a reconstruction criterion, unrelated to predictive relevance.

#dimensionality-reduction#pca

214 interview questions in this topic.