What is the curse of dimensionality?
mediumAnswer
- As dimensions grow, data becomes sparse: distances between points concentrate, the volume needed to cover the space grows exponentially, and models that rely on locality (k-NN, kernel methods) degrade.
- You need exponentially more data to maintain density.
- Remedies: feature selection, dimensionality reduction (PCA, UMAP), regularization, and models that assume structure (trees, deep nets with inductive biases).
Check yourself — multiple choice
- More dimensions always improve accuracy
- Distances become less informative as dimensionality grows
- It only affects deep learning
- PCA causes the curse
In high dimensions, distances concentrate and locality-based methods degrade.
#theory#features
Practise Supervised Learning
215 interview questions in this topic.
Related questions
- Which models need feature scaling and which don't?
- What is target leakage and how do you prevent it?
- Generative vs discriminative classifier — what's the difference?
- Parametric vs non-parametric models — what's the difference?
- Why do we prefer simpler models when performance is equal?
- What does the no free lunch theorem say for ML?