What does the no free lunch theorem say for ML?
mediumAnswer
- Averaged over all possible problems, no learning algorithm is better than random guessing.
- Any advantage of an algorithm on one class of problems is offset by worse performance elsewhere.
- Practically, this means model choice must match the structure of your data.
- Trees dominate tabular data; CNNs dominate images; transformers dominate sequences — because their inductive biases match the domain, not because they are universally better.
Check yourself — multiple choice
- There is one model that beats all others on every dataset
- No algorithm dominates on all problems; inductive biases must match the data
- Deep learning always wins
- Tree ensembles are theoretically optimal
No universal best learner exists; matching inductive bias to data is what matters.
#theory#fundamentals
Practise Supervised Learning
215 interview questions in this topic.
Related questions
- Parametric vs non-parametric models — what's the difference?
- Why do we prefer simpler models when performance is equal?
- What is inductive bias and why does every model have one?
- What is the bias-variance tradeoff?
- Generative vs discriminative classifier — what's the difference?
- What is the curse of dimensionality?