The 'naive' independence assumption is almost always false. Why does Naive Bayes still work?
mediumAnswer
- Because classification only requires that the argmax over classes be correct — not that the posterior probabilities themselves be accurate.
- Even when the independence assumption is violated, the decision boundary from NB can be close to optimal on many problems.
- Its probabilities, however, are usually badly miscalibrated (too extreme, close to 0 or 1).
- If you need calibrated probabilities, apply Platt or isotonic calibration on top.
Check yourself — multiple choice
- Because features really are always independent
- Because argmax over classes can be correct even when probabilities are miscalibrated
- Because it uses cross-entropy loss
- Because it always adds Laplace smoothing
NB predictions can be right (correct argmax) even when its probabilities are miscalibrated.
#naive-bayes#theory#calibration
Practise Supervised Learning
215 interview questions in this topic.
Related questions
- How does Bayes' theorem drive Naive Bayes classification?
- You trained Naive Bayes on balanced data but deploy on data where positives are 1%. What happens and what do you do?
- Generative vs discriminative classifier — what's the difference?
- What is the curse of dimensionality?
- Parametric vs non-parametric models — what's the difference?
- Why do we prefer simpler models when performance is equal?