Gini impurity, entropy, and classification error — which do trees actually use?
mediumAnswer
- For classification, trees choose the split that most reduces impurity.
- Gini = 1 - sum and entropy = -sum log are almost interchangeable in practice, both being smooth and differentiable proxies for classification error.
- Gini is slightly faster (no log) and is CART's default.
- Classification error is not smooth enough — it's rarely used to grow trees, only to prune them.
- For regression, the criterion is variance reduction (MSE).
Check yourself — multiple choice
- Classification error is the standard splitting criterion
- Gini and entropy are the practical choices; classification error is used only for pruning
- Regression trees use Gini
- Entropy is not differentiable
Gini/entropy for growth (smooth); classification error for evaluation/pruning.
#decision-trees#trees
Practise Supervised Learning
215 interview questions in this topic.
Related questions
- How do decision trees handle categorical features?
- Pre-pruning vs post-pruning in decision trees — what's the difference and when do you use each?
- How does cost-complexity (CCP) pruning work?
- How do decision trees handle missing values?
- What criterion does a regression tree use to choose splits?
- Why is a single decision tree considered a 'high-variance' model?