How is feature importance computed from a decision tree or random forest?
mediumAnswer
- Impurity-based (MDI): sum, over all splits using that feature, of the impurity decrease weighted by the number of samples reaching the split.
- Cheap and comes free from training.
- Problem: biased toward high-cardinality and continuous features.
- Permutation importance: shuffle a feature's values on a held-out set and measure the drop in metric — unbiased, model-agnostic, more expensive.
- SHAP values: consistent, additive, per-example — the gold standard when you can afford them.
Check yourself — multiple choice
- Only from
- Impurity-based (MDI, cheap but biased) or permutation / SHAP (better but slower)
- Only from cross-validation
- By counting features used at leaves
MDI (impurity) is cheap; permutation / SHAP are more reliable.
#decision-trees#trees#interpretability#feature-selection
Practise Supervised Learning
215 interview questions in this topic.
Related questions
- Gini impurity, entropy, and classification error — which do trees actually use?
- How do decision trees handle categorical features?
- Pre-pruning vs post-pruning in decision trees — what's the difference and when do you use each?
- How does cost-complexity (CCP) pruning work?
- How do decision trees handle missing values?
- What criterion does a regression tree use to choose splits?