How does permutation importance work and when is it better than impurity-based importance?
mediumAnswer
- Shuffle a feature's values on a held-out set, keeping the target intact; measure the drop in the model's metric.
- Larger drop → more important.
- Model-agnostic, works on any predictor.
- Advantages over impurity: unbiased with respect to feature cardinality (impurity favors continuous / high-cardinality), reflects the actual predictive contribution on unseen data, and works for any metric (including business KPIs).
- Downsides: doesn't handle correlated features well — permuting one just moves the importance to its correlated twin.
Check yourself — multiple choice
- Same as impurity-based importance
- Shuffle a feature and measure the metric drop — unbiased, works for any model/metric
- Only works for random forests
- It only measures training accuracy
Permutation importance = metric drop after shuffling that feature — model-agnostic, unbiased for cardinality.
#feature-selection#interpretability
Practise Supervised Learning
215 interview questions in this topic.
Related questions
- How is feature importance computed from a decision tree or random forest?
- Can you use SHAP values for feature selection?
- How do you interpret a coefficient in a multiple linear regression?
- Why does L1 (Lasso) produce sparse coefficients but L2 (Ridge) does not?
- When would you use Group Lasso instead of standard Lasso?
- How do you interpret a logistic regression coefficient as an odds ratio?