EasyDeepLearn
Supervised Learning · section 13 of 18

Feature selection

6 interview questions on feature selection, each answered in full. Free to read, no account needed.

How does Recursive Feature Elimination (RFE) work?

medium
  • Train the model; rank features by importance (coefficients  for  linear,  featureimportances  for  trees)(\mathrm{coefficients}\;\mathrm{for}\;\mathrm{linear}, \;\mathrm{feature}_{\mathrm{importances}}\;\mathrm{for}\;\mathrm{trees}); remove the least important K features; retrain; repeat until you reach the target count or the score peaks.
  • RFECV wraps this in cross-validation to automatically pick the optimal number of features.
  • Best on medium feature counts (< 1000) where retraining is cheap.
  • Can miss useful features that only matter in combination — trades subtle interactions for interpretability.
#feature-selectionPermalink & quiz →

How does permutation importance work and when is it better than impurity-based importance?

medium
  • Shuffle a feature's values on a held-out set, keeping the target intact; measure the drop in the model's metric.
  • Larger drop → more important.
  • Model-agnostic, works on any predictor.
  • Advantages over impurity: unbiased with respect to feature cardinality (impurity favors continuous / high-cardinality), reflects the actual predictive contribution on unseen data, and works for any metric (including business KPIs).
  • Downsides: doesn't handle correlated features well — permuting one just moves the importance to its correlated twin.
#feature-selection#interpretabilityPermalink & quiz →

Can you use SHAP values for feature selection?

medium
  • Yes — mean absolute SHAP value across a validation sample gives a robust, model-consistent importance ranking.
  • Advantages over impurity: unbiased for cardinality, respects the model's actual behavior, works per-example.
  • Practical selection: sort by mean  ∣SHAP\mathrm{mean}\; \mid \mathrm{SHAP}|, keep features that together explain a target fraction (e.g., 95%) of total absolute SHAP, or drop features below a floor threshold.
  • Downside: computing SHAP is more expensive than impurity, and highly correlated features share importance in ways that can be misleading.
#feature-selection#interpretabilityPermalink & quiz →

When can aggressive feature selection actually hurt performance?

medium
  • (1) Modern regularized models (L2, Lasso, elastic net) and modern gradient boosting handle irrelevant features via regularization — you rarely gain by manual selection.
  • (2) Correlated informative features: dropping one from a redundant pair can lose subtle signal.
  • (3) Interactions: a feature useless alone can be informative in combination — filter methods miss this.
  • (4) Distribution shift: features useless on training may become useful on new data; keep some slack.
  • Preference: gentle regularization > aggressive selection.
#feature-selectionPermalink & quiz →

What does VarianceThreshold do and when is it useful?

easy
  • Drops features with variance below a threshold — most commonly zero-variance (constants) that carry no information.
  • Useful as a cheap pre-filter to remove degenerate features before training or before applying more expensive selection methods.
  • In high-dimensional sparse data (bag-of-words with rare terms), a low-variance filter (e.g., threshold = 0.01) can slash the feature count dramatically with almost no signal loss.
  • Doesn't consider the target — pair with a supervised method for real feature selection.
#feature-selectionPermalink & quiz →

Mutual information vs chi-square vs ANOVA F for feature selection — how do you pick?

medium
  • Mutual information: measures general (non-linear, non-monotonic) dependency between feature and target — the most powerful filter, works for both categorical and continuous.
  • Chi-square: for categorical features + categorical target; tests independence in a contingency table.
  • ANOVA F: for continuous features + categorical target; tests whether class means differ.
  • Use MI as a default; χ2\chi^{2} for categorical-categorical; ANOVA F for numeric-categorical when you assume linear-ish relations and want speed.
#feature-selectionPermalink & quiz →

Practise Supervised Learning

All 18 sectionsFlashcardsQuiz