When can aggressive feature selection actually hurt performance?
mediumAnswer
- (1) Modern regularized models (L2, Lasso, elastic net) and modern gradient boosting handle irrelevant features via regularization — you rarely gain by manual selection.
- (2) Correlated informative features: dropping one from a redundant pair can lose subtle signal.
- (3) Interactions: a feature useless alone can be informative in combination — filter methods miss this.
- (4) Distribution shift: features useless on training may become useful on new data; keep some slack.
- Preference: gentle regularization > aggressive selection.
Check yourself — multiple choice
- Feature selection always helps
- Modern regularized models are already robust; aggressive selection can drop interactions and shift-useful features
- Only for regression tasks
- Only when using neural nets
Regularization already handles irrelevant features; aggressive selection can drop interactions.
#feature-selection
Practise Supervised Learning
215 interview questions in this topic.
Related questions
- Why does L1 (Lasso) produce sparse coefficients but L2 (Ridge) does not?
- When would you use Group Lasso instead of standard Lasso?
- How is feature importance computed from a decision tree or random forest?
- What is Weight of Evidence (WoE) encoding?
- Filter, wrapper, and embedded feature selection — how do they differ?
- How does Recursive Feature Elimination (RFE) work?