What is 'perfect separation' in logistic regression and how do you fix it?
hardAnswer
- Perfect separation happens when a feature (or combination) fully separates the classes: the MLE for the corresponding weight goes to +/- infinity and the solver never converges (or gives massive coefficients with huge standard errors).
- Fixes: (1) add L2 regularization (drives the weight to a finite value); (2) use Firth's penalized likelihood; (3) drop or bin the separating feature.
- Common on tiny or heavily imbalanced datasets and one-hot rare categories.
Check yourself — multiple choice
- It means logistic regression fits perfectly on the test set
- A feature completely separates the classes and the MLE diverges — fix with L2 or Firth
- It only affects neural networks
- It's a numerical bug in scikit-learn
MLE diverges under separation; L2 penalty (or Firth) yields a finite, well-behaved fit.
#logistic-regression#regularization
Practise Supervised Learning
215 interview questions in this topic.
Related questions
- L1 vs L2 in logistic regression — practical differences.
- How do you detect and fix overfitting?
- L1 vs L2 regularization — what's the difference?
- What is the generalization gap and how do you shrink it?
- How do you handle strong multicollinearity in a linear model?
- What's the main risk of polynomial regression?