Why not just fit a linear regression for a binary label (linear probability model)?
mediumAnswer
- You can — sometimes it even works reasonably as a quick baseline — but three problems bite in practice: (1) predicted probabilities aren't bounded to [0, 1]; (2) errors are heteroscedastic (variance = p(1-p) depends on x), so OLS standard errors are wrong; (3) it isn't the MLE for Bernoulli data.
- Logistic regression fixes all three via the sigmoid link and the Bernoulli likelihood — that's why it's the default.
Check yourself — multiple choice
- The linear probability model is the MLE for Bernoulli data
- Predictions leak outside [0, 1], errors are heteroscedastic, and it's not the Bernoulli MLE
- It only fails on multi-class
- It's always more accurate than logistic regression
LPM: unbounded probs + heteroscedastic errors + not MLE ⇒ prefer logistic regression.
#logistic-regression#linear-regression#theory
Practise Supervised Learning
215 interview questions in this topic.
Related questions
- What is a Generalized Linear Model and when is it the right tool?
- What's the difference between probability, odds, and log-odds?
- Generative vs discriminative classifier — what's the difference?
- What is the curse of dimensionality?
- Parametric vs non-parametric models — what's the difference?
- Why do we prefer simpler models when performance is equal?