Why does OLS have a closed-form solution but logistic regression doesn't?
mediumAnswer
- OLS minimizes ||y - Xw||^2 which is quadratic in w — setting the gradient to zero yields the normal equations y, a closed form.
- Logistic regression minimizes the log-loss which is convex but not quadratic (log of a sigmoid); its gradient has no closed-form root, so we solve it iteratively (IRLS, Newton, or SGD).
- Both are convex, so the iterative solution reaches the unique optimum.
Check yourself — multiple choice
- Because logistic regression is non-convex
- OLS is quadratic in the weights so its optimum is closed-form; log-loss is convex but not quadratic
- Logistic regression has no unique optimum
- Logistic regression doesn't use gradients
OLS's quadratic form yields normal equations; log-loss needs iterative solvers.
#linear-regression#linear-models#optimization
Practise Supervised Learning
215 interview questions in this topic.
Related questions
- What is a Generalized Linear Model and when is it the right tool?
- L1 vs L2 regularization — what's the difference?
- Why is logistic regression called a linear model if it uses a sigmoid?
- What is Empirical Risk Minimization?
- Why do we care whether the loss landscape is convex?
- What are the classical assumptions of linear regression?