How is logistic regression fit in practice?
mediumAnswer
- Two main approaches: (1) Iteratively Reweighted Least Squares (IRLS) — an implementation of Newton-Raphson on the log-likelihood.
- Each iteration solves a weighted least squares problem with weights = p(1-p).
- Converges in ~10-20 iterations for well-conditioned problems.
- (2) First-order methods: L-BFGS, SAG, SAGA, SGD.
- Better for large datasets and L1 penalty. scikit-learn uses 'lbfgs' by default; 'liblinear'/'saga' for L1.
Check yourself — multiple choice
- Only closed-form matrix inversion
- IRLS (Newton) or first-order methods (L-BFGS, SAGA)
- By reduction to Naive Bayes
- By solving X y
Iterative: IRLS/Newton for small data, L-BFGS/SAG/SAGA for large data.
#logistic-regression#optimization
Practise Supervised Learning
215 interview questions in this topic.
Related questions
- Why do we use cross-entropy (log loss) instead of MSE for classification?
- Your scikit-learn logistic regression warns 'lbfgs failed to converge'. What do you do?
- What is Empirical Risk Minimization?
- Why do we care whether the loss landscape is convex?
- Why does OLS have a closed-form solution but logistic regression doesn't?
- One-vs-Rest, One-vs-One, and softmax — how do you choose for multi-class?