Why do we use cross-entropy (log loss) instead of MSE for classification?
mediumAnswer
- Cross-entropy matches the maximum-likelihood estimator for a Bernoulli / categorical target and has a well-shaped gradient when combined with sigmoid/softmax outputs — large errors give large gradients so the model learns fast.
- MSE on top of sigmoid gives a very small gradient when predictions are confidently wrong (the sigmoid saturates), so training is slow and can get stuck.
- Cross-entropy is also convex in the logits for logistic regression, MSE-with-sigmoid is not.
Check yourself — multiple choice
- MSE always converges faster than cross-entropy
- Cross-entropy has stronger gradients when predictions are wrong and matches Bernoulli MLE
- MSE cannot be used for regression
- Cross-entropy is only for multi-class
Cross-entropy = Bernoulli MLE and gives strong gradients through sigmoid/softmax.
#logistic-regression#classification#optimization
Practise Supervised Learning
215 interview questions in this topic.
Related questions
- One-vs-Rest, One-vs-One, and softmax — how do you choose for multi-class?
- Write out the softmax function and its main property.
- How is logistic regression fit in practice?
- Your scikit-learn logistic regression warns 'lbfgs failed to converge'. What do you do?
- 'multinomial' vs 'ovr' setting in scikit-learn's LogisticRegression — what changes?
- When would LDA outperform logistic regression?