Why do we care whether the loss landscape is convex?
mediumAnswer
- A convex loss (linear regression MSE, logistic regression log loss, SVM hinge, LASSO) has a single global minimum, so any local optimizer finds the best solution and results are reproducible.
- Non-convex losses (neural networks, matrix factorization, deep boosting) have many local minima and saddle points — training is sensitive to initialization, learning rate, and stochasticity.
- Convex problems admit strong theory (convergence, duality); non-convex ones rely on empirical tricks.
Check yourself — multiple choice
- Convex losses have many local minima
- Convex losses have one global minimum, so optimization is well-behaved
- Non-convex losses always fail to train
- Convexity is only about training speed
Convex ⇒ single global minimum ⇒ optimizer-agnostic. Non-convex ⇒ many local minima, saddle points.
#theory#optimization
Practise Supervised Learning
215 interview questions in this topic.
Related questions
- What is Empirical Risk Minimization?
- Generative vs discriminative classifier — what's the difference?
- What is the curse of dimensionality?
- Parametric vs non-parametric models — what's the difference?
- Why do we prefer simpler models when performance is equal?
- What does the no free lunch theorem say for ML?