What is Empirical Risk Minimization?
mediumAnswer
- ERM is the standard learning principle: pick the hypothesis in your class that minimizes the average loss on the training sample, as a proxy for the true (unknown) expected loss.
- It works when the class is not too flexible relative to the sample size — otherwise you overfit.
- Structural risk minimization adds a complexity penalty (regularization) to trade off training loss and capacity.
Check yourself — multiple choice
- ERM minimizes test loss directly
- ERM minimizes training loss as a proxy for expected loss
- ERM removes the need for regularization
- ERM requires access to the true data distribution
ERM = minimize sample average loss; SRM adds a complexity penalty to control overfitting.
#theory#optimization
Practise Supervised Learning
215 interview questions in this topic.
Related questions
- Why do we care whether the loss landscape is convex?
- Generative vs discriminative classifier — what's the difference?
- What is the curse of dimensionality?
- Parametric vs non-parametric models — what's the difference?
- Why do we prefer simpler models when performance is equal?
- What does the no free lunch theorem say for ML?