How do you choose the regularization strength (lambda / alpha)?
mediumAnswer
- Cross-validation over a log-spaced grid is the standard approach — pick alpha minimizing average validation loss.
- Coefficient path plots (coefficient vs log alpha) help understand which features enter/exit at what strength.
- For efficiency, use algorithms that compute the whole path (LARS, coordinate descent with warm starts): scikit-learn's LassoCV / RidgeCV / ElasticNetCV do this out of the box.
- Standardize features first; alpha's meaningful scale depends on it.
Check yourself — multiple choice
- Pick the alpha that maximizes training accuracy
- Use cross-validation over a log-spaced grid; standardize features first
- Set always
- Choose alpha by looking at on training
CV on a log grid + feature standardization is the standard recipe.
#regularization#hyperparameter-tuning#linear-models
Practise Supervised Learning
215 interview questions in this topic.
Related questions
- L1 vs L2 regularization — what's the difference?
- Why does Ridge regression give more stable coefficients than OLS?
- Why does L1 (Lasso) produce sparse coefficients but L2 (Ridge) does not?
- When is Elastic Net better than pure Lasso or Ridge?
- In one sentence, what does the LARS algorithm compute?
- What's the Bayesian interpretation of Ridge regression?