What's the Bayesian interpretation of Ridge regression?
hardAnswer
- Ridge regression is the maximum a posteriori (MAP) estimate under a Gaussian prior on the weights: w ~ .
- Equivalently, adding an L2 penalty on w is equivalent to assuming the coefficients are a priori small.
- Lasso corresponds to a Laplace (double-exponential) prior on w, which has more probability mass at zero and heavier tails, explaining its sparsity behavior.
Check yourself — multiple choice
- Ridge assumes a Laplace prior on the weights
- Ridge is MAP under a Gaussian prior; Lasso is MAP under a Laplace prior
- Ridge is Bayesian only if
- Ridge does not have a Bayesian interpretation
L2 ≡ Gaussian prior (MAP); L1 ≡ Laplace prior — hence sparsity for Lasso.
#regularization#linear-models#theory
Practise Supervised Learning
215 interview questions in this topic.
Related questions
- L1 vs L2 regularization — what's the difference?
- What is the generalization gap and how do you shrink it?
- Why does Ridge regression give more stable coefficients than OLS?
- Why does L1 (Lasso) produce sparse coefficients but L2 (Ridge) does not?
- When is Elastic Net better than pure Lasso or Ridge?
- How do you choose the regularization strength (lambda / alpha)?