Why does Ridge regression give more stable coefficients than OLS?
mediumAnswer
- Ridge solves y instead of X y.
- Adding lambda I to the diagonal keeps the matrix invertible even when features are collinear or when p > n, and shrinks coefficients toward zero proportionally to their scale.
- The shrinkage introduces bias but drastically reduces variance — often a favourable tradeoff for prediction.
- Ridge does not zero-out coefficients, it just makes them small and stable.
Check yourself — multiple choice
- Ridge makes X singular
- It adds lambda·I to X, stabilizing the inverse and shrinking coefficients
- Ridge produces sparse coefficients
- Ridge is only used for classification
Ridge stabilizes the inverse via +λI and shrinks coefficients smoothly.
#regularization#linear-models
Practise Supervised Learning
215 interview questions in this topic.
Related questions
- L1 vs L2 regularization — what's the difference?
- Why does L1 (Lasso) produce sparse coefficients but L2 (Ridge) does not?
- When is Elastic Net better than pure Lasso or Ridge?
- How do you choose the regularization strength (lambda / alpha)?
- In one sentence, what does the LARS algorithm compute?
- What's the Bayesian interpretation of Ridge regression?