What is the role of the learning rate (shrinkage) in GBM?
easyAnswer
- Each new tree's contribution is scaled by eta in (0, 1] before being added: .
- Smaller eta needs more trees but generalizes better — analogous to a slow, cautious descent that lets many complementary trees contribute rather than one big correction.
- Rule of thumb: with a few hundred to a few thousand trees, using early stopping on validation.
- Increasing eta trades quality for training time.
Check yourself — multiple choice
- eta scales the target
- eta scales each tree's contribution — smaller eta + more trees generalizes better
- eta must always be 1
- eta is only used for regression
Shrinkage: smaller eta ⇒ more trees needed ⇒ better generalization.
#gradient-boosting#boosting#hyperparameter-tuning
Practise Supervised Learning
215 interview questions in this topic.
Related questions
- In one sentence, what is Gradient Boosting?
- What did XGBoost bring on top of vanilla GBM?
- What are LightGBM's key innovations vs XGBoost?
- What makes CatBoost different from XGBoost / LightGBM?
- Which XGBoost hyperparameters have the biggest impact, and in what tuning order?
- How does early stopping work in gradient boosting and why is it important?