What did XGBoost bring on top of vanilla GBM?
hardAnswer
- (1) Regularization terms on tree structure: gamma * T (penalty per leaf) and lambda * sum (L2 on leaf scores), reducing overfitting.
- (2) Second-order Taylor expansion of the loss for split gain (Newton-like), giving better splits.
- (3) A weighted quantile sketch for approximate split finding on huge data.
- (4) Cache-aware, parallelized histogram building.
- (5) Native handling of missing values via learned default directions.
- (6) Support for custom objectives and monotonic constraints.
Check yourself — multiple choice
- Just parallelization
- Regularized objective, second-order splits, missing-value handling, histogram/sketch, cache-aware code
- Only categorical support
- It replaced trees with linear models
Regularized objective, Newton splits, missing values, histograms, engineering — XGBoost's stack.
#xgboost#gradient-boosting#boosting
Practise Supervised Learning
215 interview questions in this topic.
Related questions
- In one sentence, what is Gradient Boosting?
- What is the role of the learning rate (shrinkage) in GBM?
- What are LightGBM's key innovations vs XGBoost?
- What makes CatBoost different from XGBoost / LightGBM?
- Which XGBoost hyperparameters have the biggest impact, and in what tuning order?
- How does early stopping work in gradient boosting and why is it important?