EasyDeepLearn
Supervised Learning · section 6 of 18

Gradient boosting

12 interview questions on gradient boosting, each answered in full. Free to read, no account needed.

In one sentence, what is Gradient Boosting?

medium
  • Fit a sequence of weak learners (typically shallow trees), where each new learner is trained to predict the negative gradient of the loss w.r.t. the current ensemble's prediction — additive stagewise optimization in function space.
  • Final prediction is the sum of the learners' outputs (each scaled by a learning rate).
  • Works for any differentiable loss: squared error for regression, log-loss for classification, quantile loss, ranking losses, etc.
#gradient-boosting#boosting#ensemblesPermalink & quiz →

What is the role of the learning rate (shrinkage) in GBM?

easy
  • Each new tree's contribution is scaled by eta in (0, 1] before being added: Fm+1  =  Fm  +  η  ⋅  hmF_{m + 1}\; = \;F_{m}\; + \;\eta\; \cdot \;h_{m}.
  • Smaller eta needs more trees (nestimators)(n_{\mathrm{estimators}}) but generalizes better — analogous to a slow, cautious descent that lets many complementary trees contribute rather than one big correction.
  • Rule of thumb: η  =  0.05−0.1\eta\; = \;0.05 - 0.1 with a few hundred to a few thousand trees, using early stopping on validation.
  • Increasing eta trades quality for training time.
#gradient-boosting#boosting#hyperparameter-tuningPermalink & quiz →

What did XGBoost bring on top of vanilla GBM?

hard
  • (1) Regularization terms on tree structure: gamma * T (penalty per leaf) and lambda * sum wj2w_{j}^{2} (L2 on leaf scores), reducing overfitting.
  • (2) Second-order Taylor expansion of the loss for split gain (Newton-like), giving better splits.
  • (3) A weighted quantile sketch for approximate split finding on huge data.
  • (4) Cache-aware, parallelized histogram building.
  • (5) Native handling of missing values via learned default directions.
  • (6) Support for custom objectives and monotonic constraints.
#xgboost#gradient-boosting#boostingPermalink & quiz →

What are LightGBM's key innovations vs XGBoost?

hard
  • (1) Leaf-wise (best-first) tree growth instead of level-wise — grows the leaf with the largest loss reduction, giving deeper asymmetric trees and often lower loss for the same tree count.
  • (2) GOSS (Gradient-based One-Side Sampling): keep all large-gradient samples, subsample small-gradient ones — trains faster with little accuracy loss.
  • (3) EFB (Exclusive Feature Bundling): bundle mutually exclusive sparse features into one, cutting feature count in high-dim sparse data.
  • (4) Native categorical handling via optimal partitioning.
#lightgbm#gradient-boosting#boostingPermalink & quiz →

What makes CatBoost different from XGBoost / LightGBM?

hard
  • (1) Ordered boosting: uses a permutation of the data so that for each example, the model used to estimate its gradient is fit on other examples only — reduces prediction shift / target leakage that plagues target encoding in vanilla GBM.
  • (2) Native categorical support via ordered target statistics — handles high-cardinality categoricals without one-hot.
  • (3) Symmetric (oblivious) trees — same split at every node of a given depth, giving fast inference and implicit regularization.
  • (4) Good defaults — often the strongest 'out-of-the-box' GBDT.
#catboost#gradient-boosting#boosting#encodingPermalink & quiz →

Which XGBoost hyperparameters have the biggest impact, and in what tuning order?

medium
  • First: fix nestimatorsn_{\mathrm{estimators}} large (eg,  2000)  +  earlystoppingrounds(eg, \;2000)\; + \;\mathrm{early}_{\mathrm{stopping}}\mathrm{rounds} so the count auto-selects.
  • Then tune (roughly in this order): (1) learningrate\mathrm{learning}_{\mathrm{rate}} (0.01-0.1); (2) max⁡depth\operatorname{max}_{\mathrm{depth}} (3-10) or numleaves\mathrm{num}_{\mathrm{leaves}}; (3) min⁡childweight\operatorname{min}_{\mathrm{child}}\mathrm{weight} (regularization on leaves); (4) subsample and colsamplebytree\mathrm{colsample}_{\mathrm{bytree}} (0.5-1); (5) γ  /  min⁡splitloss\gamma\; / \;\operatorname{min}_{\mathrm{split}}\mathrm{loss} (0-5); (6) regα\mathrm{reg}_{\alpha} (L1) and regλ\mathrm{reg}_{\lambda} (L2).
  • Do random or Bayesian search — grid search wastes budget in this many dims.
#xgboost#gradient-boosting#hyperparameter-tuningPermalink & quiz →

How do you handle class imbalance in XGBoost / LightGBM?

medium
  • Set scaleposweight  =  negatives  /  positives\mathrm{scale}_{\mathrm{pos}}\mathrm{weight}\; = \;\mathrm{negatives}\; / \;\mathrm{positives} (for binary) so gradients from the minority class are up-weighted.
  • Alternatively use isunbalance=true\mathrm{is}_{\mathrm{unbalance}} = \mathrm{true} in LightGBM.
  • This adjusts the loss weighting without changing the data.
  • Combine with a probability-based threshold tuned on validation for the actual decision.
  • If you also need calibrated probabilities, apply post-hoc calibration since scaleposweight\mathrm{scale}_{\mathrm{pos}}\mathrm{weight} distorts them.
#xgboost#lightgbm#imbalancePermalink & quiz →

How do XGBoost, LightGBM, and CatBoost each handle categorical features?

medium
  • XGBoost: needs one-hot or ordinal encoding for older versions; native categorical support from 1.5+ using ordered partitioning.
  • LightGBM: native support via Fisher-style optimal categorical partitioning; expects category indices as int type or the special 'category' dtype.
  • CatBoost: native, using ordered target statistics — no encoding needed and it typically handles high-cardinality categoricals better than the alternatives.
  • When in doubt with high-cardinality categoricals: try CatBoost first.
#xgboost#lightgbm#catboost#encodingPermalink & quiz →

How does gradient boosting handle missing values without imputation?

medium
  • For each split, the algorithm considers routing missing values to the left or the right child and picks the direction that yields the largest gain on the training data.
  • This 'learned default direction' is stored per split and used at prediction time.
  • Effect: NaNs are treated as first-class citizens; the tree can even learn that 'missingness' itself is informative.
  • XGBoost, LightGBM and CatBoost all implement this — you can pass NaNs directly.
#gradient-boosting#missing-dataPermalink & quiz →

What are monotonic constraints in XGBoost / LightGBM and when do you use them?

hard
  • You can force the prediction to be non-decreasing (or non-increasing) in a specified feature.
  • Useful for regulatory / business constraints: 'credit score shouldn't decrease when income goes up', 'insurance premium shouldn't drop when age increases'.
  • The algorithm restricts split choices during growth to preserve monotonicity.
  • Adds slight bias for a big interpretability + trust win.
  • Configured via monotoneconstraints=[+1,  0,  −1,  ]\mathrm{monotone}_{\mathrm{constraints}} = [ + 1, \;0, \; - 1, \;] in each library.
#xgboost#lightgbm#gradient-boosting#interpretabilityPermalink & quiz →

Rule of thumb: when do you pick XGBoost vs LightGBM vs CatBoost?

medium
  • Roughly: LightGBM is fastest and often top on large, mostly numeric datasets with many features; leaf-wise growth needs careful max⁡depth  /  numleaves\operatorname{max}_{\mathrm{depth}}\; / \;\mathrm{num}_{\mathrm{leaves}} control to avoid overfitting.
  • CatBoost is the safest default when you have many high-cardinality categoricals or you want strong out-of-the-box performance with less tuning.
  • XGBoost is a great generalist — the most feature-complete API, best community support, and slightly slower than LightGBM but often a hair more accurate.
  • In competitions, all three are stacked.
#xgboost#lightgbm#catboost#gradient-boostingPermalink & quiz →

Is adding more features always safe with gradient boosting?

medium
  • No. Boosting tolerates irrelevant features better than linear models, but there are real costs.
  • Each extra feature is another thing that can break, drift or be unavailable at serving time, so the maintenance burden grows.
  • Many weak, correlated features dilute the split search and can slow convergence.
  • Features derived from the target or from post-event data introduce leakage that a validation split may not catch.
  • And a wide model is harder to explain to a stakeholder.
  • In practice, prune to the features that earn their keep by permutation importance and by whether you trust their pipeline.
#feature-selection#gradient-boostingPermalink & quiz →

Practise Supervised Learning

All 18 sectionsFlashcardsQuiz