EasyDeepLearn

Which XGBoost hyperparameters have the biggest impact, and in what tuning order?

medium

Answer

  • First: fix nestimatorsn_{\mathrm{estimators}} large (eg,  2000)  +  earlystoppingrounds(eg, \;2000)\; + \;\mathrm{early}_{\mathrm{stopping}}\mathrm{rounds} so the count auto-selects.
  • Then tune (roughly in this order): (1) learningrate\mathrm{learning}_{\mathrm{rate}} (0.01-0.1); (2) max⁡depth\operatorname{max}_{\mathrm{depth}} (3-10) or numleaves\mathrm{num}_{\mathrm{leaves}}; (3) min⁡childweight\operatorname{min}_{\mathrm{child}}\mathrm{weight} (regularization on leaves); (4) subsample and colsamplebytree\mathrm{colsample}_{\mathrm{bytree}} (0.5-1); (5) γ  /  min⁡splitloss\gamma\; / \;\operatorname{min}_{\mathrm{split}}\mathrm{loss} (0-5); (6) regα\mathrm{reg}_{\alpha} (L1) and regλ\mathrm{reg}_{\lambda} (L2).
  • Do random or Bayesian search — grid search wastes budget in this many dims.
Check yourself — multiple choice
  • Only nestimatorsn_{\mathrm{estimators}}
  • learningrate\mathrm{learning}_{\mathrm{rate}} → max⁡depth\operatorname{max}_{\mathrm{depth}} → min⁡childweight\operatorname{min}_{\mathrm{child}}\mathrm{weight} → subsample/colsample → gamma → regα/λ\mathrm{reg}_{\alpha} / \lambda
  • The activation function
  • The optimizer only

LR + tree size + subsampling + regularization — Bayesian search over them.

#xgboost#gradient-boosting#hyperparameter-tuning

Practise Supervised Learning

215 interview questions in this topic.

Read the lessonFlashcardsTake the quiz

Related questions