EasyDeepLearn
Supervised Learning · section 5 of 18

Decision trees & random forests

13 interview questions on decision trees & random forests, each answered in full. Free to read, no account needed.

Random forest vs gradient boosting — which do you pick and why?

medium
  • Random forest averages many deep trees trained on bootstrapped samples with random feature subsets — it reduces variance and is robust with little tuning.
  • Gradient boosting (XGBoost, LightGBM, CatBoost) fits trees sequentially, each correcting residuals — it usually gives better accuracy on tabular data but needs more tuning and is more prone to overfitting.
  • For a strong baseline on tabular data: GBDT wins.
  • For a fast, safe baseline: random forest.
#trees#ensemblesPermalink & quiz →

Gini impurity, entropy, and classification error — which do trees actually use?

medium
  • For classification, trees choose the split that most reduces impurity.
  • Gini = 1 - sum pc2p_{c}^{2} and entropy = -sum pcp_{c} log pcp_{c} are almost interchangeable in practice, both being smooth and differentiable proxies for classification error.
  • Gini is slightly faster (no log) and is CART's default.
  • Classification error is not smooth enough — it's rarely used to grow trees, only to prune them.
  • For regression, the criterion is variance reduction (MSE).
#decision-trees#treesPermalink & quiz →

How do decision trees handle categorical features?

medium
  • Native support depends on the library.
  • Classic CART considers only binary splits: partition category set A vs the rest, or the exhaustive best partition (2(k−1)−1  possibilities,  expensive  for  high  cardinality)(2(k - 1) - 1\;\mathrm{possibilities}, \;\mathrm{expensive}\;\mathrm{for}\;\mathrm{high}\;\mathrm{cardinality}). scikit-learn does *not* support categoricals natively — you must one-hot encode.
  • LightGBM has native categorical splits via Fisher's optimal partitioning.
  • CatBoost uses ordered target encoding.
  • XGBoost added native categorical support in 1.5+.
  • Native handling is usually stronger and faster than one-hot for high-cardinality categoricals.
#decision-trees#trees#features#encodingPermalink & quiz →

What are the most impactful hyperparameters of a decision tree?

easy
  • max⁡depth\operatorname{max}_{\mathrm{depth}}: hard limit on depth — the biggest lever. min⁡samplessplit\operatorname{min}_{\mathrm{samples}}\mathrm{split}: minimum samples needed to consider splitting a node. min⁡samplesleaf\operatorname{min}_{\mathrm{samples}}\mathrm{leaf}: minimum in each leaf — controls leaf size and variance. max⁡features\operatorname{max}_{\mathrm{features}}: number of features to consider per split (=all for a tree, sqrt(d) or similar for a forest). min⁡impuritydecrease\operatorname{min}_{\mathrm{impurity}}\mathrm{decrease}: don't split unless impurity drops by at least this much.
  • Together they control the bias-variance tradeoff and inference latency.
#decision-trees#hyperparameter-tuningPermalink & quiz →

How do decision trees handle missing values?

medium
  • Options: (1) surrogate splits (CART's approach): find a backup feature that best mimics the chosen split's partition — used when the primary feature is missing at prediction time.
  • (2) Default direction (XGBoost, LightGBM): during training, learn per split which direction — left or right — missing values should go.
  • (3) Sentinel value: encode missingness as a specific value (e.g., -9999) and let the tree branch on it — works but hides information.
  • (4) Impute first then train.
#decision-trees#missing-data#treesPermalink & quiz →

What criterion does a regression tree use to choose splits?

easy
  • Variance reduction: pick the split that most reduces sum of squared errors of the target within child nodes.
  • Equivalently, maximize the total variance of the parent minus the weighted sum of children variances.
  • The prediction in each leaf is the mean of the target for training points that fall into it (median for MAE-loss trees).
  • Same tree machinery, different impurity function.
#decision-trees#trees#metrics-regressionPermalink & quiz →

How is feature importance computed from a decision tree or random forest?

medium
  • Impurity-based (MDI): sum, over all splits using that feature, of the impurity decrease weighted by the number of samples reaching the split.
  • Cheap and comes free from training.
  • Problem: biased toward high-cardinality and continuous features.
  • Permutation importance: shuffle a feature's values on a held-out set and measure the drop in metric — unbiased, model-agnostic, more expensive.
  • SHAP values: consistent, additive, per-example — the gold standard when you can afford them.
#decision-trees#trees#interpretability#feature-selectionPermalink & quiz →

How does bagging reduce variance?

medium
  • Bagging (bootstrap aggregating) trains B models on B bootstrap samples of the training set and averages predictions.
  • If the models were independent with variance σ2\sigma^{2}, the average has variance σ2  /  B\sigma^{2}\; / \;B.
  • In practice, trees trained on bootstrapped data are positively correlated (they share most of the data), so the variance reduction is bounded by the correlation rho: Var ≈ ρ⋅σ2  +  (1−ρ)⋅σ2/B\rho \cdot \sigma^{2}\; + \;(1 - \rho) \cdot \sigma^{2} / B.
  • Random Forest reduces rho by also sampling features per split.
#bagging#ensembles#random-forestPermalink & quiz →

Why does Random Forest sample features at each split, not just once per tree?

medium
  • Per-split feature subsampling forces different trees to explore different features and different splits.
  • Without it, if one feature is very predictive, every tree would keep splitting on it near the root and become highly correlated — averaging correlated trees barely reduces variance.
  • Sampling features at each split de-correlates the ensemble.
  • Default max⁡features  =  sqrt(d)\operatorname{max}_{\mathrm{features}}\; = \;\mathrm{sqrt}(d) for classification, d/3 for regression is a well-tested rule of thumb.
#random-forest#bagging#ensemblesPermalink & quiz →

How do Extra Trees differ from Random Forests?

medium
  • Two changes: (1) each tree is trained on the whole training set (no bootstrapping); (2) splits are chosen randomly — for each candidate feature, a random split threshold is drawn rather than the optimal one, and the best of those random splits is picked.
  • Result: more randomness, more bias, less variance, and *faster* training.
  • Typically slightly worse than RF on well-tuned problems but sometimes better on very noisy data and always faster.
#random-forest#bagging#ensemblesPermalink & quiz →

Which Random Forest hyperparameters actually matter for tuning?

medium
  • nestimatorsn_{\mathrm{estimators}}: 'more is better' until returns diminish — 200-500 is a solid default. max⁡features\operatorname{max}_{\mathrm{features}}: the main lever for correlation between trees; try sqrt(d), log2(d), or 0.3-0.5*d. max⁡depth  /  min⁡samplesleaf\operatorname{max}_{\mathrm{depth}}\; / \;\operatorname{min}_{\mathrm{samples}}\mathrm{leaf}: control tree size; deeper trees favor bias-reduction, larger min⁡samplesleaf\operatorname{min}_{\mathrm{samples}}\mathrm{leaf} reduces variance. classweight\mathrm{class}_{\mathrm{weight}}='balanced' for imbalanced data. njobsn_{\mathrm{jobs}}=-1 for speed.
  • Skip fiddling with min⁡samplessplit\operatorname{min}_{\mathrm{samples}}\mathrm{split} — min⁡samplesleaf\operatorname{min}_{\mathrm{samples}}\mathrm{leaf} is enough.
#random-forest#hyperparameter-tuningPermalink & quiz →

Are Random Forest probability estimates well-calibrated?

medium
  • Not always.
  • RF probabilities come from averaging tree votes and tend to be pushed away from 0 and 1 (over-cautious).
  • A single deep tree gives 0/1 predictions; averaging many gives probabilities that live mostly in a compressed middle range.
  • For calibrated probabilities, apply isotonic or Platt calibration on a held-out set (sklearn's CalibratedClassifierCV).
  • GBDTs are usually better-calibrated out of the box.
#random-forest#calibrationPermalink & quiz →

What is Boruta and when do you reach for it?

hard
  • A wrapper feature selection algorithm that compares each feature's importance to that of 'shadow' features — random permutations of that feature.
  • A feature is confirmed if its importance is significantly higher than the max shadow importance across multiple random forest fits (statistical test with Bonferroni correction).
  • Slow (many RF fits) but very rigorous — often used in biology / genomics where reliability matters more than speed.
  • Use when you need statistical confidence that selected features are truly informative.
#feature-selection#random-forestPermalink & quiz →

Practise Supervised Learning

All 18 sectionsFlashcardsQuiz