Why is a single decision tree considered a 'high-variance' model?
mediumAnswer
- A small change in the training data can produce a completely different tree: the first split cascades, so if a different feature is chosen at the top, everything below changes.
- This makes individual trees unstable — predictions vary a lot with resampling.
- Bagging (Random Forest) and boosting exploit this: averaging many independent (or corrected-residual) trees reduces variance dramatically without hurting bias.
Check yourself — multiple choice
- Trees have low variance — that's why we use them
- Small data changes flip the top splits and cascade through the whole tree
- Trees always have zero training error
- Trees are stable but slow
Top-split cascades ⇒ high variance ⇒ perfect candidate for bagging / boosting.
#decision-trees#trees#theory
Practise Supervised Learning
215 interview questions in this topic.
Related questions
- Why do decision trees (and boosted trees) fail to extrapolate?
- Gini impurity, entropy, and classification error — which do trees actually use?
- How do decision trees handle categorical features?
- Pre-pruning vs post-pruning in decision trees — what's the difference and when do you use each?
- How does cost-complexity (CCP) pruning work?
- How do decision trees handle missing values?