EasyDeepLearn

How does bagging reduce variance?

medium

Answer

  • Bagging (bootstrap aggregating) trains B models on B bootstrap samples of the training set and averages predictions.
  • If the models were independent with variance σ2\sigma^{2}, the average has variance σ2  /  B\sigma^{2}\; / \;B.
  • In practice, trees trained on bootstrapped data are positively correlated (they share most of the data), so the variance reduction is bounded by the correlation rho: Var ≈ ρ⋅σ2  +  (1−ρ)⋅σ2/B\rho \cdot \sigma^{2}\; + \;(1 - \rho) \cdot \sigma^{2} / B.
  • Random Forest reduces rho by also sampling features per split.
Check yourself — multiple choice
  • Bagging reduces bias, not variance
  • Averaging B bootstrapped models reduces variance; correlation limits how much
  • Bagging works only for classification
  • Bagging is the same as boosting

Averaging reduces variance; tree correlation is the ceiling — RF reduces it via feature subsampling.

#bagging#ensembles#random-forest

Practise Supervised Learning

215 interview questions in this topic.

Read the lessonFlashcardsTake the quiz

Related questions