How does stacking work and when does it help?
hardAnswer
- Level 0: train several diverse base models (e.g., logistic regression, random forest, XGBoost, KNN) using K-fold CV, producing out-of-fold predictions for each.
- Level 1: train a meta-learner (usually a simple regularized linear model — Ridge / logistic regression) on the level-0 predictions to combine them.
- Predict at inference: base models on the raw features → their predictions → meta-learner → final answer.
- Helps most when the base models make *different* mistakes — the meta-learner exploits the diversity.
- Small gain on well-tuned single models but reliable in competitions.
Check yourself — multiple choice
- Stacking averages predictions with equal weight
- Base models trained via K-fold produce OOF predictions; a meta-learner combines them
- Same as random forest
- Only works for regression
Stacking: OOF base predictions + meta-learner ⇒ exploits diversity of base models.
#ensembles#stacking
Practise Supervised Learning
215 interview questions in this topic.
Related questions
- Random forest vs gradient boosting — which do you pick and why?
- How does bagging reduce variance?
- Why does Random Forest sample features at each split, not just once per tree?
- How do Extra Trees differ from Random Forests?
- In one sentence, what is Gradient Boosting?
- Soft voting vs hard voting — which is usually better and why?