EasyDeepLearn
Supervised Learning · section 8 of 18

Ensembling & stacking

3 interview questions on ensembling & stacking, each answered in full. Free to read, no account needed.

How does stacking work and when does it help?

hard
  • Level 0: train several diverse base models (e.g., logistic regression, random forest, XGBoost, KNN) using K-fold CV, producing out-of-fold predictions for each.
  • Level 1: train a meta-learner (usually a simple regularized linear model — Ridge / logistic regression) on the level-0 predictions to combine them.
  • Predict at inference: base models on the raw features → their predictions → meta-learner → final answer.
  • Helps most when the base models make *different* mistakes — the meta-learner exploits the diversity.
  • Small gain on well-tuned single models but reliable in competitions.
#ensembles#stackingPermalink & quiz →

Soft voting vs hard voting — which is usually better and why?

easy
  • Hard voting: each model outputs a predicted class; the ensemble takes the majority.
  • Simple but throws away probability information.
  • Soft voting: average the predicted class probabilities across models; predict the argmax.
  • Almost always better because it captures confidence — a hesitant model contributes less than a confident one.
  • Requires all models to output calibrated probabilities (calibrate first if not — RF, SVM often aren't out of the box).
  • If probabilities aren't available (some SVMs), fall back to hard voting.
#ensemblesPermalink & quiz →

Why is diversity between base models more important than their individual accuracy in an ensemble?

medium
  • If two 95%-accurate models make exactly the same mistakes, averaging them still gives 95% — no improvement.
  • If two 90%-accurate models make *different* mistakes, averaging can push accuracy well above either.
  • Ensembles reduce variance from errors that don't correlate across models.
  • Diversify by using: different model families (linear + tree + KNN), different feature subsets, different random seeds, or different subsamples.
  • Bagging and random forests are engineered around this idea.
  • In competitions, blending diverse models is often more valuable than optimizing a single one.
#ensemblesPermalink & quiz →

Practise Supervised Learning

All 18 sectionsFlashcardsQuiz