What is nested cross-validation and when do you need it?
hardAnswer
- You need it whenever you tune hyperparameters *and* want an unbiased estimate of the tuned model's generalization error.
- Outer loop: k-fold split — each outer fold's test set is truly held out.
- Inner loop: on each outer training set, do a full k-fold hyperparameter search (or Bayesian search) and refit at the best config.
- Report the average outer-test score.
- Without nesting, hyperparameter selection leaks into your reported score.
- Standard for small-to-medium datasets and academic evaluation; too expensive for very large data or deep models.
Check yourself — multiple choice
- It's the same as regular k-fold
- Outer CV for evaluation + inner CV for hyperparameter tuning — unbiased tuned score
- It only works for classification
- It replaces the test set
Nested CV: outer for evaluation, inner for tuning ⇒ unbiased score of the tuned pipeline.
#cv#hyperparameter-tuning#validation
Practise Supervised Learning
215 interview questions in this topic.
Related questions
- Leave-One-Out CV — when is it a good idea, and when is it a bad idea?
- Why would you use repeated k-fold instead of standard k-fold?
- How do you set up cross-validation for time series?
- How do you pick the train/validation/test split sizes?
- How does cross-validation change for heavily imbalanced classification?
- You use stratified k-fold on a dataset with duplicate customer records — why is it wrong?