EasyDeepLearn
Supervised Learning · section 11 of 18

Cross-validation & data splitting

10 interview questions on cross-validation & data splitting, each answered in full. Free to read, no account needed.

Leave-One-Out CV — when is it a good idea, and when is it a bad idea?

medium
  • LOOCV uses n-1 samples for training and 1 for validation, repeated n times.
  • Pros: nearly unbiased estimate of generalization error, uses maximum training data per fold.
  • Cons: (1) computationally brutal for anything but linear models with closed-form updates; (2) high variance of the estimate because folds are almost identical; (3) doesn't work with grouped data (each row is its own fold).
  • Prefer 5- or 10-fold CV for most practical cases.
  • LOOCV is only clearly best when n is very small.
#cv#validationPermalink & quiz →

What is nested cross-validation and when do you need it?

hard
  • You need it whenever you tune hyperparameters *and* want an unbiased estimate of the tuned model's generalization error.
  • Outer loop: k-fold split — each outer fold's test set is truly held out.
  • Inner loop: on each outer training set, do a full k-fold hyperparameter search (or Bayesian search) and refit at the best config.
  • Report the average outer-test score.
  • Without nesting, hyperparameter selection leaks into your reported score.
  • Standard for small-to-medium datasets and academic evaluation; too expensive for very large data or deep models.
#cv#hyperparameter-tuning#validationPermalink & quiz →

Why would you use repeated k-fold instead of standard k-fold?

medium
  • A single k-fold estimate has significant variance from the specific random split — especially on small data.
  • Repeated k-fold runs k-fold R times with different seeds, averaging results.
  • Cost: R * k fits.
  • Benefit: much tighter estimate of generalization error (variance drops roughly by 1/R), and you get a distribution of scores for statistical tests.
  • Common: 5-fold × 10 repeats.
  • For very large data, one 5-fold pass is usually enough — variance is already low.
#cv#validationPermalink & quiz →

How do you set up cross-validation for time series?

hard
  • Never random splits — that leaks future info into training.
  • Options: (1) walk-forward / rolling-origin: expand or slide a training window and evaluate on the next block.
  • (2) blocked / purged k-fold: keep folds contiguous in time and add a purge gap between train and validation to remove any labeled points near the boundary.
  • (3) time-series split with combinatorial purging + embargoes (Lopez de Prado) for financial data with overlapping labels.
  • Always check the last training timestamp precedes the first validation timestamp.
#cv#validation#time-seriesPermalink & quiz →

How do you pick the train/validation/test split sizes?

easy
  • Depends on total n and problem noise.
  • Small (n < 10k): 70/15/15 or use CV on train+val and hold out ~15% for final test.
  • Medium (10k-1M): 80/10/10.
  • Large (1M+): 98/1/1 is often enough — validation just needs a stable estimate of the metric (typically ≥ 10k samples).
  • Time series: contiguous, in temporal order, with test as the newest chunk.
  • Imbalanced: ensure enough minority-class examples in val and test — stratify.
#cv#validationPermalink & quiz →

How does cross-validation change for heavily imbalanced classification?

medium
  • Two things: (1) use stratified k-fold to guarantee each fold contains a representative share of the minority class — otherwise some folds might have zero positives, breaking metrics.
  • (2) evaluate with imbalance-aware metrics (PR-AUC, F1, recall at fixed precision) — accuracy is meaningless.
  • If the minority class is extremely rare (<0.1%), consider repeated stratified splits, or bootstrap with stratification, to get a reliable metric estimate.
#cv#imbalance#validationPermalink & quiz →

You use stratified k-fold on a dataset with duplicate customer records — why is it wrong?

medium
  • Stratified k-fold preserves class proportions but ignores group structure.
  • If the same customer appears in both training and validation, the model can 'memorize' that customer via any customer-specific features (device, location patterns, etc.) — leakage that inflates validation performance.
  • Fix: use StratifiedGroupKFold (sklearn) or manually group-then-stratify.
  • The group is the leakage unit (customer, patient, product) — everything sharing an id must go into the same fold.
#cv#validation#data-qualityPermalink & quiz →

When would you use the bootstrap for model evaluation instead of k-fold?

hard
  • Bootstrap resamples the training set with replacement to build many pseudo-datasets.
  • Fit the model on each and evaluate on the out-of-bootstrap (OOB) samples (~37% left out).
  • Advantages: gives you a full distribution over the metric — confidence intervals come free, useful for statistical comparisons.
  • Weaknesses: overlap across bootstrap samples means each replicate is highly correlated; slightly optimistic estimate.
  • Prefer k-fold for model selection; bootstrap when you specifically need CIs on the metric.
#cv#validationPermalink & quiz →

When is cross-validation NOT necessary?

easy
  • When you have enough data that a single held-out validation set is already a low-variance estimate (typically  nval  ≥  10k  with  well−behaved  labels)(\mathrm{typically}\;n_{\mathrm{val}}\; \ge \;10k\;\mathrm{with}\;\mathrm{well} - \mathrm{behaved}\;\mathrm{labels}).
  • At Google/Facebook scale you almost never do k-fold — a single split is enough.
  • Also: pure production monitoring uses live traffic slices, not CV.
  • Also: pure inference-time evaluation on a fixed test set.
  • CV shines on small-to-medium data where any single split's variance is meaningful.
#cv#validationPermalink & quiz →

When does a random train/test split give you a misleading score?

medium
  • Whenever rows are not independent.
  • With repeated measurements per user, a random split puts the same user on both sides and the model recognizes the user rather than the pattern; use GroupKFold on the user id.
  • With time-ordered data, a random split lets the model see the future; use a forward-chaining split.
  • With near-duplicate rows, such as augmented images or scraped pages, duplicates straddle the split and inflate the score; deduplicate first.
  • The rule is that the split has to mimic the generalization you actually need.
#cv#validationPermalink & quiz →

Practise Supervised Learning

All 18 sectionsFlashcardsQuiz