When does a random train/test split give you a misleading score?
mediumAnswer
- Whenever rows are not independent.
- With repeated measurements per user, a random split puts the same user on both sides and the model recognizes the user rather than the pattern; use GroupKFold on the user id.
- With time-ordered data, a random split lets the model see the future; use a forward-chaining split.
- With near-duplicate rows, such as augmented images or scraped pages, duplicates straddle the split and inflate the score; deduplicate first.
- The rule is that the split has to mimic the generalization you actually need.
Check yourself — multiple choice
- Random splits are always fine
- When rows are dependent — repeated users, time order or near-duplicates — the split must mimic real generalization
- Only for small datasets
- Only in regression
Dependence between rows breaks random splitting; group, time-based or dedup-aware splits are needed.
#cv#validation
Practise Supervised Learning
215 interview questions in this topic.
Related questions
- Leave-One-Out CV — when is it a good idea, and when is it a bad idea?
- What is nested cross-validation and when do you need it?
- Why would you use repeated k-fold instead of standard k-fold?
- How do you set up cross-validation for time series?
- How do you pick the train/validation/test split sizes?
- How does cross-validation change for heavily imbalanced classification?