What are the roles of the training, validation and test sets?
easyAnswer
- Training set: fit the model parameters.
- Validation set: tune hyperparameters, choose between models, early stop.
- Test set: give one final unbiased estimate of generalization on unseen data.
- If you touch the test set during model selection you leak information and overestimate performance.
- Typical split for medium datasets: 70/15/15 or 60/20/20; for large data, ~1-2% is enough per set.
- For small data use cross-validation on train+val and hold out a small test.
Check yourself — multiple choice
- Train fits, validation tunes hyperparameters, test measures final generalization
- Test set is used to tune hyperparameters
- Validation set is unnecessary if you use k-fold
- All three sets should be shuffled together each epoch
Train fits parameters, validation tunes hyperparameters, test is the final unbiased evaluation.
#evaluation#validation
Practise Supervised Learning
215 interview questions in this topic.
Related questions
- What is k-fold cross-validation and when do you use stratified or grouped folds?
- What is the bias-variance tradeoff?
- How do you detect and fix overfitting?
- What does the IID assumption mean and when is it violated?
- What is the generalization gap and how do you shrink it?
- What is the out-of-bag (OOB) score in Random Forests?