Your validation loss is lower than your training loss. Is something broken?
mediumAnswer
- Usually not, and there are three ordinary explanations before you go looking for a bug.
- Regularization is active during training but not at evaluation, so dropout and stochastic depth make the training forward pass genuinely harder than the validation one.
- Augmentation does the same thing more strongly, since the model is scored on clean images but trained on distorted ones.
- And the reported training loss is an average over the epoch while the model was still improving, whereas validation is measured once at the end, so the training number reflects an older, worse model by roughly half an epoch.
- The signs that it really is a bug are a validation set that is easier than the training set because of a bad split, leakage of training examples into validation, or evaluation code that silently skips the hard cases.
Check yourself — multiple choice
- It always means a data leak
- Usually benign: dropout and augmentation only apply during training, and the training loss is an epoch average from a worse model — suspect a bug only if the split is unbalanced or examples leaked
- It means the model is underfitting
- Validation loss cannot be lower
Train-time regularization and the epoch-average timing explain most cases; a bad split or leakage is the exception.
#training#regularization#augmentation
Practise Deep Learning
214 interview questions in this topic.