What is grokking in deep learning?
hardAnswer
- Phenomenon (Power et al., 2022) where a network trained on a small algorithmic task memorizes the training set quickly (train loss → 0) but validation stays random for a long time — then, after many more epochs of continued training, suddenly generalizes to near-perfect validation accuracy.
- Suggests the network first memorizes, then slowly discovers a more compressed, generalizable circuit under continued regularization pressure.
Check yourself — multiple choice
- Overfitting that never generalizes
- Memorize quickly, then generalize much later after continued training
- Same as double descent
- Only affects transformers
Grokking = delayed generalization long after training loss hits zero.
#training#phase-transitions
Practise Deep Learning
214 interview questions in this topic.