Why is 'overfit a single batch' the first test you should run on a new model?
easyAnswer
- Because it isolates implementation bugs from learning problems.
- A correctly wired model with enough capacity can drive the loss on a handful of examples to nearly zero, since it can simply memorize them.
- If it cannot, the fault is mechanical rather than statistical: labels misaligned with inputs, the loss reading the wrong axis, gradients not flowing because of a detached tensor or a frozen parameter, or a learning rate so small nothing moves.
- The test costs seconds and rules out an entire class of silent failures before you spend GPU hours.
- Only once a single batch overfits does it make sense to talk about regularization, augmentation, or architecture.
Check yourself — multiple choice
- It measures generalization
- A wired-up model can memorize a few examples, so failure to overfit points to a mechanical bug — misaligned labels, detached gradients, frozen weights — not a learning problem
- It tunes the learning rate
- It replaces a validation set
The test separates implementation bugs from genuine learning difficulty at almost no cost.
#training#fundamentals
Practise Deep Learning
214 interview questions in this topic.
Related questions
- The same training script gives different results on two runs. How much of that can you remove?
- ReLU vs sigmoid vs GELU — when do you use each?
- What are vanishing and exploding gradients, and how do you fix them?
- What does Batch Normalization do?
- How does dropout work and when is it applied?
- Why use a learning-rate schedule (warmup + cosine decay)?