Your training loss becomes NaN. Debug it.
mediumAnswer
- (1) Check for divisions by zero and log of non-positive values — most common cause.
- (2) Reduce learning rate; loss NaN often means gradient exploded.
- (3) Add gradient clipping.
- (4) Verify data has no NaN / inf.
- (5) In fp16 / mixed precision, check the loss scale; use bf16 if available (more forgiving range).
- (6) Reset from a known-good checkpoint.
- (7) Print the loss and gradient norms every N steps in a rerun to isolate the offending step.
Check yourself — multiple choice
- Only reset the seed
- Check for div-by-zero / log-of-zero, reduce LR, add grad clipping, verify data, use bf16, trace step-by-step
- Increase learning rate
- Delete the training set
NaN debugging: numerical operations first, then LR/clipping/precision, then data checks.
#training#mixed-precision
Practise Deep Learning
214 interview questions in this topic.
Related questions
- How does mixed-precision training work with fp16 / bf16?
- Why does fp16 training need loss scaling?
- What are vanishing and exploding gradients, and how do you fix them?
- What does Batch Normalization do?
- How does dropout work and when is it applied?
- Why use a learning-rate schedule (warmup + cosine decay)?