Why does fp16 training need loss scaling?
hardAnswer
- fp16 has a small dynamic range (~5e-8 to 6.5e4).
- Gradients often live near the underflow boundary — many gradient elements silently become zero.
- Solution: multiply loss by a scale factor S (e.g., 128) before backward → gradients are S× larger → survive fp16 range → divide by S before optimizer step.
- Dynamic loss scaling adjusts S automatically: increase when no NaN, halve when a NaN appears.
- Not needed with bf16 (has fp32's exponent range).
Check yourself — multiple choice
- No loss scaling needed for fp16
- Scale the loss up before backward so gradients don't underflow in fp16; dynamic scaling adjusts S automatically
- Only applied for bf16
- Loss scaling changes the objective
fp16 loss scaling: prevent gradient underflow; not needed with bf16.
#mixed-precision#training
Practise Deep Learning
214 interview questions in this topic.
Related questions
- Your training loss becomes NaN. Debug it.
- How does mixed-precision training work with fp16 / bf16?
- What are vanishing and exploding gradients, and how do you fix them?
- What does Batch Normalization do?
- How does dropout work and when is it applied?
- Why use a learning-rate schedule (warmup + cosine decay)?