How does mixed-precision training work with fp16 / bf16?
mediumAnswer
- Store weights in fp32, cast to fp16 (or bf16) for forward and backward passes, accumulate gradients in fp32, then update weights in fp32.
- Cuts activation memory ~2x and speeds up compute 2-3x on modern GPUs with tensor cores. fp16 has small range (needs loss scaling to prevent gradient underflow); bf16 has fp32's range but less precision — usually plug-and-play, preferred on Ampere+.
- Use torch.cuda.amp / bf16 autocast.
Check yourself — multiple choice
- Purely fp32
- fp16/bf16 forward-backward, fp32 weights/accum — ~2-3× faster + half memory; bf16 avoids loss scaling
- Requires TPU only
- No speedup
Mixed precision: fp16/bf16 compute + fp32 master weights → faster, less memory.
#mixed-precision#distributed#training
Practise Deep Learning
214 interview questions in this topic.