The same training script gives different results on two runs. How much of that can you remove?
mediumAnswer
- Less than people expect, and it is worth knowing which sources you control.
- Seeding the framework, Python and NumPy fixes weight initialization, dropout masks and shuffling order, which removes most of the variance.
- What remains comes from the hardware: several GPU kernels, notably atomics used in scatter operations and some convolution algorithms, accumulate in nondeterministic order, and floating-point addition is not associative, so results differ in the last bits and then diverge over thousands of steps.
- Frameworks expose a deterministic mode that swaps in slower ordered kernels, which gets you bit-identical runs at a real throughput cost.
- Data loader worker count and any non-seeded augmentation also matter.
- The practical stance is to seed everything, report results over several seeds rather than one, and reserve full determinism for debugging.
Check yourself — multiple choice
- Setting one seed makes training fully reproducible
- Seeding removes initialization, dropout and shuffling variance, but nondeterministic GPU kernels plus non-associative float addition remain — deterministic mode fixes it at a throughput cost
- Nothing can be controlled
- Only the batch size matters
Seeds cover the software sources; bit-identical runs additionally require slower deterministic kernels.
#training#fundamentals
Practise Deep Learning
214 interview questions in this topic.
Related questions
- Why is 'overfit a single batch' the first test you should run on a new model?
- ReLU vs sigmoid vs GELU — when do you use each?
- What are vanishing and exploding gradients, and how do you fix them?
- What does Batch Normalization do?
- How does dropout work and when is it applied?
- Why use a learning-rate schedule (warmup + cosine decay)?