Why isn't setting a seed enough for full reproducibility?

hard

Answer

$Even with fixed seed, non-determinism from: (1) parallel GPU ops (cuDNN atomic ops, reduction order varies).$
$(2) multi-threaded data loading order.$
$(3) hardware differences (fp32 vs tf32 vs bf16).$
$(4) library version updates.$
$(5) non-deterministic algorithms (scatter add, some conv).$
$Fix: (a) torch.use_deterministic_algorithms(True) .$
$\mathrm{CUBLAS}_{\mathrm{WORKSPACE}}\mathrm{CONFIG} = 40968$
$(c) single-threaded data loading.$
$(d) exact library + CUDA versions.$
$Cost: slower training.$
$Full determinism only within same HW + versions.$

Check yourself — multiple choice

Random
Seed alone: parallel GPU non-determinism + threading + HW precision (fp/tf/bf) + library versions + non-det algos; fix via deterministic mode + CUBLAS config + single-thread + pinned versions; slower train
Always same
Not real

Non-determinism: GPU / threads / HW / versions / algos; seed alone insufficient.

#reproducibility

215 interview questions in this topic.