What is gradient noise scale (GNS) and how do you use it?
hardAnswer
- GNS (McCandlish et al., 2018) measures the ratio between the variance of the mini-batch gradient and its squared mean, roughly telling you how much batch size you can grow before returns diminish.
- Compute / ||g||².
- Below , doubling batch halves steps.
- Above, benefits plateau.
- Guides choice of batch size for large-model training.
Check yourself — multiple choice
- GNS is a measure of activation variance
- Ratio Tr(Cov(g)) / ||g||² — tells you the critical batch size beyond which scaling gives diminishing returns
- GNS is an activation function
- GNS is a loss
GNS: Tr(Cov(g))/||g||² = critical batch size — guides scaling of large-model batches.
#batch-size#distributed#training
Practise Deep Learning
214 interview questions in this topic.
Related questions
- What is gradient accumulation and when do you use it?
- How does gradient accumulation let you train with a batch that does not fit in memory, and what does it not fix?
- How does batch size affect training dynamics?
- What problem does LAMB solve?
- How should learning rate scale with batch size?
- What is LARS and when is it used?