What is LARS and when is it used?
hardAnswer
- Layer-wise Adaptive Rate Scaling (You et al., 2017).
- Scales the update per layer by |||| / ||||, keeping the ratio update-to-weight at ~1%.
- Enables training ResNet-50 in a few minutes with batch sizes up to 32k.
- Same idea as LAMB but for SGD-with-momentum.
- Standard for very-large-batch training in vision.
Check yourself — multiple choice
- Global LR scaling
- Per-layer LR scaling by ||θ||/||g|| — enables extreme batch sizes without divergence
- LARS is a learning-rate schedule
- LARS uses Adam moments
LARS: per-layer scaling → stable extreme-batch training with SGD.
#optimizers#distributed#batch-size
Practise Deep Learning
214 interview questions in this topic.