What problem does LAMB solve?
hardAnswer
- LAMB (Layer-wise Adaptive Moments for Batch training) enables very large-batch training (32k+ per step) without loss of accuracy.
- It scales the Adam update per layer by the ratio |||| / ||||.
- Used to train BERT-Large in 76 minutes on TPU pods.
- Similar spirit to LARS (which does the same for SGD).
Check yourself — multiple choice
- LAMB is a small-batch optimizer
- Layer-wise adaptive scaling of Adam — enables very-large-batch training without accuracy loss
- LAMB removes normalization
- LAMB replaces cross-entropy
LAMB: per-layer scaling of the Adam step — makes 32k-batch training feasible.
#optimizers#distributed#batch-size
Practise Deep Learning
214 interview questions in this topic.