Why does Adafactor use less memory than Adam?
hardAnswer
- Adafactor factorizes the second-moment matrix v: instead of storing a full per parameter (O(N) memory), it stores row-sum and column-sum statistics for each 2D weight matrix (O(sqrt(N))).
- Massive memory savings — enabled T5 training.
- Trade-off: some hyperparameters harder to tune, occasionally less stable than full Adam.
Check yourself — multiple choice
- Adafactor uses more memory
- Factorizes the second moment via row/column stats — O(√N) memory instead of O(N)
- Adafactor uses second-order Hessian
- Removes momentum entirely
Adafactor factorizes → dramatically less optimizer memory (used for T5).
#optimizers#distributed
Practise Deep Learning
214 interview questions in this topic.