How much extra memory does Adam use versus SGD?
mediumAnswer
- SGD with momentum: 1x parameter count in optimizer state (momentum).
- Adam: 2x parameter count (first and second moment estimates).
- AdamW: same as Adam.
- For a 7B model in fp32, this is 28 GB just for m and v — a big reason why large-model training uses fp16 / bf16 optimizer states, sharded optimizers (ZeRO), or memory-efficient optimizers (Adafactor, 8-bit Adam).
Check yourself — multiple choice
- Adam and SGD use identical memory
- Adam: 2× params (m and v); SGD momentum: 1× params — sharded optimizers or 8-bit Adam save memory
- Adam uses less than SGD
- Optimizer state fits in cache
Adam stores m and v (2× params) — motivates ZeRO, 8-bit Adam, Adafactor.
#optimizers#distributed
Practise Deep Learning
214 interview questions in this topic.