Write Adam's full update rule.

medium

Answer

$m_{t}\; = \;{\beta}_{1}\; \cdot \;m_{t} - 1\; + \;(1 - {\beta}_{1})\; \cdot \;g_{t}$
$m_{\mathrm{hat}}\; = \;m_{t}\; / \;(1\; - \;{\beta}_{1}^{t})$
${\theta}\; - \;{\eta}\; \cdot \;m_{\mathrm{hat}}\; / \;(\mathrm{sqrt}(v_{\mathrm{hat}})\; + \;{\varepsilon})$
${\beta}_{1} = 0.9$
$Combines momentum and per-param LR — robust default optimizer.$

Check yourself — multiple choice

Adam uses only the second moment
Adam maintains EMAs of g (m) and $g^{2}$ (v), applies bias correction, updates θ - η·m̂/√v̂
Adam is deterministic
Adam removes momentum

Adam: EMA of g and $g^{2}$ , bias-corrected → per-param LR + momentum.

#optimizers

214 interview questions in this topic.