What is AdaGrad and its main drawback?

medium

Answer

$Per-parameter learning rate: θ \leftarrow θ - η * g / (sqrt(sum of squared past gradients) + ε).$
$Parameters with large historical gradients get smaller effective LRs — great for sparse features (NLP with one-hot inputs).$
$Drawback: accumulator grows monotonically, so effective LR \to 0.$
$Model stops learning after enough steps.$
$RMSProp and Adam fix this with exponential moving averages.$

Check yourself — multiple choice

AdaGrad has constant learning rate
Per-param LR ∝ 1/√ ${\Sigma}g^{2}$ — great for sparse features but LR shrinks to zero over time
Same as SGD
Only works for CNNs

AdaGrad accumulates squared gradients → LR → 0 → RMSProp/Adam use EMA instead.

#optimizers

214 interview questions in this topic.