Write the vanilla SGD update rule and explain each term.
easyAnswer
- , where θ is the parameter vector, η the learning rate, and = ∇_θ is the stochastic gradient on the current mini-batch.
- Each step is a noisy estimate of the true gradient — the noise itself acts as regularizer and helps escape saddle points.
- Convergence rate for convex problems: O(1/√T).
Check yourself — multiple choice
- θ ← θ + η · g (add gradient)
- θ ← θ - η · g — noisy estimate on mini-batch, regularizes and escapes saddles
- SGD uses second-order info
- SGD requires full-batch gradient
Vanilla SGD: subtract learning rate × mini-batch gradient.
#optimizers#optimization
Practise Deep Learning
214 interview questions in this topic.
Related questions
- SGD vs Adam vs AdamW — how do you choose?
- What is Shampoo and when does second-order optimization pay off?
- When do you prefer SGD with momentum over Adam?
- What is the 'implicit bias' of SGD?
- How does classical momentum modify SGD?
- What is Nesterov momentum and how is it different from classical momentum?