What is the 'implicit bias' of SGD?
hardAnswer
- Among the infinite minimizers of the training loss in an over-parameterized net, SGD tends to select ones with certain 'flat' properties — approximately minimum-norm solutions in linear cases, and empirically flat-minimum / max-margin solutions in nonlinear ones.
- This implicit bias explains why deep nets trained by SGD generalize despite having capacity to fit noise.
- Weight decay and small LR reinforce this bias.
Check yourself — multiple choice
- SGD picks a random minimizer
- SGD prefers 'flat', low-norm, high-margin minima — explaining generalization in over-parameterized nets
- SGD only works with tiny models
- Adam has no implicit bias
SGD implicitly biases toward flat / low-norm / high-margin minima.
#training#theory#optimization
Practise Deep Learning
214 interview questions in this topic.