What is deep double descent?

hard

Answer

$Test error as a function of model capacity is not monotonically U-shaped: it goes up as capacity crosses the interpolation threshold (train error \to 0), then goes down again for extremely overparameterized models.$
$Documented in Belkin et al. (2019) and Nakkiran et al. (2020).$
$Implication: in the overparameterized regime, bigger models can generalize better despite fitting the training data perfectly.$

Check yourself — multiple choice

Test error is always U-shaped in capacity
Test error peaks at the interpolation threshold then decreases again for very large models
Only observed in linear models
Requires early stopping

Double descent: test error peaks at interpolation, then drops for overparameterized models.

#training#phase-transitions#theory

214 interview questions in this topic.