Why does dropout play a smaller role in large language model training than in older vision models?

hard

Answer

$Because the regularization pressure comes from the data instead.$
$When a model sees each token roughly once, on a corpus far larger than its parameter count, memorization is not the binding constraint, so the variance reduction dropout provides is not needed and its added gradient noise mostly slows convergence.$
$Older vision models made many passes over a small labelled set, where overfitting was the dominant failure and dropout genuinely helped.$
$Modern large-scale pretraining therefore uses little or no dropout, relying on data volume, weight decay and early stopping in the form of a single epoch.$
$Dropout returns during fine-tuning on small task datasets, where the classic overfitting regime is back.$

Check yourself — multiple choice

Dropout was proven wrong
At web scale each example is seen about once, so overfitting is not the binding constraint and dropout's noise only slows training — it returns for small-dataset fine-tuning
Transformers cannot use dropout
It is only a memory concern

Regularization need depends on the data-to-parameter ratio and number of passes, not on architecture alone.

#regularization#transformers#fine-tuning

214 interview questions in this topic.