How does dropout work and when is it applied?
easyAnswer
- During training, dropout randomly zeros a fraction p of activations per forward pass, forcing the network to distribute knowledge and not co-adapt.
- At inference, all units are active and outputs are scaled (or scaling is done at train time — 'inverted dropout').
- Common rates: 0.1-0.5.
- Modern transformers use dropout in attention and MLP layers.
Check yourself — multiple choice
- Dropout is applied only at inference
- Dropout randomly zeros activations during training
- Dropout replaces activations with random values at inference
- Dropout is the same as weight decay
Dropout is a train-time regularization; inference uses the full network.
#regularization#training
Practise Deep Learning
214 interview questions in this topic.
Related questions
- What is stochastic depth / DropPath and where is it used?
- What is early stopping and how do you configure it?
- What is Stochastic Weight Averaging (SWA)?
- What is adversarial training (PGD, FGSM)?
- What is an EMA of weights and why do modern training recipes use it?
- Your validation loss is lower than your training loss. Is something broken?