What is stochastic depth / DropPath and where is it used?
mediumAnswer
- During training, randomly drop entire residual branches (set them to 0 and pass identity through the skip connection) with some probability p that increases with depth.
- Effectively trains an ensemble of networks of varying depth.
- Standard trick in modern deep vision transformers (Swin, ConvNeXt, DeiT) — enables much deeper training at higher accuracy.
Check yourself — multiple choice
- Same as dropout on activations
- Randomly drop entire residual branches — trains an implicit ensemble of shallower nets; standard in modern ViT/CNN variants
- Only used in RNNs
- DropPath removes the input
Stochastic depth: drop residual branches → deep-net ensemble regularization.
#regularization#training#architectures
Practise Deep Learning
214 interview questions in this topic.
Related questions
- How does dropout work and when is it applied?
- Why do residual (skip) connections help training deep networks?
- What is early stopping and how do you configure it?
- What is Stochastic Weight Averaging (SWA)?
- What is adversarial training (PGD, FGSM)?
- What is an EMA of weights and why do modern training recipes use it?