What is DropConnect and how is it different from Dropout?
hardAnswer
- Dropout zeros activations; DropConnect zeros weights (elements of the weight matrix) — i.e., randomly severs individual connections.
- More granular regularization but harder to implement efficiently on GPUs.
- Occasionally used in specialized architectures but Dropout dominates in practice due to speed.
Check yourself — multiple choice
- Same as Dropout
- Zeros weights instead of activations — finer regularization but slower on GPUs
- DropConnect zeros gradients
- Adds noise to labels
DropConnect: zero individual weights, not activations.
#regularization
Practise Deep Learning
214 interview questions in this topic.
Related questions
- How does dropout work and when is it applied?
- Why does AdamW work better than Adam + L2 weight decay for transformers?
- How does weight decay change the objective and the update?
- What does SAM (Sharpness-Aware Minimization) do?
- What is stochastic depth / DropPath and where is it used?
- Where is dropout typically inserted in a transformer block?