Where is dropout typically inserted in a transformer block?
mediumAnswer
- (1) On attention weights (after softmax), (2) on attention output projections, (3) on MLP hidden and output activations, (4) on residual sums (dropout after each residual add — 'residual dropout').
- Rates: ~0.1 for pretraining (BERT), sometimes 0.0 for LLM pretraining (large models overfit less).
- Fine-tuning usually adds dropout on the classifier head.
Check yourself — multiple choice
- Only in the loss
- On softmax attention, on output projections, on MLP activations, and on residual sums
- Only in the embedding
- Never in transformers
Transformer dropout spots: attention softmax, output proj, MLP, residual.
#regularization#transformers
Practise Deep Learning
214 interview questions in this topic.
Related questions
- Why does dropout play a smaller role in large language model training than in older vision models?
- How does dropout work and when is it applied?
- How does self-attention work in a transformer?
- Why does a transformer need positional encoding?
- Why did transformers replace RNNs for sequence modeling?
- What are SiLU (Swish) and GELU, and why do transformers prefer them over ReLU?