What are SiLU (Swish) and GELU, and why do transformers prefer them over ReLU?
mediumAnswer
- .
- ≈ x * Phi(x) — smooth version of ReLU using a Gaussian CDF.
- Both are smooth and non-monotonic near zero, giving richer gradients than ReLU's hard elbow.
- Empirically better on transformers and modern vision (ConvNeXt, EfficientNet).
- GELU is standard in BERT / GPT / most LLMs; SiLU in vision (EfficientNet) and some LLMs (Llama, Mistral).
Check yourself — multiple choice
- GELU is only used for regression
- Smooth ReLU variants — GELU ≈ x·Φ(x), SiLU = x·; better gradients, standard in transformers
- ReLU is smoother than GELU
- SiLU is only used in CNNs
GELU / SiLU: smooth ReLU variants ⇒ better gradients ⇒ standard in transformers.
#activations#transformers
Practise Deep Learning
214 interview questions in this topic.
Related questions
- ReLU vs sigmoid vs GELU — when do you use each?
- How does self-attention work in a transformer?
- Why does a transformer need positional encoding?
- Why did transformers replace RNNs for sequence modeling?
- Why does stacking linear layers without nonlinearity collapse to a single linear layer?
- What is 'dying ReLU' and how do you fix it?