ReLU vs sigmoid vs GELU — when do you use each?
easyAnswer
- Sigmoid/tanh saturate and cause vanishing gradients — avoid in hidden layers of deep nets, keep sigmoid for a binary output.
- ReLU is the default hidden activation: fast, sparse, but suffers from 'dying ReLU' (permanently zero neurons).
- Leaky ReLU / ELU fix that.
- GELU is a smooth approximation of ReLU and is the standard in transformers (BERT, GPT).
- SiLU/Swish is also popular in modern vision and LLMs.
Check yourself — multiple choice
- Sigmoid is the best hidden-layer activation for deep nets
- GELU is standard in transformer architectures
- ReLU never has a 'dying' problem
- Tanh does not saturate
Transformers (BERT, GPT) use GELU or a similar smooth activation.
#activations#fundamentals
Practise Deep Learning
214 interview questions in this topic.
Related questions
- Why does stacking linear layers without nonlinearity collapse to a single linear layer?
- What does temperature do in a softmax?
- In one sentence, what is backpropagation?
- State the universal approximation theorem in one sentence and its practical caveat.
- Depth vs width — which do you scale first?
- Why can't you initialize a neural net with all zeros?