What is 'dying ReLU' and how do you fix it?
mediumAnswer
- A ReLU neuron that always outputs zero has a zero gradient everywhere and stops learning permanently.
- Causes: large negative bias, extreme learning rate, poor init.
- Fixes: Leaky ReLU or ELU (nonzero slope on the negative side), He initialization, lower learning rate, better data normalization, or use GELU / SiLU which are smooth and don't fully saturate to zero.
Check yourself — multiple choice
- Dying ReLU only affects the final layer
- A neuron stuck at 0 with zero gradient — fix with Leaky/ELU/GELU, He init, or lower LR
- It resolves by itself
- Only dropout can fix it
Zero output → zero gradient → permanently dead. Leaky/ELU/GELU + good init fix it.
#activations#training
Practise Deep Learning
214 interview questions in this topic.
Related questions
- ReLU vs sigmoid vs GELU — when do you use each?
- What are vanishing and exploding gradients, and how do you fix them?
- What does Batch Normalization do?
- How does dropout work and when is it applied?
- Why use a learning-rate schedule (warmup + cosine decay)?
- Why do residual (skip) connections help training deep networks?