Interview · flashcards

Deep Learning

Neural networks, training tricks, CNNs, RNNs, transformers and everything in between.

easyactivationsfundamentals

ReLU vs sigmoid vs GELU — when do you use each?

Click, press Space or use the button below to reveal the answer

Interview-style answer

$Sigmoid/tanh saturate and cause vanishing gradients — avoid in hidden layers of deep nets, keep sigmoid for a binary output.$
$ReLU is the default hidden activation: fast, sparse, but suffers from 'dying ReLU' (permanently zero neurons).$
$Leaky ReLU / ELU fix that.$
$GELU is a smooth approximation of ReLU and is the standard in transformers (BERT, GPT).$
$SiLU/Swish is also popular in modern vision and LLMs.$

Click again to flip back

1 / 214

Shortcuts: Space/Enter flip · ← / → navigate · K = known · R = review