Describe a single transformer encoder block in detail.
hardAnswer
- Input x.
- Sublayer 1: y = x + Dropout(MultiHeadAttention(LayerNorm(x))).
- Sublayer 2: z = y + Dropout(MLP(LayerNorm(y))), where MLP = Linear → GELU → Linear with d.
- Modern variants use RMSNorm instead of LN, GLU-family MLPs, and pre-norm placement.
- All ops are parallel across sequence positions — no recurrence.
Check yourself — multiple choice
- MLP + attention with no residuals
- Attention sublayer + MLP sublayer, each wrapped in pre-norm + residual + dropout; MLP has ≈ 4d
- Uses recurrence
- No MLP
Transformer block: attn sublayer + MLP sublayer, both with pre-norm + residual + dropout.
#transformers#attention#architectures
Practise Deep Learning
214 interview questions in this topic.