How does a decoder block differ from an encoder block?
hardAnswer
- Decoder has three sublayers: (1) masked self-attention (each token can only attend to earlier positions — causal mask); (2) cross-attention over encoder outputs (queries from decoder, keys/values from encoder); (3) MLP.
- Each still wrapped in pre-norm + residual + dropout.
- Decoder-only LLMs (GPT) skip cross-attention and use only masked self-attention + MLP.
Check yourself — multiple choice
- Same as encoder
- Masked self-attn + cross-attn + MLP; GPT-style decoder-only drops cross-attn
- No masking used
- Only MLP
Decoder: masked self-attn + cross-attn + MLP; decoder-only LLM drops cross-attn.
#transformers#attention#architectures
Practise Deep Learning
214 interview questions in this topic.