Encoder-only vs decoder-only vs encoder-decoder — when do you pick each?
mediumAnswer
- Encoder-only (BERT, RoBERTa): bidirectional attention over the whole input → best for understanding tasks (classification, NER, retrieval embeddings).
- Decoder-only (GPT, LLaMA): causal attention → best for open-ended generation and general chat assistants.
- Encoder-decoder (T5, BART): full attention encoder + causal decoder + cross-attention → best for seq2seq (translation, summarization).
- Modern LLMs converged on decoder-only for scale simplicity and general-purpose behavior.
Check yourself — multiple choice
- All three are identical
- Encoder-only: understanding; decoder-only: open generation; encoder-decoder: seq2seq — LLMs converged on decoder-only
- Only encoder-decoder exists
- Decoder-only is only for translation
Understanding→encoder; generation→decoder; seq2seq→encoder-decoder; LLMs are decoder-only.
#transformers#architectures
Practise Deep Learning
214 interview questions in this topic.
Related questions
- How does a Vision Transformer (ViT) treat images?
- What does Swin Transformer do differently from ViT?
- Describe a single transformer encoder block in detail.
- How does a decoder block differ from an encoder block?
- Why do residual (skip) connections help training deep networks?
- How does self-attention work in a transformer?