Contrast the pretraining objectives of BERT, GPT, and T5.
mediumAnswer
- BERT: masked language modeling — mask 15% of tokens, predict them (bidirectional context).
- GPT: causal / next-token prediction — predict token t from tokens 1..t-1 (unidirectional).
- T5: 'span corruption' — replace random spans with sentinel tokens, decoder generates the missing spans (encoder-decoder).
- Each objective aligns with the target usage: BERT for embeddings, GPT for generation, T5 for text-to-text.
Check yourself — multiple choice
- All three use masked LM
- BERT: masked LM; GPT: next-token; T5: span corruption — each maps to the intended downstream use
- All three use classification loss
- Only next-token
BERT/GPT/T5 differ in pretraining: masked / causal / span-corruption.
#transformers#training
Practise Deep Learning
214 interview questions in this topic.
Related questions
- What is teacher forcing and its main pitfall?
- State the Chinchilla scaling insight in one sentence.
- What are vanishing and exploding gradients, and how do you fix them?
- What does Batch Normalization do?
- How does dropout work and when is it applied?
- Why use a learning-rate schedule (warmup + cosine decay)?