State the Chinchilla scaling insight in one sentence.
hardAnswer
- For a given compute budget C = 6 * N * T (params × tokens), the compute-optimal split has N and T roughly proportional (~20 tokens per parameter), meaning older large models (GPT-3, Gopher) were undertrained: same compute would have given a smaller model trained on more data with much better loss.
- Reshaped modern LLM training toward smaller-but-longer-trained models (LLaMA, Mistral).
Check yourself — multiple choice
- Bigger is always better
- Compute-optimal: N and T grow together (~20 tokens per parameter) — older LLMs were undertrained (Chinchilla)
- Data doesn't matter
- Params don't matter
Chinchilla: ~20 tokens per parameter is compute-optimal; GPT-3 was undertrained.
#training#transformers
Practise Deep Learning
214 interview questions in this topic.
Related questions
- What is teacher forcing and its main pitfall?
- Contrast the pretraining objectives of BERT, GPT, and T5.
- What are vanishing and exploding gradients, and how do you fix them?
- What does Batch Normalization do?
- How does dropout work and when is it applied?
- Why use a learning-rate schedule (warmup + cosine decay)?