State the Chinchilla scaling insight and its practical impact.
hardAnswer
- For compute C ~= 6*N*D, the compute-optimal split gives N and D roughly proportional (~20 tokens per parameter).
- Older LLMs like GPT-3 (175B, 300B tokens = 1.7 tokens/param) were massively undertrained.
- Practical shift: modern open LLMs (LLaMA-2 7B on 2T tokens = 285 tokens/param; LLaMA-3 8B on 15T = 1875 tokens/param) train small models on huge data — cheaper inference, stronger quality per active parameter.
Check yourself — multiple choice
- GPT-3 was compute-optimal
- Compute-optimal: ~20 tokens/param; GPT-3 was 10× undertrained → modern LLMs train smaller nets on 100-1000× more data
- Params matter, tokens don't
- Bigger is always better
Chinchilla: 20 tokens/param optimum → modern LLMs are small + very-well-trained.
#pretraining#scaling
Practise LLMs & GenAI
214 interview questions in this topic.
Related questions
- Summarize Kaplan et al. (2020) scaling laws in one sentence.
- What's the practical minimum tokens-per-parameter for a competitive LLM?
- Walk through how one training step of an autoregressive LLM works.
- What data goes into modern LLM pretraining?
- Why is data deduplication critical for LLM pretraining?
- What does 'emergence' mean in LLM capabilities and is it real?