Summarize Kaplan et al. (2020) scaling laws in one sentence.
hardAnswer
- Loss scales as a power law in three variables — parameters N, dataset size D, and compute C — with each factor showing predictable diminishing returns.
- Kaplan concluded compute-optimal training used relatively large N and small D; Chinchilla (Hoffmann 2022) corrected this by fitting jointly and showed the optimum uses roughly 20 tokens per parameter — bigger data than Kaplan predicted.
Check yourself — multiple choice
- No systematic pattern
- Loss follows power laws in N, D, C; Kaplan under-weighted D → Chinchilla showed ~20 tokens/param optimum
- Only params matter
- Only data matters
Scaling laws: power laws in N/D/C; Chinchilla corrected Kaplan's data/param ratio.
#pretraining#scaling
Practise LLMs & GenAI
214 interview questions in this topic.
Related questions
- State the Chinchilla scaling insight and its practical impact.
- What's the practical minimum tokens-per-parameter for a competitive LLM?
- Walk through how one training step of an autoregressive LLM works.
- What data goes into modern LLM pretraining?
- Why is data deduplication critical for LLM pretraining?
- What does 'emergence' mean in LLM capabilities and is it real?