What's the practical minimum tokens-per-parameter for a competitive LLM?
hardAnswer
- Chinchilla optimal: ~20 tokens/param.
- But 'compute-optimal at fixed budget' is not the same as 'best quality per param at fixed inference cost'.
- Modern practice trains way past Chinchilla to get a smaller model with the same quality: LLaMA-3 8B on 15T tokens = 1875 tokens/param.
- Rationale: inference cost dominates the model's lifetime; a smaller model that took more training compute is cheaper to serve.
Check yourself — multiple choice
- Exactly 20 tokens/param always
- Chinchilla: ~20 tokens/param; practice trains way past that (100-2000 tokens/param) for cheaper inference
- 1 token/param is enough
- Impossible to over-train
Modern LLMs train past Chinchilla — small models on huge data → cheap inference forever.
#pretraining#scaling#cost
Practise LLMs & GenAI
214 interview questions in this topic.
Related questions
- Summarize Kaplan et al. (2020) scaling laws in one sentence.
- State the Chinchilla scaling insight and its practical impact.
- How much does it cost to pretrain a modern LLM?
- Walk through how one training step of an autoregressive LLM works.
- What data goes into modern LLM pretraining?
- Why is data deduplication critical for LLM pretraining?