What data goes into modern LLM pretraining?
mediumAnswer
- A mix of: (1) filtered CommonCrawl web text (60-80%), (2) high-quality curated corpora — books, Wikipedia, arXiv, StackExchange (5-20%), (3) code — GitHub, competitive programming, docs (5-20%), (4) mathematical text and problem sets.
- Multi-lingual coverage.
- Aggressive deduplication (near-duplicate removal at document, paragraph, and n-gram levels), quality filtering (classifier trained on Wikipedia vs random web), and PII / toxic content removal.
- Data quality > data quantity beyond a point.
Check yourself — multiple choice
- Only Wikipedia
- Filtered web + books + code + math, aggressively deduplicated and quality-filtered; multi-lingual
- Random internet only
- Only English books
Modern pretraining: filtered mix of web/books/code/math + aggressive dedup and quality filtering.
#pretraining#data
Practise LLMs & GenAI
214 interview questions in this topic.
Related questions
- Why is data deduplication critical for LLM pretraining?
- Is data curriculum used in LLM pretraining?
- Can you repeat data across epochs in LLM pretraining?
- What is benchmark contamination and how do you detect it?
- Walk through how one training step of an autoregressive LLM works.
- Summarize Kaplan et al. (2020) scaling laws in one sentence.