Is data curriculum used in LLM pretraining?
hardAnswer
- Yes but with restraint.
- Common practice: start with a slightly noisier / broader mix, then finish the last 10-20% of training on a higher-quality curated 'annealing' set (e.g., Wikipedia, curated books, high-quality code, math) at a decayed learning rate.
- LLaMA-3, Qwen-2 and Gemma explicitly use this 'quality upsampling at the end'.
- Careful with curriculum: overly aggressive orderings can hurt generalization vs uniform sampling.
Check yourself — multiple choice
- No curriculum ever used
- Broad mix then quality-upsampled annealing on the last 10-20% of steps at decayed LR — LLaMA-3, Qwen-2, Gemma
- Only train on Wikipedia
- Random order strictly required
Modern LLMs: quality upsampling at the end of training — annealing curriculum.
#pretraining#data
Practise LLMs & GenAI
214 interview questions in this topic.
Related questions
- What data goes into modern LLM pretraining?
- Why is data deduplication critical for LLM pretraining?
- Can you repeat data across epochs in LLM pretraining?
- What is benchmark contamination and how do you detect it?
- Walk through how one training step of an autoregressive LLM works.
- Summarize Kaplan et al. (2020) scaling laws in one sentence.