Can you repeat data across epochs in LLM pretraining?
hardAnswer
- Yes, moderately — repeating a fixed pretraining set for 2-4 epochs is roughly equivalent to training on 2-4x more unique tokens on a log scale (Muennighoff 2023).
- Beyond ~4 epochs, gains diminish sharply and memorization increases.
- Data-constrained regimes (specialized languages, code, medicine) rely on this — modern practice mixes fresh data with high-quality repeated corpora, with repeat counts up to ~5-10 for the best subsets.
Check yourself — multiple choice
- Never repeat
- Repeat up to ~4 epochs ≈ same as scaling data; diminishing returns beyond, memorization rises
- Repeat 100 times
- Only for classification
Data-constrained: repeat up to ~4 epochs ≈ log-scale equivalent to more unique tokens.
#pretraining#data
Practise LLMs & GenAI
214 interview questions in this topic.
Related questions
- What data goes into modern LLM pretraining?
- Why is data deduplication critical for LLM pretraining?
- Is data curriculum used in LLM pretraining?
- What is benchmark contamination and how do you detect it?
- Walk through how one training step of an autoregressive LLM works.
- Summarize Kaplan et al. (2020) scaling laws in one sentence.