Why is data deduplication critical for LLM pretraining?
mediumAnswer
- Duplicate documents cause the model to memorize verbatim rather than generalize (Carlini et al., 2020).
- Effects: worse generalization, higher memorization risk (training data extraction attacks), inflated benchmark scores if test data leaks via duplicates, wasted compute.
- Modern pretraining pipelines run exact-hash dedup + fuzzy dedup (MinHash / SimHash on shingles) at document and paragraph levels — often removing 20-40% of the raw corpus.
Check yourself — multiple choice
- Dedup doesn't matter
- Prevents memorization + benchmark leakage + wasted compute; runs exact + fuzzy (MinHash) dedup, removes 20-40% of corpus
- Only for legal reasons
- Only run at inference
Dedup: prevents memorization / benchmark leakage / compute waste; MinHash + exact-hash pipelines.
#pretraining#data
Practise LLMs & GenAI
214 interview questions in this topic.
Related questions
- What data goes into modern LLM pretraining?
- Is data curriculum used in LLM pretraining?
- Can you repeat data across epochs in LLM pretraining?
- What is benchmark contamination and how do you detect it?
- Walk through how one training step of an autoregressive LLM works.
- Summarize Kaplan et al. (2020) scaling laws in one sentence.