What is continual pretraining and when does it beat fine-tuning?
hardAnswer
- Take a pretrained base model and continue training on a large in-domain corpus (say, 10-100B tokens of medical literature, or code, or a specific language) with a reduced learning rate.
- Result: a domain-adapted base model that can be further instruction-tuned.
- Continual pretraining beats fine-tuning when: the domain corpus is large (>1B tokens), the target vocabulary/style differs from the base (medicine, law, non-English), or you need the model to know new facts baked in rather than retrieved.
Check yourself — multiple choice
- Same as fine-tuning
- Continue next-token training on a large in-domain corpus (>1B tokens) → domain-adapted base for further fine-tune
- Only fine-tuning is used
- Not possible without full retrain
Continual pretraining: >1B in-domain tokens on the base model → serious domain adaptation.
#pretraining#fine-tuning
Practise LLMs & GenAI
214 interview questions in this topic.
Related questions
- Fine-tuning vs RAG — which do you use when?
- What is LoRA and why is it popular for fine-tuning LLMs?
- Walk through how one training step of an autoregressive LLM works.
- What data goes into modern LLM pretraining?
- Why is data deduplication critical for LLM pretraining?
- Summarize Kaplan et al. (2020) scaling laws in one sentence.