How do you train an LLM to a long context length?
hardAnswer
- Pretrain on shorter sequences (4-8k) for most of training — cheaper and more diverse.
- Then extend the context in a final continued-pretraining phase: increase to target (32k, 128k), rescale RoPE (linear interpolation or NTK-aware / YaRN), and train on filtered long-context documents (books, code repos, arXiv).
- Adds ~5-10% compute for a huge context boost.
- Alternative: 'position interpolation' fine-tuning on a small long-context set — cheap and effective for moderate extensions (2-8x).
Check yourself — multiple choice
- Train the full run at 128k
- Pretrain short, then continue-pretrain long with rescaled RoPE + long-context data — much cheaper than end-to-end long training
- Impossible without full retraining
- Context is fixed forever
Long-context training: short pretraining → long continued-pretraining with rescaled RoPE.
#pretraining#long-context
Practise LLMs & GenAI
214 interview questions in this topic.
Related questions
- What is the context window and what tricks extend it?
- Walk through how one training step of an autoregressive LLM works.
- What data goes into modern LLM pretraining?
- Why is data deduplication critical for LLM pretraining?
- Summarize Kaplan et al. (2020) scaling laws in one sentence.
- State the Chinchilla scaling insight and its practical impact.