How does YaRN extend an LLM's context beyond training length?
hardAnswer
- YaRN (Peng et al., 2023) is a RoPE frequency-rescaling technique.
- Split the RoPE frequency dimensions into three regions by their wavelength: high-frequency dims are left untouched (short-range), low-frequency dims are linearly interpolated (long-range), mid-range gets a smooth interpolation.
- Add a per-frequency temperature adjustment on attention logits to counteract the entropy shift from position scaling.
- Fine-tune for ~1B tokens with the new scale.
- Result: LLaMA-2 → 128k context at minimal quality loss.
Check yourself — multiple choice
- Retrain from scratch
- RoPE frequency-rescaling by wavelength band + temperature adjust + ~1B-token fine-tune → 128k context
- Same as PI naive linear
- Same as ALiBi
YaRN: per-wavelength RoPE rescale + temperature + short fine-tune → long-context extension.
#long-context#architecture
Practise LLMs & GenAI
214 interview questions in this topic.
Related questions
- What is the context window and what tricks extend it?
- What is a Mixture-of-Experts (MoE) LLM?
- Why do modern LLMs use decoder-only architectures?
- What is SwiGLU and why do LLMs use it in the MLP block?
- Why did RoPE replace learned positional embeddings in modern LLMs?
- What is Grouped-Query Attention (GQA) and why does it matter?