What are NTK-aware RoPE scaling and YaRN?
hardAnswer
- Techniques to extend a RoPE-based LLM's context window beyond the training length without retraining from scratch.
- Naive linear interpolation of positions (Position Interpolation) shrinks the effective frequency and hurts short-range attention.
- NTK-aware scaling adjusts higher frequencies less than lower ones (motivated by the Neural Tangent Kernel view).
- YaRN (Peng et al., 2023) refines this with per-frequency temperature and a small fine-tune — lets LLaMA-2 extend from 4k to 128k tokens at minimal quality loss.
Check yourself — multiple choice
- Retrain from scratch
- NTK-aware / YaRN: adjust RoPE frequencies unevenly + light fine-tune → extend context 10-30× cheaply
- Same as ALiBi
- Impossible without full retraining
NTK-aware / YaRN: rescale RoPE frequencies + small fine-tune → cheap context extension.
#positional-encoding#transformers#efficient-attention
Practise Deep Learning
214 interview questions in this topic.