What is the context window and what tricks extend it?
mediumAnswer
- The context window is the maximum number of tokens (prompt + response) the model can attend to.
- Extending it: RoPE scaling (position interpolation, YaRN), sparse/linear attention, sliding-window attention, retrieval-augmented long-context, and KV-cache tricks.
- Longer context helps but is not free — attention is quadratic in sequence length, and quality often degrades far from the beginning ('lost in the middle').
Check yourself — multiple choice
- Attention scales linearly with sequence length by default
- Standard attention scales quadratically with sequence length
- Longer context always improves quality uniformly
- The KV cache is unrelated to context length
Vanilla attention is in the sequence length.
#long-context#architecture#inference
Practise LLMs & GenAI
214 interview questions in this topic.
Related questions
- What is Grouped-Query Attention (GQA) and why does it matter?
- How does YaRN extend an LLM's context beyond training length?
- How does Flash Attention 2 speed up training?
- What is a Mixture-of-Experts (MoE) LLM?
- What is quantization and what tradeoffs come with it?
- What is the KV cache and why does it matter for LLM inference?