How do you extend a transformer to very long contexts?

hard

Answer

$Options: (1) sparse/local attention (Longformer, Mistral's sliding window); (2) linear-attention approximations (Performer, RWKV, Mamba SSM); (3) retrieval augmentation (chunk documents, retrieve top-k with a vector store, stuff into the prompt); (4) position-encoding extrapolation (RoPE scaling, NTK-aware interpolation, YaRN); (5) hierarchical models (encode chunks separately then attend at a higher level).$
$Combine multiple techniques in practice.$

Check yourself — multiple choice

Only bigger GPUs
Sparse/linear attention, retrieval, RoPE scaling, hierarchical modeling — usually combined
Cannot extend context
Only remove attention

Long context: sparse/linear attn + retrieval + RoPE scaling + hierarchical modeling.

#transformers#efficient-attention

214 interview questions in this topic.