How do Longformer / BigBird combine sparse and global attention?
hardAnswer
- Combine local sliding-window attention (each token sees a window of size w) with a handful of 'global' tokens (e.g., [CLS], question tokens in QA) that attend to every position and are attended by every position.
- BigBird adds a random-attention pattern for provably close approximation of full attention.
- Enables 4k-16k context on hardware where full attention would OOM.
Check yourself — multiple choice
- Only local windows
- Local windows + a few global tokens (+ random in BigBird) → linear-ish scaling, provably close to full attention
- Removes attention
- Same as MoE
Longformer/BigBird: local + global (+ random) attention → long-context scaling.
#attention#efficient-attention#transformers
Practise Deep Learning
214 interview questions in this topic.
Related questions
- Why is standard self-attention O(n²) in sequence length?
- How do Performer / Linformer / linear attention reduce O(n²)?
- What is sparse / sliding-window attention?
- What is Flash Attention?
- What is the KV cache in transformer inference?
- Attention is quadratic in sequence length. Why is FlashAttention still a major win without changing that?