What is sparse / sliding-window attention?

medium

Answer

$Restrict each token to attend to a fixed neighborhood (window of size w) rather than all n tokens.$
$O(n^{2})$
$Stack layers so the effective receptive field grows with depth.$
$Used in Longformer (window + a few global tokens), BigBird, Mistral's sliding window attention.$
$Great when relevant context is local.$
$Combine with a few globally-attending tokens for long-range info.$

Check yourself — multiple choice

Removes softmax
Each token attends within a fixed local window → O(n·w); stack layers to grow the effective range
Same as vanilla attention
Doubles complexity

Sliding-window / sparse attention: local window → O(n·w) instead of $O(n^{2})$ .

#attention#efficient-attention#transformers

214 interview questions in this topic.