What is Flash Attention?
hardAnswer
- Dao et al. (2022) IO-aware attention algorithm: tile Q, K, V into blocks that fit in on-chip SRAM, compute softmax incrementally with the online softmax trick, and never materialize the full n×n attention matrix in HBM.
- Same math as standard attention (exact, not approximate), but 2-4x faster and O(n) memory instead of .
- Enabled longer contexts (32k, 100k) on the same hardware.
- Flash-Attention-2 improves parallelism further.
Check yourself — multiple choice
- Approximates attention
- Exact attention, IO-aware tiling with online softmax → 2-4× faster, O(n) memory, enables long context
- Same as Performer
- Removes softmax
Flash Attention: exact + IO-aware tiling → dramatic memory reduction, longer contexts.
#attention#efficient-attention#transformers
Practise Deep Learning
214 interview questions in this topic.
Related questions
- Why is standard self-attention O(n²) in sequence length?
- How do Performer / Linformer / linear attention reduce O(n²)?
- What is sparse / sliding-window attention?
- How do Longformer / BigBird combine sparse and global attention?
- What is the KV cache in transformer inference?
- Attention is quadratic in sequence length. Why is FlashAttention still a major win without changing that?