FlashAttention — why is it faster?

hard

Answer

$O(N^{2})$
$FlashAttention (Dao 2022): (1) tiling: compute attention block-by-block in SRAM (fast on-chip memory).$
$(2) online softmax: recompute normalizer per tile.$
$(3) never materialize full attention matrix. \to 2-4x speedup + linear memory.$
$FlashAttention 2 + 3 further optimize.$
$Enables long-context LLMs (100k+ tokens).$
$Adopted in all modern implementations (PyTorch native, vLLM, HF, xformers).$

Check yourself — multiple choice

Random
Standard: $O(N^{2})$ memory for $\mathrm{QK}^{T}$ ; Flash: tiling in SRAM + online softmax + never materialize full matrix → 2-4x speed + linear memory; enables 100k+ context; adopted everywhere (PyTorch native/vLLM/HF)
Same
Not real

FlashAttention: tiling + online softmax; 2-4x + linear memory.

#serving#llmops

Practise MLOps & Data Quality

215 interview questions in this topic.

Read the lesson Flashcards Take the quiz

Related questions