How does Flash Attention 2 speed up training?
hardAnswer
- Flash Attention (Dao 2022) tiles Q, K, V into blocks fitting in SRAM and uses online softmax to compute exact attention with O(n) memory instead of — 2-4x speedup.
- Flash Attention 2 (Dao 2023) restructures the parallelism: parallelizes along sequence length in the forward pass and along heads/batch in the backward pass, uses fewer non-matmul FLOPs, achieves 50-70% of theoretical GPU throughput.
- Standard kernel in modern LLM training and inference.
Check yourself — multiple choice
- Approximate attention
- Exact attention with SRAM tiling + online softmax; v2 restructures parallelism → 50-70% peak GPU throughput
- Only for training
- Removes softmax
Flash Attention v2: exact, IO-aware, better parallelism → 2-4× speedup near peak GPU.
#architecture#attention#inference
Practise LLMs & GenAI
214 interview questions in this topic.
Related questions
- What is Grouped-Query Attention (GQA) and why does it matter?
- What is the context window and what tricks extend it?
- What is a Mixture-of-Experts (MoE) LLM?
- What is quantization and what tradeoffs come with it?
- What is the KV cache and why does it matter for LLM inference?
- Why do modern LLMs use decoder-only architectures?