FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness
FlashAttention, authored by Tri Dao and collaborators, represents one of the most foundational hardware-software breakthroughs in modern transformer systems engineering. Prior to FlashAttention, standard multi-head self-attention required materializing the full N-by-N attention matrix in High Bandwidth Memory (HBM), creating an O(N^2) memory bottleneck that severely constrained sequence lengths and wasted GPU compute cycles waiting for memory transfers.
Dao recognized that GPU execution is frequently memory-bound rather than compute-bound. FlashAttention redesigns self-attention to be IO-aware: it tiles input matrices into blocks, loads them into fast on-chip SRAM (shared memory), computes partial softmax statistics incrementally using the online softmax algorithm, and never materializes the quadratic intermediate attention matrix in slow HBM.
During the backward pass, rather than storing the massive attention matrix from the forward pass, FlashAttention recomputes attention on-the-fly directly within SRAM. Because SRAM read/write speeds are an order of magnitude faster than HBM, this recomputation trick yields a 2x to 4x overall training speedup while reducing memory footprint from quadratic O(N^2) to linear O(N).
FlashAttention has been incorporated into PyTorch core (torch.nn.functional.scaled_dot_product_attention), Hugging Face, vLLM, and every major frontier model training pipeline (including GPT-4, Llama 2/3, and Claude), enabling context windows to scale from 2k tokens to 128k+ tokens.
Tiles attention inputs across fast GPU SRAM blocks, eliminating quadratic HBM memory roundtrips.
Calculates exact softmax statistics dynamically across tiles without needing full row normalizers in advance.
Reduces attention memory footprint from O(N^2) to O(N), enabling 100k+ sequence context lengths.
Integrated natively into PyTorch 2.0+ core and universally adopted across LLM inference engines.
SRAM Block Tiling
Partitions Query, Key, and Value matrices into blocks sized specifically to fit within GPU on-chip shared memory.
Online Softmax Computation
Updates running maximums and normalization factors incrementally as Key-Value blocks stream through shared memory.
Activation Gradient Recomputation
Avoids storing N-by-N attention matrices by recomputing attention blocks directly in fast SRAM during backward pass.
Fused Kernel Execution
Dispatches fused Triton or CUDA kernels across streaming multiprocessors, achieving near-theoretical Tensor Core peak FLOPS.
CUDA C++, Triton, PyTorch C++ extensions, and modern GPU architectures.
Requires GPU compute capability >= 8.0 (NVIDIA Ampere, Ada, or Hopper) and compiled CUDA/Triton kernels.
Is FlashAttention an approximation or does it compute exact attention?
FlashAttention computes exact, mathematically identical attention to standard scaled dot-product attention. It is not an approximation (unlike sparse attention or linear attention). In fact, because it uses the online softmax algorithm with high-precision accumulators, it is often numerically more stable than standard PyTorch attention.
How does FlashAttention-2 improve upon the original FlashAttention?
FlashAttention-2 further optimizes work partitioning across CUDA thread blocks and warps, decreases non-matmul FLOPs, and parallelizes across sequence length rather than only batch size and attention heads, achieving up to 73% of theoretical peak A100 GPU compute throughput.
This proof of work artifact was source-checked on Sep 20, 2026 by the AI Experts Directory editorial team. Our source review confirms that public code repositories, research papers, and technical artifacts directly corroborate Tri Dao's active contributions. For full verification criteria, read our editorial methodology.
Inspect original artifact sources
Review raw code repositories, benchmark datasets, and technical citations directly on github.com.