Learn · Kernels & silicon

FlashAttention and paged attention: attention as GPU kernels

Updated 10 Nov 2026 · 10 min read

Attention and the KV cache explains what attention computes. This page is about how GPU kernels compute it fast. Attention is two matrix multiplies with a softmax in between. The hard part is not the math. It is the size of the matrix in the middle.

The idea first

What FlashAttention is for. Attention compares every token with every earlier token. For a 4,000-token prompt that is a 4,000 × 4,000 sheet of scores per head. The math on that sheet is cheap for a GPU; writing the sheet to memory and reading it back is not. FlashAttention computes the same answer without ever writing the sheet down.

The analogy: a desk and a warehouse. HBM is a big warehouse, far away. The chip’s LDS and registers are a small desk, close by. Naive attention fills the warehouse with the whole score sheet, then walks back and forth to read it. FlashAttention keeps a block of questions (queries) on the desk, has the warehouse send the keys and values over one block at a time, and keeps only three running notes per question:

Note What it means The question it answers
m the best match score seen so far “What is my reference point?” (softmax needs the row’s max)
l the total weight of everything seen so far “What do I divide by at the end?” (softmax’s denominator)
o the weighted blend of values seen so far “What is my answer so far?” (the numerator)

When a better match shows up, the old notes were written relative to the old best, so they are scaled down once (multiply by exp(old m − new m)) and the new block is added. Like converting a running total into a new currency: one multiply, no need to recount what came before.

Desk, warehouse, three notes
HBM (thewarehouse)Q, K, V, O: big, far, slowLDS and registers of one workgroup: a few hundred KB, many times faster than HBM.On chip (the desk)Q block + one K/V blockThree small numbers (o is a d-wide vector) per query row. They replace the whole row of S and P.Running notesm = best score · l = total · o= blendFull score sheet ST × T: never writtenAnswer Owritten onceone K/V block at a timeupdatenaive onlyo ÷ l
  1. Questions on the deskA workgroup copies a block of query rows (for example 128) onto the chip once.
  2. One block of keys and values at a timeIt asks HBM for the next K and V block (for example 64 keys) and scores its queries against them, right there in registers.
  3. Update the notes, throw the scores awayThe block’s scores update m, l and o and are then discarded. The next block overwrites them.
  4. What naive attention does insteadNaive attention writes the whole T × T score sheet (and then the weight sheet P) to HBM and reads both back. For 64 heads and 4,000 tokens that is gigabytes per layer.
  5. Write the answer onceAfter the last block: answer = o ÷ l, written to HBM once.
S and P exist only one small tile at a time, on the desk.

Where the analogy breaks. The desk is tiny (LDS is 160 KB per CU on MI355X), so block sizes are a tuning trade-off, not a free choice. And the result is not an estimate: the running notes give the exact same answer as the full sheet (checked with numbers below).

The naive version and its problem

For one head, with TT tokens and head size dd:

  1. S=Q⋅K⊤/dS = Q \cdot K^\top / \sqrt{d}. QQ and KK are T×dT \times d, so SS is T×TT \times T.
  2. P=softmax⁡(S)P = \operatorname{softmax}(S), row by row, with future positions masked (causal).
  3. O=P⋅VO = P \cdot V, T×dT \times d.

Take a 4,000-token prompt and gpt-oss-120b’s 64 query heads (published config). One head’s S in FP32 is 4,000 × 4,000 × 4 B = 64 MB; all heads of one layer, 4 GB. A naive implementation writes S to HBM, reads it back for the softmax, writes P, and reads P again for P⋅VP \cdot V. That traffic dwarfs the 4×T2×d4 \times T^2 \times d FLOPs of the two products, so naive attention is memory-bound and its memory use grows with T2T^2. Move the sliders to see by how much:

HBM traffic: naive attention vs FlashAttention (one layer, prefill)
S for one head (FP32)
Naive: bytes moved
Flash: bytes moved
Matrix FLOPs (QKᵀ + PV)
Naive: FLOP per byte
Flash: FLOP per byte

Model: naive reads Q, K, V, writes O, and writes then reads S (FP32) and P (BF16); flash reads Q and writes O once, and reads K and V once per query block and KV head (with GQA the query heads of a group share one read). BF16 Q/K/V/O. Non-causal (causal halves both FLOPs and flash K/V reads), no L2 hits. Time = max(bytes ÷ 8 TB/s, FLOPs ÷ 2.5 PFLOP/s), MI355X spec peaks, approximate: a lower bound, not a prediction.

Read the FLOP-per-byte tiles. FlashAttention’s intensity stays near the query block size times the GQA group (at most 128 × 8 = 1,024; the Q and O traffic pull it a little lower), whatever the prompt length: each K/V byte it reads is used by every query row of the block and every query head of the group. Naive attention’s intensity stays in the tens (about d/3d/3) because its T×TT \times T sheets grow as fast as the math.

FlashAttention: never store S

FlashAttention (Dao et al., 2022) computes exact attention without ever writing S or P to HBM. A workgroup takes a block of query rows, keeps them on chip, and streams K and V through LDS one block at a time. For each K/V block it computes a tile of S in registers, turns it into a tile of P, multiplies by V, and accumulates into O. The T×TT \times T matrix exists only one tile at a time.

The obstacle is the softmax. Each row of PP is exp⁡(si−max⁡)/∑jexp⁡(sj−max⁡)\exp(s_i - \max) / \sum_j \exp(s_j - \max), and the max and the sum run over the whole row, which you have not seen yet when you process the first block.

The online softmax

The trick (Milakov and Gimelshein, 2018, “online normalizer calculation”) is the three running notes from the idea first. Per query row and per K/V block:

m_new = max(m, max_j s_j)
scale = exp(m - m_new)                       # shrink old terms to the new reference
l     = l * scale + sum_j exp(s_j - m_new)
o     = o * scale + sum_j exp(s_j - m_new) * v_j
m     = m_new
# after the last block: output = o / l

Worked example: one query row, three blocks

The same toy model as the attention walkthrough (d=4d = 4, one head, tokens “The GPU is fast ! .”). We follow the query of the last token “.” through its 6 keys, 2 keys per block. Every number is computed at build time.

Online softmax for the query of “.”

1 · What naive attention would store

S = Q·Kᵀ ÷ √4, causal (6 × 6)
query ↓ key →TheGPUisfast!.
The0.21−∞−∞−∞−∞−∞
GPU0.592.53−∞−∞−∞−∞
is0.251.020.47−∞−∞−∞
fast0.652.591.162.59−∞−∞
!0.591.910.662.151.57−∞
.0.561.800.622.031.481.60

Naive attention writes this whole matrix to HBM, then reads it back. FlashAttention cuts it into tiles (here 2 × 2), computes one tile at a time in registers, and skips the tiles above the diagonal. We follow the outlined row: q“.” = [0.5, 0.7, 1, 1.2] against all six keys, two keys at a time.

2 · Block 1: keys “The”, “GPU”

Scores s (this block)
TheGPU
.0.561.80
→
p = exp(s − m), m = 1.795
TheGPU
.0.2911.000
×
V block
d1d2d3d4
The0.500.200.000.60
GPU1.000.800.801.00
o (running, not yet divided)
d1d2d3d4
o1.1450.8580.8001.175
o ÷ l so far
d1d2d3d4
.0.890.660.620.91
running max m
−∞ → 1.795
rescale old by
nothing yet
running sum l
1.291

The first block sets m to its larger score, 1.795. p = exp(0.56 − 1.795) = 0.291, exp(1.795 − 1.795) = 1; l = 0.291 + 1 = 1.291. o is p times the two value rows. If the row ended here, o ÷ l would already be the right answer for these two keys.

3 · Block 2: keys “is”, “fast” (new max: rescale)

Scores s (this block)
isfast
.0.622.03
→
p = exp(s − m), m = 2.03
isfast
.0.2431.000
×
V block
d1d2d3d4
is0.101.200.200.20
fast1.200.400.901.90
o before (× 0.791)
d1d2d3d4
o1.1450.8580.8001.175
→
o (running, not yet divided)
d1d2d3d4
o2.1301.3701.5812.877
o ÷ l so far
d1d2d3d4
.0.940.610.701.27
running max m
1.795 → 2.03
rescale old by
0.791
running sum l
2.263

Key “fast” scores 2.03, above the old max 1.795. Everything accumulated so far was computed relative to the old max, so l and o are multiplied by exp(1.795 − 2.03) = 0.791 before the new terms are added: l = 1.291 × 0.791 + 0.243 + 1 = 2.263. This one multiply is the whole trick: no earlier score has to be revisited.

4 · Block 3: keys “!”, “.” (max unchanged)

Scores s (this block)
!.
.1.481.60
→
p = exp(s − m), m = 2.03
!.
.0.5770.647
×
V block
d1d2d3d4
!0.901.600.101.30
.1.000.600.402.20
o before (× 1)
d1d2d3d4
o2.1301.3701.5812.877
→
o (running, not yet divided)
d1d2d3d4
o3.2962.6811.8985.051
o ÷ l so far
d1d2d3d4
.0.950.770.541.45
running max m
2.03 → 2.03
rescale old by
1
running sum l
3.488

Both scores are below m = 2.03, so the scale is exp(0) = 1 and the block just adds its terms: l = 2.263 × 1 + 0.577 + 0.647 = 3.488. This was the last block.

5 · Divide once, and check against plain softmax

Flash: o ÷ l
d1d2d3d4
.0.9450.7690.5441.448
=
Plain: softmax(s) · V
d1d2d3d4
.0.9450.7690.5441.448
Plain softmax weights (the row naive attention stores as P)
TheGPUisfast!.
.0.070.230.070.290.170.19

One division at the end gives the output. It matches the textbook result to within 2e-16 (floating-point rounding), and the weights are the same row as for “.” in the attention walkthrough. The result is exact, not an approximation, and the full row of weights was never stored. Subtracting the running max also keeps exp() from overflowing, which matters in FP16 and BF16.

6 · Split-KV: two workgroups, then merge

Workgroup 1: keys 1–3
d1d2d3d4
o₁1.1761.2270.8611.236

m₁ = 1.795, l₁ = 1.598

Workgroup 2: keys 4–6
d1d2d3d4
o₂2.3671.7111.2174.074

m₂ = 2.03, l₂ = 2.224

→
Merged: (a₁·o₁ + a₂·o₂) ÷ l
d1d2d3d4
.0.9450.7690.5441.448

In decode there is one query row, so one workgroup per sequence would leave most CUs idle. Split-KV gives each half of the keys to its own workgroup. Each returns its own (m, l, o). A small merge kernel rescales both to the larger max (2.03: a₁ = 0.791, a₂ = 1), adds them (l = 3.488) and divides. Same answer, the same rescaling as step 3.

A real kernel does the same with 64 to 128 keys per block, 64 to 128 query rows at once, and d = 64 or 128. The blocks of S and P live in registers only.

The kernel

FlashAttention forward, one workgroup
FOR EACH K/V BLOCK J (CAUSAL: ONLY J UP TO THE DIAGONAL)HBMQ, K, V, OLDSK and V blockMatrix coresv_mfma / mmaVector ALUmax, exp, scaleload Q block (e.g. 128 rows) intoregisters, onceload K_j, V_j (e.g. 64 keys)S = Q·K_jᵀ ÷ √dS tile stays in registersmask, new row max m, P = exp(S − m),update sum l, rescale OP (cast to BF16/FP8)O += P·V_jafter last block: O ÷ l, store O (and l, m)
Block sizes are typical examples. S and P never leave registers. Step through the messages with the player.
  • Grid: one workgroup per (query block, head, sequence). A 4,000-token prompt with 128-row query blocks and 64 heads gives 32 × 64 = 2,048 workgroups: plenty for 256 CUs.
  • Two GEMMs per K/V block run on the matrix cores (v_mfma_* on AMD, mma / wgmma / tcgen05.mma on NVIDIA). The softmax steps (max, exp, scale) run on the ordinary vector ALUs, which are many times slower per FLOP than matrix cores. Keeping that part small and overlapped is most of the tuning work.
  • Causal masking skips K/V blocks entirely above the diagonal: roughly half the work for prefill.
  • Grouped-query attention (GQA): gpt-oss has 64 query heads but only 8 KV heads, so 8 query heads read the same K and V. Kernels load a K/V block once and use it for all 8.
  • Sliding window: gpt-oss alternates sliding_attention layers (window 128 tokens) with full_attention layers (published config). In sliding layers each query only visits the last 128 keys, so the K/V loop is a few blocks long however long the prompt is.

HBM traffic is now roughly: read Q once, read K and V once per query block, write O once. Prefill attention becomes compute-bound for long prompts, like a GEMM.

Generations

Version What changed (from the papers)
FlashAttention (2022) IO-aware tiling; S never written to HBM; memory linear in T
FlashAttention-2 (2023) fewer non-matmul FLOPs; one head split across thread blocks; better work split between warps. About 2× FlashAttention: 50–73 % of A100 peak, up from 25–40 %
FlashAttention-3 (2024) Hopper-specific: asynchronous Tensor Cores and TMA, warp specialization, softmax interleaved with matmul, FP8. 1.5–2× FlashAttention-2 on H100

On AMD, FlashAttention-style kernels come from Composable Kernel (CK), AITER and Triton. vLLM picks one per platform; see the backend list below.

Decode: paged attention

In decode each sequence brings one new query token per step, and its keys and values sit in the KV cache. Two things change.

1. The shape is a matrix-vector product. Per head, qq is 1×d1 \times d and KK is n×dn \times d for nn cached tokens. There is no reuse of K or V within a sequence: each cached byte is used once per step. For gpt-oss with 16-bit KV, each cached token is 2 × 8 KV heads × 64 × 2 B = 2,048 bytes per layer and yields about 16,000 FLOPs across the 64 query heads: ≈8 FLOP/B, far below any ridge point. Decode attention speed is set by how fast the kernel streams the KV cache from HBM. → Roofline

2. The KV cache is paged. vLLM stores K and V in fixed-size blocks (for example 16 tokens) scattered across HBM, with a block table per sequence that lists its block numbers in order, like a page table (PagedAttention, Kwon et al., 2023). The attention kernel follows the table:

// Illustrative inner loop of a paged decode kernel, one workgroup = one (sequence, KV head)
for (logical_block = 0; logical_block < num_blocks(seq); logical_block++) {
  phys  = block_table[seq][logical_block];          // indirection, like a page table
  K_blk = key_cache[phys][kv_head];                  // e.g. 16 tokens x 64 dims
  V_blk = value_cache[phys][kv_head];
  s     = q[8 query heads] . K_blk^T / sqrt(d);      // GQA: 8 queries share this K
  online-softmax update of (m, l, o) with s and V_blk
}
out[seq][8 query heads] = o / l
One paged decode workgroup reading its sequence's KV
KV CACHE POOL IN HBM (16 TOKENS PER BLOCK)Holds q for the 8 query heads that share KV head 0 (GQA), and the running m, l, o in registers.Workgroupsequence A, KV head 0, 8query headsLogical block 0 → physical 7, 1 → 2, 2 → 9. Written by the vLLM scheduler on the CPU, read by the kernel.Block table, seq A[7, 2, 9]Block 7tokens 0–15Block 2tokens 16–31Block 9tokens 32–40 (9 of 16used)Block 4another sequenceOutputo ÷ l for 8 query headslook up
  1. The workgroup handles sequence A and KV head 0, so it holds the queries of the 8 query heads in that GQA group. It first reads the sequence’s block table: logical blocks 0, 1, 2 live in physical blocks 7, 2 and 9.
  2. Physical block 7: K and V of tokens 0–15 for this KV head (16 × 64 × 2 B = 2 KiB each). Scores for all 8 queries, then the online-softmax update of m, l, o.
  3. Block 2, tokens 16–31. The blocks are not adjacent in memory; only the table knows the order. Each byte is used once per step, by 8 query heads.
  4. Block 9, the partly filled last block (9 of 16 slots). The kernel masks the unused slots. Block 4 belongs to another sequence and is never touched.
  5. Divide o by l for each of the 8 query heads and write the output. With split-KV, several workgroups each take some blocks and a merge kernel combines them (step 6 above).

Too few workgroups: split-KV

One workgroup per (sequence, KV head) gives a batch of 4 sequences × 8 KV heads = 32 workgroups for 256 CUs. Long contexts make each of those workgroups long and the GPU mostly idle. Split-KV (also called Flash-Decoding) splits each sequence’s KV cache into chunks, runs one workgroup per chunk, and then a small reduction kernel combines the partial (m, l, o) triples, using the same rescaling as the online softmax (step 6 of the walkthrough). This is the attention version of split-K.

Mixed batches

Servers like vLLM V1 batch prefill chunks and decode tokens from different requests into one step. Their attention backends therefore take a variable number of query tokens per sequence (thousands for a prefill chunk, one for a decode) over the same paged KV cache, and pick tiling per sequence inside one launch.

vLLM attention backends

vLLM chooses a backend at start-up and logs the choice (Using %s backend out of potential backends: ..., or selected via --attention-backend). Current vLLM source registers, among others:

Backend Platform Kernel source
FLASH_ATTN NVIDIA FlashAttention 2/3
FLASHINFER NVIDIA FlashInfer
TRITON_ATTN both Triton
ROCM_ATTN AMD vLLM’s ROCm kernels
ROCM_AITER_FA AMD AITER FlashAttention

Which backend a server uses is printed in its start-up log; backend names change between vLLM versions.

How this shows up in operations

  • Long prompts cost quadratically in prefill attention, linearly in KV memory. TTFT for a 32k-token prompt is dominated by attention far more than for a 4k prompt. → Prefill
  • KV-cache block size and layout are part of the contract between prefill and decode in disaggregated serving: the KV transfer must reorder blocks into the decode side’s layout.
  • Backend choice is a start-up decision. A different image, vLLM version or environment variable can switch the attention backend and change speed with no code change.

Further reading