Learn · LLM inference
Attention and the KV cache
Tokens and transformers showed that each layer has an attention block, the only place where tokens see each other. This page opens that block, then explains the KV cache: the per-token memory that makes generation fast, and that decides how many users fit on a GPU and how many bytes a disaggregated server must ship from the prefill GPU to the decode GPU.
The idea first
The problem attention solves. A word’s meaning depends on the words around it. “Fast” means something different in “the GPU is fast” and in “Ramadan fast”. “It” means nothing until you know what it points to. So every token needs a way to pull in information from the earlier tokens that matter to it, and to ignore the rest. Attention is that mechanism: a soft lookup that every token runs over all earlier tokens, in every layer.
Three roles, three matrices. For the lookup, each token plays three different parts. The model learns one weight matrix per part and multiplies the token’s vector by it:
| Matrix | Gives each token a… | The question it answers | Library analogy | In “The GPU is fast” |
|---|---|---|---|---|
| WQ | query | “What am I looking for?” | the search you type | “fast” asks: what thing am I describing? |
| WK | key | “What can I be found for?” | the label on a book’s spine | “GPU” advertises: I am a thing, a noun; “The”: I am a function word |
| WV | value | “What do I hand over if picked?” | the book’s contents | “GPU” hands over: what a GPU is |
Why three separate matrices, not one? Because how a word is found and what it gives are different things. “GPU” should be found by any word looking for a noun (its key), but what it should pass on is its meaning (its value), and what it is itself looking for (its query, perhaps a verb) is a third thing again. Training adjusts the three matrices so that useful lookups score high.
- AskThe token “fast” turns into a query: a vector that encodes what it is looking for.
- Compare with every labelIts query is compared with the key of every token so far (including itself). A dot product measures how well two vectors point the same way: big when the query and key match.
- Take a blend of contentsThe match strengths become shares that add up to 1. The token takes that share of each value and adds the blend to itself. “Fast” now carries information about “GPU”.
Why each step is there:
- Dot product (query · key): a cheap similarity score. Vectors that point the same way give a big number.
- Divide by : longer vectors give bigger dot products; scaling keeps scores in a range where the next step stays smooth instead of all-or-nothing.
- Softmax: turns scores into shares of a fixed budget of 1. A token can spread its attention or focus it, but cannot attend “more than everything”.
- Causal mask: a token may only look backwards. You cannot use words that have not been written yet when generating left to right.
- Weighted sum of values: the actual information transfer. The output is mostly the values of the best matches.
Many heads = many questions at once. One head asks one kind of question. A layer runs many heads in parallel, each with its own three matrices: one may track which noun an adjective describes, another what “it” refers to, another nearby punctuation. (Heads specialise during training; their “questions” are not always human-readable.)
Where the library analogy breaks. The lookup is soft: the token takes a blend of all books, weighted, not one book. Queries, keys and values are learned vectors, not words. And it happens in every layer: later layers look up over representations that earlier layers already mixed.
The KV cache, intuitively. Once a token is in the text, its label (key) and its contents (value) never change, because later words cannot affect earlier ones (the mask). So the server keeps them on the shelf. Each new token brings one new question (its query, used once and thrown away) and adds one new label + book to the shelf. The shelf only grows: that growing shelf, in every layer and every head, is the KV cache, and it is what fills GPU memory and what disaggregated serving ships from the prefill machine to the decode machine.
Now the same thing with real numbers.
Worked example: four tokens, real numbers
Watch the idea above run on a toy model: one head, vectors of 4 numbers instead of 64, and the prompt “The GPU is fast”. Every number below is computed from the ones before it (the embeddings and weights are made up; a real model learns them). Step through with the arrows, or switch to Spread to see every step at once.
Attention on “The GPU is fast”, then two decode steps
1 · Each token is a vector
| token | d1 | d2 | d3 | d4 |
|---|---|---|---|---|
| The | 0.20 | 0.10 | 0.00 | 0.30 |
| GPU | 0.90 | 0.40 | 0.80 | 0.10 |
| is | 0.10 | 0.60 | 0.20 | 0.00 |
| fast | 0.70 | 0.20 | 0.90 | 0.50 |
Each row is one token’s hidden state as it enters the attention block: 4 numbers here, 2,880 in gpt-oss-120b. Nothing yet connects the tokens; each row knows only about itself.
2 · Project: Q, K, V = X × three weight matrices
| The | 0.20 | 0.10 | 0.00 | 0.30 |
|---|---|---|---|---|
| GPU | 0.90 | 0.40 | 0.80 | 0.10 |
| is | 0.10 | 0.60 | 0.20 | 0.00 |
| fast | 0.70 | 0.20 | 0.90 | 0.50 |
| 1 | 0 | 1 | 0 |
| 0 | 1 | 0 | 1 |
| 1 | 1 | 0 | 0 |
| 0 | 0 | 1 | 1 |
| d1 | d2 | d3 | d4 | |
|---|---|---|---|---|
| The | 0.20 | 0.10 | 0.50 | 0.40 |
| GPU | 1.70 | 1.20 | 1.00 | 0.50 |
| is | 0.30 | 0.80 | 0.10 | 0.60 |
| fast | 1.60 | 1.10 | 1.20 | 0.70 |
| d1 | d2 | d3 | d4 | |
|---|---|---|---|---|
| The | 0.30 | 0.10 | 0.30 | 0.50 |
| GPU | 1.30 | 1.20 | 0.90 | 1.00 |
| is | 0.70 | 0.80 | 0.20 | 0.10 |
| fast | 0.90 | 1.10 | 1.40 | 1.20 |
| d1 | d2 | d3 | d4 | |
|---|---|---|---|---|
| The | 0.50 | 0.20 | 0.00 | 0.60 |
| GPU | 1.00 | 0.80 | 0.80 | 1.00 |
| is | 0.10 | 1.20 | 0.20 | 0.20 |
| fast | 1.20 | 0.40 | 0.90 | 1.90 |
Three matrix multiplies give every token a query (“what am I looking for?”), a key (“what do I offer?”) and a value (“what I pass on if chosen”). Row “GPU” of Q is row “GPU” of X times WQ: 0.9·[1,0,1,0] + 0.4·[0,1,0,1] + 0.8·[1,1,0,0] + 0.1·[0,0,1,1] = [1.7, 1.2, 1, 0.5].
3 · Score every query against every key: Q·Kᵀ / √d
| The | 0.20 | 0.10 | 0.50 | 0.40 |
|---|---|---|---|---|
| GPU | 1.70 | 1.20 | 1.00 | 0.50 |
| is | 0.30 | 0.80 | 0.10 | 0.60 |
| fast | 1.60 | 1.10 | 1.20 | 0.70 |
| The | GPU | is | fast | |
|---|---|---|---|---|
| d1 | 0.30 | 1.30 | 0.70 | 0.90 |
| d2 | 0.10 | 1.20 | 0.80 | 1.10 |
| d3 | 0.30 | 0.90 | 0.20 | 1.40 |
| d4 | 0.50 | 1.00 | 0.10 | 1.20 |
| query ↓ key → | The | GPU | is | fast |
|---|---|---|---|---|
| The | 0.21 | 0.62 | 0.18 | 0.74 |
| GPU | 0.59 | 2.53 | 1.20 | 2.42 |
| is | 0.25 | 1.02 | 0.47 | 1.00 |
| fast | 0.65 | 2.59 | 1.16 | 2.59 |
Score[i][j] is how well token i’s query matches token j’s key: a dot product, divided by √d so the numbers stay in a range softmax handles well. Row “fast”, column “GPU” (outlined): (1.6·1.3 + 1.1·1.2 + 1.2·0.9 + 0.7·1) ÷ 2 = 2.59.
4 · Causal mask: no peeking at the future
| query ↓ key → | The | GPU | is | fast |
|---|---|---|---|---|
| The | 0.21 | −∞ | −∞ | −∞ |
| GPU | 0.59 | 2.53 | −∞ | −∞ |
| is | 0.25 | 1.02 | 0.47 | −∞ |
| fast | 0.65 | 2.59 | 1.16 | 2.59 |
Above the diagonal = later tokens, set to −∞.
A token may only look at itself and earlier tokens, so everything above the diagonal becomes −∞. This is what lets the model generate left to right: token 3’s result can never depend on token 4. It is also why earlier tokens’ keys and values never change later, which is the whole reason a KV cache works.
5 · Softmax each row into weights that sum to 1
| The | GPU | is | fast | |
|---|---|---|---|---|
| The | 0.21 | −∞ | −∞ | −∞ |
| GPU | 0.59 | 2.53 | −∞ | −∞ |
| is | 0.25 | 1.02 | 0.47 | −∞ |
| fast | 0.65 | 2.59 | 1.16 | 2.59 |
| The | GPU | is | fast | |
|---|---|---|---|---|
| The | 1.00 | −∞ | −∞ | −∞ |
| GPU | 0.13 | 0.87 | −∞ | −∞ |
| is | 0.23 | 0.49 | 0.28 | −∞ |
| fast | 0.06 | 0.42 | 0.10 | 0.42 |
Softmax makes each row positive and sum to 1 (−∞ becomes 0). Read row “fast”: it puts 0.42 on “GPU”, 0.42 on itself, and little on “The” and “is”. In a trained model this is how an adjective finds the noun it describes.
6 · Mix the values: output = weights × V
| The | GPU | is | fast | |
|---|---|---|---|---|
| The | 1.00 | −∞ | −∞ | −∞ |
| GPU | 0.13 | 0.87 | −∞ | −∞ |
| is | 0.23 | 0.49 | 0.28 | −∞ |
| fast | 0.06 | 0.42 | 0.10 | 0.42 |
| d1 | d2 | d3 | d4 | |
|---|---|---|---|---|
| The | 0.50 | 0.20 | 0.00 | 0.60 |
| GPU | 1.00 | 0.80 | 0.80 | 1.00 |
| is | 0.10 | 1.20 | 0.20 | 0.20 |
| fast | 1.20 | 0.40 | 0.90 | 1.90 |
| d1 | d2 | d3 | d4 | |
|---|---|---|---|---|
| The | 0.50 | 0.20 | 0.00 | 0.60 |
| GPU | 0.94 | 0.72 | 0.70 | 0.95 |
| is | 0.63 | 0.78 | 0.45 | 0.68 |
| fast | 0.96 | 0.64 | 0.73 | 1.27 |
Each output row is a weighted average of the value rows. Output “fast” ≈ 0.42 × V[“GPU”] + 0.42 × V[“fast”] + small parts of the rest: the token now carries information about the GPU. This is prefill: all four rows computed at once, as one big matrix multiply. The K and V matrices are now saved: that is the KV cache for this layer and head.
7 · Decode “!”: one new row, read the whole cache
| d1 | d2 | d3 | d4 | |
|---|---|---|---|---|
| ! | 0.40 | 0.90 | 0.90 | 1.40 |
| d1 | d2 | d3 | d4 | |
|---|---|---|---|---|
| The | 0.30 | 0.10 | 0.30 | 0.50 |
| GPU | 1.30 | 1.20 | 0.90 | 1.00 |
| is | 0.70 | 0.80 | 0.20 | 0.10 |
| fast | 0.90 | 1.10 | 1.40 | 1.20 |
| ! | 1.10 | 0.90 | 0.70 | 0.90 |
| The | GPU | is | fast | ! | |
|---|---|---|---|---|---|
| ! | 0.08 | 0.28 | 0.08 | 0.36 | 0.20 |
| d1 | d2 | d3 | d4 | |
|---|---|---|---|---|
| The | 0.50 | 0.20 | 0.00 | 0.60 |
| GPU | 1.00 | 0.80 | 0.80 | 1.00 |
| is | 0.10 | 1.20 | 0.20 | 0.20 |
| fast | 1.20 | 0.40 | 0.90 | 1.90 |
| ! | 0.90 | 1.60 | 0.10 | 1.30 |
| d1 | d2 | d3 | d4 | |
|---|---|---|---|---|
| ! | 0.94 | 0.81 | 0.59 | 1.29 |
The model has produced “!”. Now only that token is projected: one q, one k, one v. Its k and v are appended to the cache (highlighted row). Its q is scored against all 5 keys, and the output mixes all 5 values. The 4 old rows were not recomputed, only read. This one-row-at-a-time shape is decode.
8 · Decode “.”: the cache grows by one row per token
| d1 | d2 | d3 | d4 | |
|---|---|---|---|---|
| The | 0.30 | 0.10 | 0.30 | 0.50 |
| GPU | 1.30 | 1.20 | 0.90 | 1.00 |
| is | 0.70 | 0.80 | 0.20 | 0.10 |
| fast | 0.90 | 1.10 | 1.40 | 1.20 |
| ! | 1.10 | 0.90 | 0.70 | 0.90 |
| . | 0.40 | 0.70 | 1.30 | 1.00 |
| d1 | d2 | d3 | d4 | |
|---|---|---|---|---|
| The | 0.50 | 0.20 | 0.00 | 0.60 |
| GPU | 1.00 | 0.80 | 0.80 | 1.00 |
| is | 0.10 | 1.20 | 0.20 | 0.20 |
| fast | 1.20 | 0.40 | 0.90 | 1.90 |
| ! | 0.90 | 1.60 | 0.10 | 1.30 |
| . | 1.00 | 0.60 | 0.40 | 2.20 |
| The | GPU | is | fast | ! | . | |
|---|---|---|---|---|---|---|
| . | 0.07 | 0.23 | 0.07 | 0.29 | 0.17 | 0.19 |
Each step adds one K row and one V row, in every layer and every KV head. For gpt-oss-120b that is 36 layers × 8 KV heads × 64 numbers × 2 (K and V): 36,864 numbers, 72 KiB in 16-bit, per token, if every layer kept every token. Half its layers keep only the last 128 tokens, so long contexts cost about 36 KiB per token (worked out below). A 4,000-token prompt has written ~4,000 rows into each of those caches before decode starts; every decode step reads all of them. That read is why decode is bandwidth-bound, and those rows are what disaggregated serving ships between machines.
Toy sizes: d = 4, one head, no RoPE. gpt-oss-120b does the same with d = 64, 64 query heads, 8 KV heads and 36 layers.
Attention, step by step
Take one layer and one head. The input is the normed hidden state of each token, a vector of 2,880 numbers in gpt-oss-120b.
- Project. Multiply by three learned matrices to get a query , a key and a value , each 64 numbers (the head size,
head_dim). - Position. Apply RoPE to and : rotate them by angles that depend on the token’s position.
- Score. For token , compute the dot product of its query with the key of every token :
score[s] = q_t · k_s / sqrt(64). Dividing by the square root of the head size keeps scores in a range where softmax behaves. - Mask. Tokens after are not allowed (score set to minus infinity). This causal mask is why a model can generate left to right: no token’s output depends on the future.
- Normalize. Softmax turns the scores into weights that are positive and sum to 1.
- Mix. The head’s output is the weighted sum of the values:
out_t = Σ weight[s] · v_s.
In matrix form, for all tokens at once: softmax(Q·Kᵀ / sqrt(d) + mask) · V. With tokens, Q·Kᵀ is an score matrix: attention cost grows with the square of the sequence length during prefill.
Many heads
One head can only learn one kind of “look-up”. So the layer runs many heads side by side, each with its own projections. gpt-oss-120b has 64 query heads of size 64 per layer: the query projection maps 2,880 → 64 × 64 = 4,096 numbers. The 64 head outputs are concatenated (4,096 numbers) and an output matrix maps them back to 2,880, which is added to the residual stream.
Grouped-query attention: fewer keys and values
In the original design (multi-head attention, MHA) every query head has its own key and value head. Grouped-query attention (GQA) lets a group of query heads share one key/value head. gpt-oss-120b has 64 query heads and 8 KV heads: each K/V head serves 8 query heads. The extreme case, one K/V head for all, is multi-query attention (MQA).
GQA barely changes quality but cuts the keys and values that must be stored by the group factor, here 8×. Since storage of K and V is the whole problem below, almost every current model uses it. Llama 3.3 70B also has 8 KV heads (for 64 query heads of size 128).
Sliding windows and sinks (gpt-oss specifics)
- gpt-oss alternates two layer types (
layer_typesin the config): 18 sliding-window layers, where each token only attends to the previous 128 tokens (sliding_window: 128), and 18 full layers that see the whole context. - Each head has a learned extra term in the softmax denominator (the model card calls it similar to attention sinks). It lets a head put weight on “nothing” instead of being forced to spread weight over tokens. It costs no cache memory.
Why cache K and V
Look at step 3 again. To produce token 1,001, the model needs the keys and values of tokens 1 to 1,000 in every layer. Those vectors depend only on earlier tokens (causal mask), so once computed they never change.
Without a cache, every new token would rerun the whole sequence through all 36 layers: generating 1,000 tokens after a 4,000-token prompt would process about 4.5 million token-positions instead of 5,000. With a cache, each step computes Q, K and V for one new token, appends its K and V, and reads the stored ones. That stored set is the KV cache.
The trade: compute is saved, memory is spent. And reading that memory every step is what makes decode bandwidth-bound.
How big: the formula
For each token, each layer stores one key and one value per KV head:
Query heads do not appear: queries are used once and thrown away. That is why GQA shrinks the cache.
Try it: pick a model, a context length and a number of users.
gpt-oss-120b
From the config: 36 layers, 8 KV heads, head size 64. In 16-bit (2 bytes) per number:
- Per token per layer: 2 × 8 × 64 × 2 B = 2 KiB.
- Per token, all 36 layers: 72 KiB (73,728 bytes).
That is the simple answer, and it holds for contexts up to 128 tokens. For longer contexts the sliding-window layers matter: they only need the last 128 tokens. A server that exploits this (vLLM’s hybrid KV cache manager frees blocks that fall outside the window) stores:
full layers: 18 × 2 KiB = 36 KiB per token, for every token
sliding layers: 18 × 128 tokens × 2 KiB = 4.5 MiB per sequence, at most
| Context | Naive (all 36 layers) | Window-aware | Window-aware, FP8 KV |
|---|---|---|---|
| 1 token | 72 KiB | 72 KiB | 36 KiB |
| 4,085 tokens (a long prompt) | 287 MiB | 148 MiB | 74 MiB |
| 32,768 tokens | 2.25 GiB | 1.13 GiB | 0.57 GiB |
| 131,072 tokens (max) | 9 GiB | 4.5 GiB | 2.25 GiB |
| 100 users × 32,768 tokens | 225 GiB | 113 GiB | 57 GiB |
Which column applies depends on the server and its configuration: a server that does not free out-of-window blocks pays the naive figure.
Llama 3.3 70B, for contrast
80 layers, 8 KV heads, head size 128, 16-bit:
- Per token: 2 × 80 × 8 × 128 × 2 B = 320 KiB, 4.4× gpt-oss-120b’s naive figure.
- A 4,085-token prompt: about 1.25 GiB. A 32k context: 10 GiB. The full 131,072-token context: 40 GiB, for one user.
- Without GQA (64 KV heads), it would be 2.5 MiB per token, and a 32k context would need 80 GiB.
What fits on one MI355X
An MI355X has 288 GB of HBM. gpt-oss-120b’s weights take about 65 GB. vLLM by default uses 90% of HBM (gpu_memory_utilization=0.9, about 259 GB), and after weights, activations and buffers it reserves the rest for KV, roughly 150 GiB.
So one MI355X holds about 4 million tokens of gpt-oss-120b KV in 16-bit, window-aware: 150 GiB / 36 KiB ≈ 4.4 M (illustrative; the exact figure depends on the server’s buffers). With the naive 72 KiB per token it would be half that. 4 M tokens divided by the 131,072-token maximum context is about 32 users at full length, or about 1,000 users at 4k. With Llama 3.3 70B (141 GB of weights, 320 KiB per token) the same GPU would hold only a few hundred thousand tokens.
This is why, for long contexts, KV memory, not compute, limits how many users a server can hold at once, and why servers fight for every byte with paging, prefix sharing and KV quantization: see KV cache management.
Attention cost per step
Attention also costs compute and bandwidth that grow with context:
- FLOPs. Per layer per new token:
2 × query heads × head size × contextfor the scores plus the same for the weighted sum, so4 × 64 × 64 × context= 16,384 × context FLOPs in gpt-oss-120b. At a 32k context across the 18 full layers that is about 9.7 GFLOP per token: as much as all the model’s linear layers (about 9 GFLOP). - Bytes. In decode each step reads the sequence’s whole cache once: 1.13 GiB per token generated at 32k (window-aware). That is about 8 FLOPs per byte read. A GPU needs about 600 FLOPs per byte to be compute-bound (roofline), so decode attention is firmly bandwidth-bound, and batching users does not help: each sequence has its own KV.
FlashAttention and PagedAttention are the GPU kernels that do this efficiently: see Prefill and decode as kernels.
Common misconceptions
- “The KV cache is a cache of answers.” It is a cache of intermediate per-token vectors for this sequence. Two identical prompts share it only if a prefix cache is on.
- “More query heads means a bigger cache.” Only KV heads count. gpt-oss has 64 query heads and stores K/V for 8.
- “KV size depends on hidden size.” It depends on . gpt-oss’s hidden size is 2,880 but its K/V width per layer is 8 × 64 = 512.
- “Long contexts cost only memory.” They also cost bandwidth on every decode step and quadratic compute in prefill.
Further reading
- Attention Is All You Need (scaled dot-product and multi-head attention)
- GQA: training generalized multi-query transformer models · Multi-query attention
- Efficient streaming language models with attention sinks
- DeepSeek-V2 (multi-head latent attention: another way to shrink K/V, by compressing them into a small latent vector)
- gpt-oss model card · config.json
- vLLM hybrid KV cache manager (sliding-window plus full-attention models)
- The Illustrated Transformer (attention section)