Learn · LLM inference
Decode: generating tokens
After prefill has filled the KV cache, decode produces the answer one token at a time. Each decode step runs the full model on one new token per sequence.
The idea
Picture a clerk who must answer a letter one word at a time, and who works in a huge archive. For every single word they write, they must walk past every shelf of the rulebook (the model’s weights) and reread their own notes on this letter (the KV cache). Only then may they write the next word.
- The walk is the cost, not the thinking. Deciding the word takes a moment; walking the archive takes ages. On a GPU, decode spends its time reading bytes from memory, not doing arithmetic. That is what bandwidth-bound means.
- Words come strictly one after another. Word 5 depends on word 4, so a 500-word reply needs 500 walks.
- Batching is many clerks walking together. If 64 clerks each need the next word of their own letter, one walk past the rulebook serves all 64. That is why servers batch users.
Where the analogy breaks: each clerk’s notes are private, so 64 clerks still read 64 sets of notes, and the notes grow by one line per word. And in a mixture-of-experts model like gpt-oss, different clerks need different shelves (experts), so a bigger group walks more of the archive. Both effects show up in the numbers below.
- The token chosen in the last step is embedded and enters the first layer.
- In every layer, attention reads the K and V of every earlier token of this sequence from the cache. With a 4k context that is about 148 MiB per step.
- The new token’s own K and V are appended: one row per layer, 72 KiB in all.
- The last layer’s output becomes logits, the sampler picks one token, and it streams to the user.
- That token is the input of the next step. Repeat until an end token, a stop string or
max_tokens.
Worked example: one user, token by token
One user on one MI355X, a 4,085-token prompt already prefilled. The answer tokens are illustrative; every byte count is computed from the gpt-oss-120b config and checkpoint formats. Times are the bandwidth ceiling at ~8 TB/s (spec sheet, approximate): real steps are slower.
Decoding “The pod is pending” after a 4,085-token prompt
Step 1 · produce “The” (context 4,085)
- Bytes read
- 5.14 GB
- KV read
- 148 MiB
- 4,085 tokens
- KV written
- 72 KiB
- 1 row × 36 layers
- Step time ≥
- 0.64 ms
- bytes ÷ 8 TB/s
- FLOPs per byte
- 2.0
- ridge ≈ 625
The first decode step. The model's shared weights (violet: attention in every layer, indigo: the unembedding) are read in full: 3.08 GB for a single token. The arithmetic is only about 10.2 GFLOP, which the GPU could do in 2.0 µs. Reading the bytes takes at least 0.64 ms.
Step 2 · produce “pod” (context 4,086)
- Bytes read
- 5.14 GB
- KV read
- 148 MiB
- 4,086 tokens
- KV written
- 72 KiB
- 1 row × 36 layers
- Step time ≥
- 0.64 ms
- bytes ÷ 8 TB/s
- FLOPs per byte
- 2.0
- ridge ≈ 625
The router picks 4 of 128 experts in each layer, so only 1.90 GB of the ~61 GB of experts are read (orange). Another token may pick 4 different experts per layer; the bytes stay the same for one user.
Step 3 · produce “is” (context 4,087)
- Bytes read
- 5.14 GB
- KV read
- 148 MiB
- 4,087 tokens
- KV written
- 72 KiB
- 1 row × 36 layers
- Step time ≥
- 0.64 ms
- bytes ÷ 8 TB/s
- FLOPs per byte
- 2.0
- ridge ≈ 625
The KV part (blue) is this user's private history: 4,087 tokens in 18 full-attention layers plus 128 tokens in 18 sliding-window layers. It grows by one token per step, so it creeps up: 148 MiB at step 1, 148 MiB now.
Step 4 · produce “pending” (context 4,088)
- Bytes read
- 5.14 GB
- KV read
- 148 MiB
- 4,088 tokens
- KV written
- 72 KiB
- 1 row × 36 layers
- Step time ≥
- 0.64 ms
- bytes ÷ 8 TB/s
- FLOPs per byte
- 2.0
- ridge ≈ 625
Four tokens, four complete trips through memory: 20.56 GB read to produce four words. Speed per user is capped at about 1,556 tokens/s, and compute use is 0.32% of peak.
Step 1,000 · context 5,084
- Bytes read
- 5.18 GB
- KV read
- 183 MiB
- 5,084 tokens
- Step time ≥
- 0.65 ms
- Per user ≤
- 1,545 tok/s
A thousand tokens in, the context is 5,084 and the KV read has grown to 183 MiB. Weights still dominate for one user.
Step 28,683 · context 32,767
- Bytes read
- 6.20 GB
- KV read
- 1156 MiB
- 32,767 tokens
- Step time ≥
- 0.77 ms
- Per user ≤
- 1,291 tok/s
At a 32k context the KV alone is 1.21 GB per step, about 20% of the bytes. Long conversations decode measurably slower, even with nothing else on the GPU.
Bytes per step: attention weights 1.92 GB and unembedding 1.16 GB (16-bit, all read), 4 experts per layer × 36 layers (MXFP4), plus this sequence's KV cache (window-aware, 16-bit).
The latency metrics
- Inter-token latency (ITL): time between two consecutive output tokens of one request, about one decode step. Also called time per output token (TPOT; vLLM exports
vllm:inter_token_latency_secondsandvllm:request_time_per_output_token_seconds). - Output speed per user = 1 / ITL. 20 ms ITL is 50 tokens/s.
- End-to-end latency . A 500-token answer at 20 ms ITL spends 10 s in decode alone. For long answers and reasoning models, decode dominates.
- Throughput = tokens per second summed over all users of a server; it sets cost per token.
Why decode is bandwidth-bound
Per step and sequence, from the example: about 9 GFLOP against about 5.14 GB read. That is under 2 FLOPs per byte. An MI355X does about 5 PFLOP/s (dense FP8) but reads about 8 TB/s (spec sheet, approximate): its ridge point is about 625 FLOPs per byte (roofline). Decode sits 300× below it, so:
For one user on one MI355X that is about 0.64 ms, a ceiling of about 1,556 tokens/s. Real servers reach well below this (kernel launch gaps, imperfect bandwidth use, sampling, scheduling), but the ratio is what matters: the step costs bytes, not FLOPs. For a dense model it is worse: Llama 3.3 70B in 16-bit reads all 141 GB every step, at least 17.8 ms, so at most about 56 tokens/s per user on one MI355X, however idle its compute units are.
Batching: sharing the weight read
If 64 users each need a decode step, the server runs them together: one pass over the weights, each weight matrix multiplied by a 64-row matrix instead of a single vector. The weight bytes are paid once for all 64. Two things are not shared:
- KV cache: each sequence attends over its own history, so KV bytes grow linearly with the batch.
- Experts (MoE): different users route to different experts, so more experts are read as the batch grows (Mixture of experts).
Try it. Move the batch from 1 to 64 with gpt-oss-120b, then switch to Llama 3.3 70B:
The same model in a table, every user at a 4k context:
| Batch | gpt-oss experts read per layer | gpt-oss bytes per step | Step time | Total tokens/s | Per user | Llama 3.3 70B step time | Llama per user |
|---|---|---|---|---|---|---|---|
| 1 | ~4 | 5.14 GB | 0.64 ms | ~1,556 | ~1,556 | 17.79 ms | ~56 |
| 16 | ~51 | 29.85 GB | 3.73 ms | ~4,288 | ~268 | 20.31 ms | ~49 |
| 64 | ~111 | 66.01 GB | 8.25 ms | ~7,756 | ~121 | 28.36 ms | ~35 |
| 256 | ~128 | 103.88 GB | 12.98 ms | ~19,715 | ~77 | KV does not fit | — |
The pattern: batching multiplies throughput but lowers per-user speed. For the dense model, 64× the users costs only 1.6× the step time: batching is almost free. For MoE, batching first wakes up more experts, so per-user speed drops faster. At large batches the KV reads take over, and KV memory also caps how many sequences fit at all. Even at batch 256, compute use stays at a few percent of peak: on HBM GPUs decode is a bandwidth problem at every batch size people actually run.
Continuous batching
Requests start and finish at different times. Static batching waits for a full batch and runs it until the longest answer ends, leaving finished slots idle. Continuous batching (iteration-level scheduling, from the Orca system) re-forms the batch at every step: finished sequences leave, new ones join after their prefill. vLLM, SGLang and TensorRT-LLM all do this. Details: Batching and scheduling.
What makes decode faster
Decode speed per user is bounded by . On an MI355X the weights live in HBM at about 8 TB/s, so the example step above takes at least 0.64 ms (5.14 GB read). Every lever attacks one side of that fraction:
- Fewer bytes per token. Quantize weights (MXFP4 experts) and the KV cache (FP8); MoE already reads only the chosen experts.
- More tokens per byte read. Batch users so each weight read serves many sequences; speculative decoding verifies several tokens of one user per weight read.
- More bandwidth. Newer HBM generations, splitting the model across GPUs so each reads only its share, or accelerators that keep weights in on-chip SRAM.
- No interference. Running decode on its own pool, apart from prefill, keeps long prompts from stretching decode steps (Batching and scheduling).
Common misconceptions
- “More FLOPs make decode faster.” Not below the ridge point. Bandwidth (or keeping weights on-chip) does.
- “Batching is free throughput.” It costs per-user speed and KV memory; for MoE models it also costs extra weight reads.
- “Decode only reads the new token.” It reads the KV of every earlier token every step, so it slows as the context grows.
- “Tokens per second” is one number. Per-user speed and server throughput can move in opposite directions.
Further reading
- Efficient memory management for LLM serving with PagedAttention (batching limited by KV memory)
- vLLM metrics (ITL and TPOT histograms)
- Roofline model