Learn · LLM inference

Decode: generating tokens

Updated 10 Nov 2026 · 7 min read

After prefill has filled the KV cache, decode produces the answer one token at a time. Each decode step runs the full model on one new token per sequence.

The idea

Picture a clerk who must answer a letter one word at a time, and who works in a huge archive. For every single word they write, they must walk past every shelf of the rulebook (the model’s weights) and reread their own notes on this letter (the KV cache). Only then may they write the next word.

  • The walk is the cost, not the thinking. Deciding the word takes a moment; walking the archive takes ages. On a GPU, decode spends its time reading bytes from memory, not doing arithmetic. That is what bandwidth-bound means.
  • Words come strictly one after another. Word 5 depends on word 4, so a 500-word reply needs 500 walks.
  • Batching is many clerks walking together. If 64 clerks each need the next word of their own letter, one walk past the rulebook serves all 64. That is why servers batch users.

Where the analogy breaks: each clerk’s notes are private, so 64 clerks still read 64 sets of notes, and the notes grow by one line per word. And in a mixture-of-experts model like gpt-oss, different clerks need different shelves (experts), so a bigger group walks more of the archive. Both effects show up in the numbers below.

The decode loop
Last token of eachsequencebatch of B tokensWeights are shared by the batch; experts only partly (MoE).Forward pass, 36layersreads weights once for thebatchKV cacheeach sequence reads itsown historySampleone token per sequenceStream to usersread all K, Vappend 1 K, V per layerlogitsnext step
  1. The token chosen in the last step is embedded and enters the first layer.
  2. In every layer, attention reads the K and V of every earlier token of this sequence from the cache. With a 4k context that is about 148 MiB per step.
  3. The new token’s own K and V are appended: one row per layer, 72 KiB in all.
  4. The last layer’s output becomes logits, the sampler picks one token, and it streams to the user.
  5. That token is the input of the next step. Repeat until an end token, a stop string or max_tokens.
Each step reads the weights it needs and the whole KV cache of every sequence in the batch, and writes one new K and V per layer per sequence.

Worked example: one user, token by token

One user on one MI355X, a 4,085-token prompt already prefilled. The answer tokens are illustrative; every byte count is computed from the gpt-oss-120b config and checkpoint formats. Times are the bandwidth ceiling at ~8 TB/s (spec sheet, approximate): real steps are slower.

Decoding “The pod is pending” after a 4,085-token prompt

Step 1 · produce “The” (context 4,085)

Bytes readattention W4 experts × 36unembed0123456 GB
Bytes read
5.14 GB
KV read
148 MiB
4,085 tokens
KV written
72 KiB
1 row × 36 layers
Step time ≥
0.64 ms
bytes ÷ 8 TB/s
FLOPs per byte
2.0
ridge ≈ 625

The first decode step. The model's shared weights (violet: attention in every layer, indigo: the unembedding) are read in full: 3.08 GB for a single token. The arithmetic is only about 10.2 GFLOP, which the GPU could do in 2.0 µs. Reading the bytes takes at least 0.64 ms.

Step 2 · produce “pod” (context 4,086)

Bytes readattention W4 experts × 36unembed0123456 GB
Bytes read
5.14 GB
KV read
148 MiB
4,086 tokens
KV written
72 KiB
1 row × 36 layers
Step time ≥
0.64 ms
bytes ÷ 8 TB/s
FLOPs per byte
2.0
ridge ≈ 625

The router picks 4 of 128 experts in each layer, so only 1.90 GB of the ~61 GB of experts are read (orange). Another token may pick 4 different experts per layer; the bytes stay the same for one user.

Step 3 · produce “is” (context 4,087)

Bytes readattention W4 experts × 36unembed0123456 GB
Bytes read
5.14 GB
KV read
148 MiB
4,087 tokens
KV written
72 KiB
1 row × 36 layers
Step time ≥
0.64 ms
bytes ÷ 8 TB/s
FLOPs per byte
2.0
ridge ≈ 625

The KV part (blue) is this user's private history: 4,087 tokens in 18 full-attention layers plus 128 tokens in 18 sliding-window layers. It grows by one token per step, so it creeps up: 148 MiB at step 1, 148 MiB now.

Step 4 · produce “pending” (context 4,088)

Bytes readattention W4 experts × 36unembed0123456 GB
Bytes read
5.14 GB
KV read
148 MiB
4,088 tokens
KV written
72 KiB
1 row × 36 layers
Step time ≥
0.64 ms
bytes ÷ 8 TB/s
FLOPs per byte
2.0
ridge ≈ 625

Four tokens, four complete trips through memory: 20.56 GB read to produce four words. Speed per user is capped at about 1,556 tokens/s, and compute use is 0.32% of peak.

Step 1,000 · context 5,084

Bytes readattention W4 experts × 36unembed0123456 GB
Bytes read
5.18 GB
KV read
183 MiB
5,084 tokens
Step time ≥
0.65 ms
Per user ≤
1,545 tok/s

A thousand tokens in, the context is 5,084 and the KV read has grown to 183 MiB. Weights still dominate for one user.

Step 28,683 · context 32,767

Bytes readattention W4 experts × 36unembedKV 1156 MiB0123456 GB
Bytes read
6.20 GB
KV read
1156 MiB
32,767 tokens
Step time ≥
0.77 ms
Per user ≤
1,291 tok/s

At a 32k context the KV alone is 1.21 GB per step, about 20% of the bytes. Long conversations decode measurably slower, even with nothing else on the GPU.

Bytes per step: attention weights 1.92 GB and unembedding 1.16 GB (16-bit, all read), 4 experts per layer × 36 layers (MXFP4), plus this sequence's KV cache (window-aware, 16-bit).

The latency metrics

  • Inter-token latency (ITL): time between two consecutive output tokens of one request, about one decode step. Also called time per output token (TPOT; vLLM exports vllm:inter_token_latency_seconds and vllm:request_time_per_output_token_seconds).
  • Output speed per user = 1 / ITL. 20 ms ITL is 50 tokens/s.
  • End-to-end latency ≈TTFT+(output tokens−1)×ITL\approx \text{TTFT} + (\text{output tokens} - 1) \times \text{ITL}. A 500-token answer at 20 ms ITL spends 10 s in decode alone. For long answers and reasoning models, decode dominates.
  • Throughput = tokens per second summed over all users of a server; it sets cost per token.

Why decode is bandwidth-bound

Per step and sequence, from the example: about 9 GFLOP against about 5.14 GB read. That is under 2 FLOPs per byte. An MI355X does about 5 PFLOP/s (dense FP8) but reads about 8 TB/s (spec sheet, approximate): its ridge point is about 625 FLOPs per byte (roofline). Decode sits 300× below it, so:

step time  ≥  bytes read per stepmemory bandwidth\text{step time} ;\ge; \frac{\text{bytes read per step}}{\text{memory bandwidth}}

For one user on one MI355X that is about 0.64 ms, a ceiling of about 1,556 tokens/s. Real servers reach well below this (kernel launch gaps, imperfect bandwidth use, sampling, scheduling), but the ratio is what matters: the step costs bytes, not FLOPs. For a dense model it is worse: Llama 3.3 70B in 16-bit reads all 141 GB every step, at least 17.8 ms, so at most about 56 tokens/s per user on one MI355X, however idle its compute units are.

Batching: sharing the weight read

If 64 users each need a decode step, the server runs them together: one pass over the weights, each weight matrix multiplied by a 64-row matrix instead of a single vector. The weight bytes are paid once for all 64. Two things are not shared:

  • KV cache: each sequence attends over its own history, so KV bytes grow linearly with the batch.
  • Experts (MoE): different users route to different experts, so more experts are read as the batch grows (Mixture of experts).

Try it. Move the batch from 1 to 64 with gpt-oss-120b, then switch to Llama 3.3 70B:

Decode speed ceiling on one MI355X
Bytes read per step
Step time (≈ ITL)
Per user
Whole server
FLOPs per byte (ridge ~625)
Matrix units busy

The same model in a table, every user at a 4k context:

Batchgpt-oss experts read per layergpt-oss bytes per stepStep timeTotal tokens/sPer userLlama 3.3 70B step timeLlama per user
1~45.14 GB0.64 ms~1,556~1,55617.79 ms~56
16~5129.85 GB3.73 ms~4,288~26820.31 ms~49
64~11166.01 GB8.25 ms~7,756~12128.36 ms~35
256~128103.88 GB12.98 ms~19,715~77KV does not fit—
Bytes per step for gpt-oss-120b, by batch (GB)
batch 1batch 16experts 24.28 GBbatch 64experts 52.96 GBKV 9.97 GBbatch 256experts 60.94 GBKV 39.86 GB020406080100 GB

Violet: shared weights. Orange: experts woken by the batch. Blue: every user's own KV.

The pattern: batching multiplies throughput but lowers per-user speed. For the dense model, 64× the users costs only 1.6× the step time: batching is almost free. For MoE, batching first wakes up more experts, so per-user speed drops faster. At large batches the KV reads take over, and KV memory also caps how many sequences fit at all. Even at batch 256, compute use stays at a few percent of peak: on HBM GPUs decode is a bandwidth problem at every batch size people actually run.

Continuous batching

Requests start and finish at different times. Static batching waits for a full batch and runs it until the longest answer ends, leaving finished slots idle. Continuous batching (iteration-level scheduling, from the Orca system) re-forms the batch at every step: finished sequences leave, new ones join after their prefill. vLLM, SGLang and TensorRT-LLM all do this. Details: Batching and scheduling.

What makes decode faster

Decode speed per user is bounded by bandwidth/bytes per token\text{bandwidth} / \text{bytes per token}. On an MI355X the weights live in HBM at about 8 TB/s, so the example step above takes at least 0.64 ms (5.14 GB read). Every lever attacks one side of that fraction:

  • Fewer bytes per token. Quantize weights (MXFP4 experts) and the KV cache (FP8); MoE already reads only the chosen experts.
  • More tokens per byte read. Batch users so each weight read serves many sequences; speculative decoding verifies several tokens of one user per weight read.
  • More bandwidth. Newer HBM generations, splitting the model across GPUs so each reads only its share, or accelerators that keep weights in on-chip SRAM.
  • No interference. Running decode on its own pool, apart from prefill, keeps long prompts from stretching decode steps (Batching and scheduling).

Common misconceptions

  • “More FLOPs make decode faster.” Not below the ridge point. Bandwidth (or keeping weights on-chip) does.
  • “Batching is free throughput.” It costs per-user speed and KV memory; for MoE models it also costs extra weight reads.
  • “Decode only reads the new token.” It reads the KV of every earlier token every step, so it slows as the context grows.
  • “Tokens per second” is one number. Per-user speed and server throughput can move in opposite directions.

Further reading