Learn · Overview
LLM inference from the model's side (overview)
This track explains what a large language model computes when it serves a request, mechanism by mechanism, and why that shapes how inference servers are built: prefill is compute-bound, decode is bandwidth-bound, and large deployments even run them on separate machines. The running example is gpt-oss-120b, an open-weight mixture-of-experts model, with numbers from its published config.
The idea first
An LLM is a next-token guesser called in a loop. Give it text, it returns a probability for every possible next token (a token is a word piece). Pick one, append it, ask again. Everything a server does is about running that loop fast for many users at once.
Reading versus writing. Think of answering an exam question. First you read the whole question: you can take in all of it at once, in parallel. Then you write the answer one word at a time, and each word depends on the words before it. A model does the same:
| Phase | What it does | Question it answers | What limits it | Where it lives |
|---|---|---|---|---|
| Prefill (reading) | runs all prompt tokens through the model in one pass | “What does the prompt mean, and what is the first token?” | arithmetic (lots of math per weight read) | GPU compute units |
| KV cache (the notes) | keeps every token’s keys and values from every layer | “What do I need to remember so I never re-read the prompt?” | memory size | GPU memory (HBM) |
| Decode (writing) | one new token per step, re-reading all the notes | “Given everything so far, what comes next?” | memory bandwidth (little math per byte read) | GPU memory bandwidth |
The notes are the key idea. While reading, the model writes notes about every token (in every layer). When writing, each new word consults all the notes instead of re-reading the question. Those notes, the KV cache, are what fills GPU memory, and, in a disaggregated server, what is shipped over the network from the machine that read the prompt to the one that writes the answer.
Where the analogy breaks. The notes are not a summary: they grow by one entry per token, per layer, and are never compressed. “Reading” is not cheap: for a typical chat request it is most of the arithmetic. And “writing” is slow not because of thinking but because every word means re-fetching the model’s weights and all the notes from memory.
- Read the promptThe whole prompt (here 4,085 tokens) enters prefill at once.
- Write the notesPrefill computes keys and values for every token in every layer and stores them: the KV cache. A disaggregated server sends this cache to a separate decode machine, often while prefill is still running.
- Write one tokenDecode makes the first output token by reading the whole cache.
- Add to the notes, repeatEach new token adds its own keys and values to the cache, and the loop runs again until an end token or the length limit.
One request, end to end: prefill with numbers
An illustrative request on one MI355X, followed step by step. Every number is computed from the gpt-oss-120b config; times are at spec-sheet peak and at an assumed 30% of it.
A 2,760-token prompt through prefill
1 · The prompt arrives as tokens
- Prompt
- 2,760 tokens
- chat template + history + question
- Model
- gpt-oss-120b
- one MI355X, 36 layers
The text was turned into integer token ids upstream (how: Tokens and transformers). From here on, only token counts matter.
2 · Prefix cache: which 128-token blocks are already known?
| 128 | 128 | 128 | 128 | 128 | 128 | 128 | 128 | 128 | 128 | 128 |
| 128 | 128 | 128 | 128 | 128 | 128 | 128 | 128 | 128 | 128 | 72 |
The server hashes the prompt block by block from the start and finds the first 2 blocks already in its cache (256 tokens), for example a shared system prompt. The rest, 2,504 tokens, must be computed. Only whole blocks match, so a hit is always a multiple of 128. Mechanism: KV cache management.
3 · Wait for a slot
The engine schedules the request into the next step, within the token budget and free KV memory (Batching and scheduling). On an idle server this takes a few milliseconds; behind a long prompt it can take hundreds.
4 · One forward pass: 24.0 TFLOP
- Math for 2,504 new tokens
- 24.0 TFLOP
- ~9 GFLOP per token + attention
- At MI355X FP8 peak
- 4.8 ms
- 5 PFLOP/s, approximate
- At 30% of peak
- 16.0 ms
All 2,504 new tokens go through the 36 layers together, in one engine step. The GPU is doing big matrix multiplies, yet a step of this size reaches only part of its peak: it still spends a large share of its time reading the ~61 GB of expert weights (7.6 ms at 8 TB/s), plus fixed overhead (Prefill).
5 · The notes: 101.5 MiB of KV
- KV for 2,760 tokens
- 101.5 MiB
- 16-bit, window-aware
- If every layer kept every token
- 194.1 MiB
The KV covers the whole prompt, cached blocks included: decode needs all of it. In a disaggregated server it is sent to the decode side layer by layer as each layer finishes, so the transfer overlaps the forward pass.
6 · Done: the first token
The prefill step samples the first output token. With speculative decoding, a small draft model also processes the prompt. From here decode runs, one token per step, at a speed set by memory bandwidth (Decode).
Prefill side only. Illustrative prompt; FLOPs and bytes from the published model config.
Try it: where the time goes
Change the prompt, the cache hit and the answer length. Short answers to long prompts are prefill-heavy; long answers are decode-heavy, which is why the two phases are sized separately.
The ideas in order
- Tokens and layers. Text becomes integer tokens (
kubectlis two). Each token becomes a vector of 2,880 numbers, and 36 transformer layers update it: RMSNorm, attention with RoPE, a mixture-of-experts MLP, residual adds. The last vector picks the next token. Almost all of it is matrix multiplies, about 9 GFLOP per token. → Tokens and transformers - Mixture of experts. Each layer has 128 expert MLPs; a router sends each token to 4. So gpt-oss has 117 B parameters to store but uses 5.1 B per token to compute. Batching helps less than for dense models because different tokens wake different experts. → Mixture of experts
- Attention and the KV cache. Attention is where tokens see each other through queries, keys and values. Keys and values of past tokens never change, so the server keeps them: per token, 36–72 KiB for gpt-oss-120b, 320 KiB for Llama 3.3 70B. → Attention and the KV cache
- KV cache management. The cache is stored in 128-token blocks with block tables (paging), shared across requests with the same prefix, offloaded to CPU memory, optionally quantized to FP8. → KV cache management
- Prefill. The whole prompt in one pass: fills the KV cache, sets time to first token, compute-bound once a step holds a few thousand tokens. Long prompts are chunked, for example at 8,192 tokens per step. → Prefill
- Decode. One token per sequence per step, re-reading weights and KV each time: about 2 FLOPs per byte, so bandwidth-bound on any HBM GPU. Sets inter-token latency. → Decode
- Batching and scheduling. Each step the scheduler picks which requests run within a token budget and the free KV memory. Continuous batching re-forms the batch every step. Separating prefill and decode is itself a scheduling decision. → Batching and scheduling
- Speculative decoding. A small draft guesses several tokens; the target checks them in one pass and keeps the ones it agrees with, with the same output. A disaggregated server then needs a draft prefill as well as a target prefill. → Speculative decoding
- Kernels. On the GPU, prefill becomes big GEMMs and FlashAttention; decode becomes skinny GEMMs and paged attention. → Prefill and decode as kernels
- Hardware fit. On the roofline, prefill sits at a GPU’s compute roof; decode sits far below its ridge point, limited by memory bandwidth. That is why the phases are sized, and sometimes placed, separately.
Words you will see in logs and dashboards
| Term | Meaning | Where it shows |
|---|---|---|
| context tokens | prompt tokens processed by prefill in a step | vLLM Iteration(...) log line |
| generation tokens | tokens produced by decode in a step | same line; always 0 on a prefill-only server |
| prompt tokens by source | prompt tokens served from the prefix cache / computed | vllm:prompt_tokens_by_source_total |
| KV cache usage | share of reserved KV blocks in use | GPU KV cache usage, vllm:kv_cache_usage_perc |
| TTFT | time to first token | vllm:time_to_first_token_seconds |
| ITL / TPOT | inter-token latency / time per output token | vllm:inter_token_latency_seconds |
| batch | requests processed together in one step | vllm:num_requests_running |
| preemption | a running request evicted to free KV memory | vllm:num_preemptions |
| acceptance rate | share of draft tokens the target keeps | decode side |
Further reading
- gpt-oss model card · config.json
- vLLM documentation
- Efficient memory management for LLM serving with PagedAttention
- DistServe (prefill/decode disaggregation)