Learn · LLM inference

Prefill: reading the prompt

Updated 10 Nov 2026 · 7 min read

Prefill is the first forward pass of a request: the whole prompt goes through the model at once. It runs once per request, before decode generates the answer. In a disaggregated server, prefill and decode even run on separate pools of machines.

The idea

Think of a support engineer handed a long ticket. Before replying, they read the whole ticket once and jot a margin note next to every line (“this is the error”, “this is the node name”). Then they write the answer, glancing at the notes instead of rereading the ticket.

  • Reading the whole ticket at once is prefill. Every prompt token is known up front, so nothing forces the model to go one token at a time: it pushes all of them through each layer together.
  • The margin notes are the KV cache: a key and a value for every prompt token in every layer. Decode reads these notes at every step instead of recomputing the prompt.
  • The first word of the reply is the first output token, predicted from the last prompt position. The wait until it appears is time to first token (TTFT).

Where the analogy breaks: a person reads a short ticket faster than a long one. A GPU barely does. Much of its cost is reading the model’s weights from memory, and that happens once per layer whether the prompt is 30 tokens or 3,000. So prefill is efficient only when many tokens share each weight read. That is the performance story of this page.

What prefill is for
Whole promptevery token known upfrontLike reading a whole page at once instead of word by word: each layer handles every prompt token in one big matrix multiply.One pass, all tokenstogetherweights read once, usedby allThe notes prefill leaves behind, so decode never has to reread the prompt.KV cachea key and value per tokenper layerFirst output tokenfrom the last positionDecodesame GPU or a separatepoolwritespredictsread every step
  1. The prompt arrives complete: all its tokens are known.
  2. All prompt tokens go through each of the 36 layers together, as one matrix.
  3. Each layer writes the K and V of every prompt token: the notes decode will need.
  4. The last position’s output picks the first new token.
  5. Decode takes over and reads the KV cache every step.
Prefill does two jobs: fill the KV cache and predict the first output token.

One prefill, worked through

An illustrative request on one MI355X: a 2,760-token prompt whose first 256 tokens (two 128-token blocks: the shared start of a chat template) are already in the prefix cache. Only 2,504 tokens must be computed.

Linear layers
22.5 TFLOP
2,504 tokens × 9 GFLOP
Attention
1.2 TFLOP
new tokens × context seen
At peak (5 PFLOP/s)
4.7 ms
At 30% of peak
15.8 ms
a more realistic share
  1. Wait. The engine looks up the prefix cache and schedules the request into the next step. If another request is mid-prefill, this is where the time goes.
  2. Forward pass. All 2,504 new tokens go through 36 layers together and attend to the 256 cached ones.
  3. KV written. Every layer stores the K and V of every new token: about 102 MiB for the whole prompt (window-aware, 16-bit). A disaggregated server ships this to the decode side, usually layer by layer while later layers still compute.
  4. First token. The last position’s logits pick the first output token, and decode takes over.

A 30-token prompt costs far less math but not proportionally less time: that near-fixed floor is explained in the small-prompt floor.

Why prefill is one big parallel pass

All nn prompt tokens move through each layer together, as an n×2,880n \times 2{,}880 matrix:

  • each weight matrix is read from memory once per layer and used for all nn tokens: matrix × matrix multiplies (GEMMs);
  • attention computes an n×nn \times n score matrix per head (FlashAttention does it tile by tile without storing it; see kernels). The causal mask stops each position from seeing later ones, so the parallel pass gives the same result as going one token at a time.

Reusing each weight byte for thousands of tokens is what makes prefill compute-bound on a GPU, unlike decode.

The compute cost, worked out

prefill FLOPs  ≈  2×active non-embedding parameters×new tokens(linear layers)+4×query heads×head size×∑context seen(attention, per layer)\begin{aligned} \text{prefill FLOPs} ;\approx;& 2 \times \text{active non-embedding parameters} \times \text{new tokens} && \text{(linear layers)} \ &+ 4 \times \text{query heads} \times \text{head size} \times \textstyle\sum \text{context seen} && \text{(attention, per layer)} \end{aligned}

For gpt-oss-120b (details): about 9 GFLOP per token in linear layers, plus 4 × 64 × 64 = 16,384 FLOPs per token per unit of context per layer. In the 18 full layers a causal prompt of nn tokens sees about n2/2n^2/2 tokens in total; in the 18 sliding layers at most 128 per token. Try it:

Prefill cost calculator
Tokens to compute
Engine steps (chunks)
Linear FLOPs
Attention FLOPs
Time at 100% of 5 PFLOP/s
Time at chosen MFU
KV to push (16-bit)

What to read from it:

  • Linear cost grows with nn; attention with n2n^2. At 4k tokens attention is about 6% of the work; at 32k about a third; at 131k it dominates.
  • Real speed is well below spec. Long prompts typically reach a modest fraction of peak. Part of the gap is real (attention runs in 16-bit, whose peak is half; MoE routing and kernel launches add overhead), part is that peaks are never reached. The share of peak a workload uses is MFU, model FLOPs utilization.
  • Dense models cost far more. Llama 3.3 70B needs about 140 GFLOP per token, so a 4k prompt is about 570 TFLOP before attention.

The small-prompt floor

The formula says a 200-token prompt needs under 2 TFLOP: under a millisecond at peak. In practice a short prefill takes tens of milliseconds. For an MoE model a short prompt gives each expert only a handful of tokens, so the step is limited by reading the weights (about 61 GB of experts at ~8 TB/s ≈ 7.6 ms), plus fixed per-request overhead (scheduling, kernel launches, sampling). See Mixture of experts. Prefill is compute-bound only once a step holds a few thousand tokens.

TTFT: what the user waits for

TTFT  ≈  queue wait+prefill compute+first token’s sampling+network\text{TTFT} ;\approx; \text{queue wait} + \text{prefill compute} + \text{first token's sampling} + \text{network}

In a disaggregated server it also includes shipping the KV to the decode side and the first decode step. vLLM exports the parts it can see: vllm:request_queue_time_seconds, vllm:request_prefill_time_seconds and vllm:time_to_first_token_seconds. The user-visible TTFT adds the network and anything in front of the engine.

Chunked prefill: a 30,000-token prompt in four steps

A 100k-token prompt in one step would occupy the GPU for seconds and need a lot of activation memory. Chunked prefill caps the tokens per engine step (vLLM’s max_num_batched_tokens). Each chunk computes Q, K, V for its tokens, attends to its own tokens and to the KV of all earlier chunks, and appends its K and V. The result is identical to one big pass. With a budget of 8,192 tokens:

Chunked prefill of 30,000 tokens (budget 8,192)

1 · Chunk 1: tokens 1–8,192

Promptchunk 1: 8,19208,19216,38424,57630,000 tokens
New tokensContext attendedLinearAttentionAttention shareTime at peak
8,192up to 8,19273.7 TFLOP10.2 TFLOP12%16.8 ms

The first chunk attends only to itself. Its K and V go into the cache (blue in the next steps).

2 · Chunk 2: tokens 8,193–16,384

Promptin KV cachechunk 2: 8,19208,19216,38424,57630,000 tokens
New tokensContext attendedLinearAttentionAttention shareTime at peak
8,192up to 16,38473.7 TFLOP30.0 TFLOP29%20.7 ms

Same 8,192 new tokens, but they also attend to the 8,192 cached ones, so attention work rises chunk by chunk while linear work stays flat.

3 · Chunk 3: tokens 16,385–24,576

Promptin KV cachein KV cachechunk 3: 8,19208,19216,38424,57630,000 tokens
New tokensContext attendedLinearAttentionAttention shareTime at peak
8,192up to 24,57673.7 TFLOP49.8 TFLOP40%24.7 ms

Same 8,192 new tokens, but they also attend to the 16,384 cached ones, so attention work rises chunk by chunk while linear work stays flat.

4 · Chunk 4: tokens 24,577–30,000

Promptin KV cachein KV cachein KV cachechunk 4: 5,42408,19216,38424,57630,000 tokens
New tokensContext attendedLinearAttentionAttention shareTime at peak
5,424up to 30,00048.8 TFLOP43.9 TFLOP47%18.5 ms

The last chunk is the remainder: 8,192 × 3 + 5,424 = 30,000. Fewer new tokens, the longest context.

FLOPs from the formula above, gpt-oss-120b. Times are at spec-sheet peak; real steps are slower.

In total: 403.8 TFLOP, 80.8 ms at peak. The cost of chunking: later chunks are slower because they attend to more context, and total work rises slightly. The gain: each step is bounded, so a long prompt cannot freeze the engine.

Head-of-line blocking

The scheduler fills each step’s token budget in arrival order. Short requests that arrive behind a chunked long prompt wait: a 200-token prompt that needs a few milliseconds of compute can wait hundreds of milliseconds while the long prompt’s chunks go first, and then finish in the same step as the long prompt’s last chunk. Chunking bounds each step; it does not let a short request jump the queue. Mitigations are policy: a smaller budget, priority for short prompts, or separate replicas for long prompts (Batching and scheduling).

In an aggregated server chunked prefill matters even more: vLLM schedules running decodes first and fills the rest of the step with prefill chunks, so a smaller budget protects decode latency and a larger one improves TTFT (vLLM optimization guide).

Prefix caching cuts the work

Only uncached tokens are computed. With chat templates, system prompts and multi-turn conversations, a large share of prompt tokens can come from the prefix cache (vLLM counts them in vllm:prompt_tokens_by_source_total). Cached tokens still cost attention, because new tokens attend to them, but no linear-layer work. Mechanism and counters: KV cache management.

Common misconceptions

  • “Prefill is cheap because it is just reading.” It is most of the FLOPs of a typical chat request: a 2,700-token prompt costs ~25 TFLOP, while generating 300 tokens costs about 3 TFLOP of linear work (but much more memory traffic).
  • “TTFT is proportional to prompt length.” Only above a few thousand tokens. Below that a near-fixed floor dominates, and cached tokens are nearly free.
  • “Chunked prefill makes prefill faster.” It makes each step shorter and memory-bounded; total work slightly increases.

Further reading