Learn · LLM inference
Speculative decoding: draft and target
Decode makes one token per step, and each step is limited by reading weights and KV from memory, not by arithmetic. So a decode step has plenty of spare compute. Speculative decoding spends that spare compute to make several tokens per step, without changing what the model outputs.
The idea
A junior drafts, a senior reviews. Writing is slow because the senior (the big target model) has to “open the whole file” for every single word: one full read of its weights per token. Reading a few proposed words costs the senior almost the same as writing one, because opening the file is the expensive part. So let a fast junior (a small draft model) write the next few words, and have the senior check them all in one sitting. The senior keeps the words it would have written itself and replaces the first one it disagrees with.
| Piece | What it is for | The question it answers |
|---|---|---|
| Draft model | cheap guesses | “What are the next tokens, probably?” |
| One target pass over positions | the check, at the price of one decode step | “What would the target have said at each position?” |
| Accept/reject rule | keep the target’s answer exactly | “Do I keep this guess?” |
| Acceptance rate | how often guesses survive | “How much faster will this be?” |
Why checking is cheap. A forward pass predicts the next token at every input position at once (that is how prefill works). Feed the target the current token plus the guesses and it returns its own predictions after each prefix, in one pass that reads the weights once.
Where the analogy breaks. The review is not a judgement call: it is exact arithmetic on probabilities, so the output is exactly what the target alone would produce (same distribution). And one rejection throws away every guess after it, because those guesses were built on a word that is no longer there.
Worked example: one round, real numbers
A toy vocabulary of 9 tokens and made-up probabilities (a real model has about 200,000 tokens and learns its probabilities). The text so far is “kubectl reports that”; the draft proposes = 4 tokens. Every number below is computed from the distributions; the random numbers come from a fixed seed.
Draft 4 tokens, verify in one pass
1 · The cache holds the context
Both models already have the KV cache of “kubectl reports that” (from prefill or earlier rounds). Plain decoding would now run the target once per token. Instead, the draft goes first.
2 · Draft: 4 cheap steps
| the | a | pod | node | is | was | ready | running | . | |
|---|---|---|---|---|---|---|---|---|---|
| q₁ | 0.70 | 0.20 | 0.01 | 0.01 | 0.01 | 0.01 | 0.01 | 0.01 | 0.01 |
| q₂ | 0.01 | 0.01 | 0.60 | 0.30 | 0.01 | 0.01 | 0.01 | 0.01 | 0.01 |
| q₃ | 0.01 | 0.01 | 0.01 | 0.01 | 0.55 | 0.35 | 0.01 | 0.01 | 0.01 |
| q₄ | 0.01 | 0.01 | 0.01 | 0.01 | 0.01 | 0.01 | 0.60 | 0.30 | 0.01 |
The draft runs 4 ordinary decode steps of its own and picks “the”, “pod”, “is”, “ready”. Each step appends a KV entry in the draft’s cache. Being small, it is fast: here we assume each draft step costs 5% of a target step.
3 · Verify: one target pass over 5 positions
| the | a | pod | node | is | was | ready | running | . | |
|---|---|---|---|---|---|---|---|---|---|
| after “that” | 0.62 | 0.25 | 0.02 | 0.02 | 0.02 | 0.02 | 0.02 | 0.02 | 0.02 |
| after “the” | 0.01 | 0.01 | 0.55 | 0.35 | 0.01 | 0.01 | 0.01 | 0.01 | 0.01 |
| after “pod” | 0.02 | 0.02 | 0.02 | 0.02 | 0.30 | 0.48 | 0.06 | 0.08 | 0.02 |
| after “is” | 0.01 | 0.01 | 0.01 | 0.01 | 0.01 | 0.01 | 0.50 | 0.40 | 0.01 |
| after “ready” | 0.04 | 0.04 | 0.04 | 0.04 | 0.04 | 0.04 | 0.04 | 0.04 | 0.70 |
The target runs once over “that the pod is ready” and returns 5 distributions: what it would say after each prefix. Row 5 (after “ready”) is only used if all 4 guesses survive. The target’s cache now has 4 new KV entries too, one per drafted token.
4 · Greedy rule: keep guesses that match the target's top token
| Position | Draft guessed | Target’s top token | Verdict |
|---|---|---|---|
| 1 | the | the | accept |
| 2 | pod | pod | accept |
| 3 | is | was | reject: emit “was” |
| 4 | ready | ready | discarded (after a reject) |
With greedy decoding (always take the most likely token) the rule is a comparison. “the” and “pod” match. At position 3 the target prefers “was” (0.48) over the draft’s “is” (0.30): reject, and emit the target’s “was” instead. Position 4 is never looked at, even though “ready” happens to match. 3 tokens from one target pass instead of one.
5 · Sampling rule: accept with probability min(1, p/q)
| Position | Draft token | p | q | min(1, p/q) | random u | u < min? |
|---|---|---|---|---|---|---|
| 1 | the | 0.62 | 0.7 | 0.89 | 0.63 | accept |
| 2 | pod | 0.55 | 0.6 | 0.92 | 0.14 | accept |
| 3 | is | 0.3 | 0.55 | 0.55 | 0.75 | reject |
When the server samples (temperature above 0), a guess is kept with probability min(1, p/q): always if the target likes it at least as much as the draft did, otherwise in proportion. “the”: the target gives 0.62 where the draft gave 0.70, so it survives with chance 0.89; u = 0.63 survives. Position 3: “is” survives with chance only 0.55; u = 0.75 rejects it.
6 · On reject: sample from what the draft under-predicted
| the | a | pod | node | is | was | ready | running | . | |
|---|---|---|---|---|---|---|---|---|---|
| p₃ target | 0.02 | 0.02 | 0.02 | 0.02 | 0.30 | 0.48 | 0.06 | 0.08 | 0.02 |
| q₃ draft | 0.01 | 0.01 | 0.01 | 0.01 | 0.55 | 0.35 | 0.01 | 0.01 | 0.01 |
| max(0, p − q) | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.13 | 0.05 | 0.07 | 0.00 |
| r = normalized | 0.01 | 0.01 | 0.01 | 0.01 | 0.00 | 0.52 | 0.18 | 0.26 | 0.01 |
The replacement is drawn from the residual: the probability the target wanted but the draft did not cover. “was” holds 0.52 of it; u = 0.39 picks “was”. This correction is what makes the output distribution exactly the target’s. Then clean-up: KV entries for “is” and “ready” are freed in both caches (✕). The KV for “was” is written when the next round feeds it in.
7 · Check: the output really is the target's distribution
| the | a | pod | node | is | was | ready | running | . | |
|---|---|---|---|---|---|---|---|---|---|
| p₁ target | 0.620 | 0.250 | 0.020 | 0.020 | 0.020 | 0.020 | 0.020 | 0.020 | 0.020 |
| q₁ draft | 0.700 | 0.200 | 0.010 | 0.010 | 0.010 | 0.010 | 0.010 | 0.010 | 0.010 |
| emitted | 0.621 | 0.247 | 0.019 | 0.019 | 0.019 | 0.019 | 0.020 | 0.018 | 0.019 |
Run the rule 100,000 times at position 1 (draft samples from q₁, accept or replace) and count what comes out. The result matches p₁ (0.62 “the”, 0.25 “a”), not q₁ (0.70, 0.20): the draft changed the speed, not the answer. vLLM calls the method “theoretically lossless up to the precision limits of hardware numerics”.
p = target probabilities, q = draft probabilities. Rows are positions, columns are tokens; the outlined cell in each row is the drafted token.
Every round yields at least one token (the correction or, if all survive, a bonus token from the last distribution) and at most .
How much faster
Let be the acceptance rate: the chance a draft token survives, assumed equal at every position. For a sampled draft token it is over the vocabulary; in the example above that is 0.92, 0.95, 0.75, 0.9 at the four positions, about 0.88 on average. From Leviathan et al.:
E = (1 − αk+1) / (1 − α); cost = k·c + 1; S = E / cost. Assumes the same α at every position and that verifying k + 1 positions costs one target step (true while decode is bandwidth-bound). Real gains are lower: sampling, scheduling and KV bookkeeping cost time too.
With :
| Acceptance rate | Draft length k | Tokens per round E | Round cost | Speedup S |
|---|---|---|---|---|
| 0.6 | 4 | 2.31 | 1.20 | 1.9× |
| 0.8 | 4 | 3.36 | 1.20 | 2.8× |
| 0.8 | 8 | 4.33 | 1.40 | 3.1× |
| 0.9 | 8 | 6.13 | 1.40 | 4.4× |
| 0.5 | 8 | 2.00 | 1.40 | 1.4× |
| 0.88 (the example above) | 4 | 3.94 | 1.20 | 3.3× |
- matters most. A draft that agrees 80% of the time is worth far more than a slightly cheaper one.
- Longer guesses have diminishing returns. All of the first guesses survive with chance , so late guesses are rarely used, while each costs a draft step. Move in the calculator and watch the curve flatten, then fall.
- Acceptance depends on the text. Code, boilerplate and repeated context are easy to guess; creative text is hard. Speedup varies per request.
When it helps less
- At large batch sizes. The cheap check assumes spare compute. A big decode batch already uses more of it, and verifying positions for every sequence multiplies the work. Gains shrink as load grows; some servers adapt or turn speculation off under load (vLLM lists “dynamic speculative decoding”).
- With a poorly matched draft. Low makes each round cost more than it returns ( below 1 in the calculator).
- Memory. The draft’s weights and its own KV cache take space.
Variants
| Method | Where guesses come from |
|---|---|
| Draft model | a separate small LM with the same tokenizer (the original method) |
| Medusa | extra prediction heads on the target that guess tokens t+2, t+3… from its last hidden state |
| EAGLE | a light draft head (about one transformer layer) that runs autoregressively on the target’s hidden states rather than on tokens alone; candidates form a tree verified in one pass with a tree-shaped attention mask |
| EAGLE-2 | same, but grows the tree dynamically where the draft is confident |
| EAGLE-3 | fuses low-, mid- and high-level target features and trains the head on its own predictions, improving acceptance at longer depths |
| MTP (multi-token prediction) | modules trained with the model itself to predict further ahead (DeepSeek-V3), reused as the draft |
| N-gram / prompt lookup | copy a continuation of the last few tokens from earlier in the prompt; free, good for edits and RAG |
vLLM supports, among others, draft models, EAGLE, MTP, n-gram and suffix decoding (docs). EAGLE-style heads usually get higher than a separate draft model of similar cost because they see the target’s internal features.
Why it means two prefills in a disaggregated server
Both models need the prompt’s KV cache before the first round: the draft to make its first guess, the target to verify. A server that runs prefill and decode on separate machines therefore needs two prefill pools, one per model, and the decode side receives KV from both.
- The draft pool is cheaper. A draft with a few layers prefills several times faster than a 36-layer target and sends far less KV (one transfer per layer), so it needs fewer replicas.
- Both pools gate the first token. A request needs a target prefill and a draft prefill before decode can start, so a shortage in either pool raises TTFT for everyone. The smaller draft pool loses a larger share of its capacity when one node fails; spreading both pools across nodes matters.
Acceptance rates are measured on the decode side. vLLM-based decoders export vllm:spec_decode_num_accepted_tokens_per_pos.
Common misconceptions
- “Speculative decoding trades quality for speed.” With the standard rejection rule the output distribution is the target’s (step 7 above). Only explicitly lossy variants change it.
- “The draft writes the answer.” The target decides every token; the draft only proposes.
- “Speedup = k.” Speedup is , usually 1.5–3× in practice, and it varies with the text and the load.
Further reading
- Fast inference from transformers via speculative decoding (Leviathan et al.: algorithm and speedup formula)
- Accelerating LLM decoding with speculative sampling (Chen et al.)
- Medusa · EAGLE · EAGLE-2 · EAGLE-3
- Inference with reference: lossless acceleration (copying from the prompt)
- DeepSeek-V3 technical report (multi-token prediction)
- vLLM speculative decoding