Learn · LLM inference

Speculative decoding: draft and target

Updated 10 Nov 2026 · 7 min read

Decode makes one token per step, and each step is limited by reading weights and KV from memory, not by arithmetic. So a decode step has plenty of spare compute. Speculative decoding spends that spare compute to make several tokens per step, without changing what the model outputs.

The idea

A junior drafts, a senior reviews. Writing is slow because the senior (the big target model) has to “open the whole file” for every single word: one full read of its weights per token. Reading a few proposed words costs the senior almost the same as writing one, because opening the file is the expensive part. So let a fast junior (a small draft model) write the next few words, and have the senior check them all in one sitting. The senior keeps the words it would have written itself and replaces the first one it disagrees with.

Piece What it is for The question it answers
Draft model cheap guesses “What are the next kk tokens, probably?”
One target pass over k+1k + 1 positions the check, at the price of one decode step “What would the target have said at each position?”
Accept/reject rule keep the target’s answer exactly “Do I keep this guess?”
Acceptance rate α\alpha how often guesses survive “How much faster will this be?”

Why checking is cheap. A forward pass predicts the next token at every input position at once (that is how prefill works). Feed the target the current token plus the kk guesses and it returns its own predictions after each prefix, in one pass that reads the weights once.

Where the analogy breaks. The review is not a judgement call: it is exact arithmetic on probabilities, so the output is exactly what the target alone would produce (same distribution). And one rejection throws away every guess after it, because those guesses were built on a word that is no longer there.

One speculative round, k = 4
VERIFYDRAFTDraft modelsmall, fastTarget modelgpt-oss-120bOutput4 cheap steps: "the" "pod" "is" "ready"propose 4 tokensone forward pass over 5 positionsaccept "the", "pod"reject "is": emit target token "was"drop KV of "is" and "ready"; next round
The draft guesses four tokens; the target checks them in one pass; two are accepted and the target supplies the third.

Worked example: one round, real numbers

A toy vocabulary of 9 tokens and made-up probabilities (a real model has about 200,000 tokens and learns its probabilities). The text so far is “kubectl reports that”; the draft proposes kk = 4 tokens. Every number below is computed from the distributions; the random numbers come from a fixed seed.

Draft 4 tokens, verify in one pass

1 · The cache holds the context

Text so far
kubectlreportsthat
KVKVKV

Both models already have the KV cache of “kubectl reports that” (from prefill or earlier rounds). Plain decoding would now run the target once per token. Instead, the draft goes first.

2 · Draft: 4 cheap steps

q: draft's probabilities
theapodnodeiswasreadyrunning.
q₁0.700.200.010.010.010.010.010.010.01
q₂0.010.010.600.300.010.010.010.010.01
q₃0.010.010.010.010.550.350.010.010.01
q₄0.010.010.010.010.010.010.600.300.01
After drafting
kubectlreportsthatthepodisready
KVKVKVKVKVKVKV

The draft runs 4 ordinary decode steps of its own and picks “the”, “pod”, “is”, “ready”. Each step appends a KV entry in the draft’s cache. Being small, it is fast: here we assume each draft step costs 5% of a target step.

3 · Verify: one target pass over 5 positions

p: target's probabilities, one row per prefix
theapodnodeiswasreadyrunning.
after “that”0.620.250.020.020.020.020.020.020.02
after “the”0.010.010.550.350.010.010.010.010.01
after “pod”0.020.020.020.020.300.480.060.080.02
after “is”0.010.010.010.010.010.010.500.400.01
after “ready”0.040.040.040.040.040.040.040.040.70

The target runs once over “that the pod is ready” and returns 5 distributions: what it would say after each prefix. Row 5 (after “ready”) is only used if all 4 guesses survive. The target’s cache now has 4 new KV entries too, one per drafted token.

4 · Greedy rule: keep guesses that match the target's top token

PositionDraft guessedTarget’s top tokenVerdict
1thetheaccept
2podpodaccept
3iswasreject: emit “was”
4readyreadydiscarded (after a reject)
Result
kubectlreportsthatthepodisreadywas

With greedy decoding (always take the most likely token) the rule is a comparison. “the” and “pod” match. At position 3 the target prefers “was” (0.48) over the draft’s “is” (0.30): reject, and emit the target’s “was” instead. Position 4 is never looked at, even though “ready” happens to match. 3 tokens from one target pass instead of one.

5 · Sampling rule: accept with probability min(1, p/q)

PositionDraft tokenpqmin(1, p/q)random uu < min?
1the0.620.70.890.63accept
2pod0.550.60.920.14accept
3is0.30.550.550.75reject

When the server samples (temperature above 0), a guess is kept with probability min(1, p/q): always if the target likes it at least as much as the draft did, otherwise in proportion. “the”: the target gives 0.62 where the draft gave 0.70, so it survives with chance 0.89; u = 0.63 survives. Position 3: “is” survives with chance only 0.55; u = 0.75 rejects it.

6 · On reject: sample from what the draft under-predicted

Position 3: target, draft, and the residual
theapodnodeiswasreadyrunning.
p₃ target0.020.020.020.020.300.480.060.080.02
q₃ draft0.010.010.010.010.550.350.010.010.01
max(0, p − q)0.000.000.000.000.000.130.050.070.00
r = normalized0.010.010.010.010.000.520.180.260.01
Result
kubectlreportsthatthepodisreadywas
KVKVKVKVKV✕✕…

The replacement is drawn from the residual: the probability the target wanted but the draft did not cover. “was” holds 0.52 of it; u = 0.39 picks “was”. This correction is what makes the output distribution exactly the target’s. Then clean-up: KV entries for “is” and “ready” are freed in both caches (✕). The KV for “was” is written when the next round feeds it in.

7 · Check: the output really is the target's distribution

Position 1: 100,000 simulated rounds
theapodnodeiswasreadyrunning.
p₁ target0.6200.2500.0200.0200.0200.0200.0200.0200.020
q₁ draft0.7000.2000.0100.0100.0100.0100.0100.0100.010
emitted0.6210.2470.0190.0190.0190.0190.0200.0180.019

Run the rule 100,000 times at position 1 (draft samples from q₁, accept or replace) and count what comes out. The result matches p₁ (0.62 “the”, 0.25 “a”), not q₁ (0.70, 0.20): the draft changed the speed, not the answer. vLLM calls the method “theoretically lossless up to the precision limits of hardware numerics”.

p = target probabilities, q = draft probabilities. Rows are positions, columns are tokens; the outlined cell in each row is the drafted token.

Every round yields at least one token (the correction or, if all kk survive, a bonus token from the last distribution) and at most k+1k + 1.

How much faster

Let α\boldsymbol{\alpha} be the acceptance rate: the chance a draft token survives, assumed equal at every position. For a sampled draft token it is ∑min⁡(p,q)\sum \min(p, q) over the vocabulary; in the example above that is 0.92, 0.95, 0.75, 0.9 at the four positions, about 0.88 on average. From Leviathan et al.:

expected tokens per roundE=1−αk+11−αcost of a round=k c+1(c=draft step costtarget step cost)expected speedupS=Ek c+1\begin{aligned} \text{expected tokens per round} \quad E &= \frac{1 - \alpha^{k+1}}{1 - \alpha} \ \text{cost of a round} \quad &= k,c + 1 \qquad \left(c = \frac{\text{draft step cost}}{\text{target step cost}}\right) \ \text{expected speedup} \quad S &= \frac{E}{k,c + 1} \end{aligned}
Speedup calculator
Tokens per round E
Round cost, in target steps
Speedup S
Best k for this α and c
draft length kspeedup S

E = (1 − αk+1) / (1 − α); cost = k·c + 1; S = E / cost. Assumes the same α at every position and that verifying k + 1 positions costs one target step (true while decode is bandwidth-bound). Real gains are lower: sampling, scheduling and KV bookkeeping cost time too.

With c=0.05c = 0.05:

Acceptance rateDraft length kTokens per round ERound costSpeedup S
0.642.311.201.9×
0.843.361.202.8×
0.884.331.403.1×
0.986.131.404.4×
0.582.001.401.4×
0.88 (the example above)43.941.203.3×
  • α\alpha matters most. A draft that agrees 80% of the time is worth far more than a slightly cheaper one.
  • Longer guesses have diminishing returns. All of the first ii guesses survive with chance αi\alpha^i, so late guesses are rarely used, while each costs a draft step. Move kk in the calculator and watch the curve flatten, then fall.
  • Acceptance depends on the text. Code, boilerplate and repeated context are easy to guess; creative text is hard. Speedup varies per request.

When it helps less

  • At large batch sizes. The cheap check assumes spare compute. A big decode batch already uses more of it, and verifying k+1k + 1 positions for every sequence multiplies the work. Gains shrink as load grows; some servers adapt kk or turn speculation off under load (vLLM lists “dynamic speculative decoding”).
  • With a poorly matched draft. Low α\alpha makes each round cost more than it returns (SS below 1 in the calculator).
  • Memory. The draft’s weights and its own KV cache take space.

Variants

Method Where guesses come from
Draft model a separate small LM with the same tokenizer (the original method)
Medusa extra prediction heads on the target that guess tokens t+2, t+3… from its last hidden state
EAGLE a light draft head (about one transformer layer) that runs autoregressively on the target’s hidden states rather than on tokens alone; candidates form a tree verified in one pass with a tree-shaped attention mask
EAGLE-2 same, but grows the tree dynamically where the draft is confident
EAGLE-3 fuses low-, mid- and high-level target features and trains the head on its own predictions, improving acceptance at longer depths
MTP (multi-token prediction) modules trained with the model itself to predict further ahead (DeepSeek-V3), reused as the draft
N-gram / prompt lookup copy a continuation of the last few tokens from earlier in the prompt; free, good for edits and RAG

vLLM supports, among others, draft models, EAGLE, MTP, n-gram and suffix decoding (docs). EAGLE-style heads usually get higher α\alpha than a separate draft model of similar cost because they see the target’s internal features.

Why it means two prefills in a disaggregated server

Both models need the prompt’s KV cache before the first round: the draft to make its first guess, the target to verify. A server that runs prefill and decode on separate machines therefore needs two prefill pools, one per model, and the decode side receives KV from both.

  • The draft pool is cheaper. A draft with a few layers prefills several times faster than a 36-layer target and sends far less KV (one transfer per layer), so it needs fewer replicas.
  • Both pools gate the first token. A request needs a target prefill and a draft prefill before decode can start, so a shortage in either pool raises TTFT for everyone. The smaller draft pool loses a larger share of its capacity when one node fails; spreading both pools across nodes matters.

Acceptance rates are measured on the decode side. vLLM-based decoders export vllm:spec_decode_num_accepted_tokens_per_pos.

Common misconceptions

  • “Speculative decoding trades quality for speed.” With the standard rejection rule the output distribution is the target’s (step 7 above). Only explicitly lossy variants change it.
  • “The draft writes the answer.” The target decides every token; the draft only proposes.
  • “Speedup = k.” Speedup is E/costE / \text{cost}, usually 1.5–3× in practice, and it varies with the text and the load.

Further reading