Speculative Decoding with Draft and Target Models

12 Sep 2026 artificial intelligencealgorithms

Speculative decoding is a generation technique in which a small, fast draft model proposes several tokens and a larger, slower target model verifies them in one pass. Correct proposals can be accepted together, reducing the number of expensive target-model steps without changing the distribution of the target model’s outputs.

The problem: one expensive step per token

Autoregressive language models generate one token at a time. A token is a unit from the model’s vocabulary: it might be a whole word, part of a word, punctuation, or whitespace. After producing each token, the model uses the longer prefix to calculate the probability distribution for the next one.

The dependency is inherently sequential:

1. Generate token 1.
2. Use token 1 to generate token 2.
3. Use tokens 1 and 2 to generate token 3.
4. Continue until the response is complete.

A large model is good at this work but expensive to run. Even when a GPU can process many positions in parallel, it cannot normally calculate the next position before the previous token is known. The result is that generating a response can involve dozens or hundreds of expensive model iterations.

Speculative decoding attacks this sequential cost. Instead of asking the target model for exactly one token, the system first lets a cheaper draft model guess a short continuation.

Two models, one vocabulary

The draft and target models generally use the same tokenizer and vocabulary. The draft model is smaller, so it can produce tokens quickly, although its predictions are less accurate.

Suppose the current generated text is a prefix P. The draft model predicts a distribution q for the next token, samples one, adds it to the prefix, and predicts again. It repeats this for a fixed number of draft steps, perhaps four to eight tokens. The result is a proposed continuation:

P + t1 + t2 + t3 + t4

The target model then receives this whole sequence. A causal transformer can calculate the next-token distribution at all of those positions in one forward pass because the proposed tokens are already present. This is still a causal calculation: each position only attends to earlier positions. The important difference is that the hardware can evaluate the known positions in parallel.

The target model produces distributions for t1, t2, t3, and the token after t4. It now verifies the draft from left to right.

Verification and acceptance

Let p(x) be the target model’s probability for token x at a particular position, and let q(x) be the draft model’s probability there. For a proposed token t, speculative sampling accepts it with probability:

min(1, p(t) / q(t))

This rule has two useful properties:

• If the target considers t at least as likely as the draft does, p(t) / q(t) is at least 1, so the token is always accepted.
• If the draft overestimated t, the token is accepted only some of the time.

Verification stops at the first rejected token. The accepted prefix is kept. The rejected token and every draft token after it are discarded, because those later guesses were based on a continuation that is no longer present.

At the rejection point, the system samples a replacement from a correction distribution. For each token x, that distribution is proportional to:

max(0, p(x) - q(x))

In other words, it gives the target model the probability mass that the draft failed to account for. If every proposed token is accepted, the target model samples one additional token from its ordinary distribution after the proposed block. That extra sample both advances generation and prevents the algorithm from stalling when the draft is entirely correct.

The same process repeats from the newly extended prefix.

The target model is not simply approving the draft model’s guesses. Rejection and correction are what preserve the target model’s sampling behavior.

Why the distribution does not change

Run the target model by itself with a particular sampling configuration—such as temperature, top-p filtering, or no filtering—and imagine the probability distribution of complete outputs. Speculative decoding is designed to produce that same distribution.

The draft model changes how the target distribution is reached, not the distribution being sampled. A proposed token that both models support strongly is accepted efficiently. A token that the draft favors too much is sometimes rejected, and the correction distribution restores the probability mass the target model requires. The resulting next token has the same distribution as a direct target-model sample.

This distinction matters. A faster method that always accepts the draft model’s output would be ordinary smaller-model decoding, and it would generally produce different text. Speculative decoding uses the draft only as a proposal mechanism; the target model remains authoritative.

For greedy decoding, where the system always selects the highest-probability token rather than sampling, implementations can use a corresponding deterministic verification rule. The distribution-preservation claim is most precisely about the sampling version, where acceptance probabilities and correction are essential.

Where the speedup comes from

Suppose the draft proposes five tokens and the target accepts four. The target has performed one forward pass while producing four tokens that would otherwise have required four sequential target steps. The draft work is cheaper, and the target’s parallel hardware evaluates the proposed positions together.

The gain depends mainly on three factors:

• Draft speed: the smaller model must generate proposals substantially faster than the target.
• Acceptance rate: the draft should often agree with the target, especially on predictable text.
• Target workload: batching several positions in one target pass must cost less than running several separate one-token passes.

Natural language with familiar patterns often produces good acceptance rates. Code, unusual names, highly creative text, or a draft model that is poorly matched to the target can produce many rejections. In those cases, speculative decoding may provide little benefit or add overhead.

What this looks like in practice

If you encounter a configuration mentioning a draft model, a speculation length, or an acceptance rate, it is likely controlling this mechanism. Logs may report phrases such as “accepted draft tokens” or “tokens per verification step.” Those metrics describe how many proposed tokens survive each target pass, not a change to the target model itself.

Speculative decoding is therefore a systems optimization with a correctness contract: use a cheap model to predict the likely path, use the expensive model to verify it, and apply rejection sampling so the final output remains distributed exactly as if the expensive model had generated it alone.