Speculative Decoding with Draft and Target Models
Speculative decoding is a text-generation technique in which a fast draft model proposes several tokens and a slower, more capable target model checks them in a batch. When the proposals are usually correct, the target can produce multiple output tokens for roughly the cost of one ordinary generation step.
The problem: one expensive step per token
A language model generates text one token at a time. A token is a small piece of text chosen from the model's vocabulary: it may be a whole word, part of a word, punctuation, or whitespace. At each step, the model reads the tokens generated so far and predicts a probability distribution for the next one.
With a large model, each next-token prediction can be expensive. The model must run its layers again after every token because the next prediction depends on the newly appended token. A response of 100 tokens therefore involves roughly 100 sequential target-model steps. Even if the hardware has enough compute to process many values in parallel, this dependency chain limits how much of that capacity can help.
The usual result is visible as time to first token followed by a relatively steady inter-token latency: the delay between successive output tokens. Speculative decoding primarily targets that second number.
Instead of asking the expensive model for one token, ask a cheap model for a short continuation, then ask the expensive model to judge the whole continuation at once.
Two models, two jobs
The target model is the model whose output quality and probability distribution the system wants to preserve. It is often a large model. The draft model is a smaller, faster model that predicts what the target is likely to say next. The draft model does not need to be identical to the target, but it generally needs to produce plausible continuations for the same token vocabulary and task.
Suppose the generated text currently ends with:
The database connection pool is
The draft model might quickly propose:
running out of available connections
Rather than immediately showing those tokens, the serving system sends the current context plus the proposed continuation to the target model.
Verifying a proposed block
The target model is still causal: its prediction at a position cannot use future text. That does not prevent parallel computation when the candidate tokens are already known.
For a proposed sequence of k tokens, the target can process the sequence in one forward pass. Its output at each position gives the probability distribution for the next token, conditioned on the context and the earlier proposed tokens. The system then compares the target's predictions with the draft's proposals from left to right.
For example, the draft may propose four tokens:
running out of available connections
The target might agree with the first three but prefer connection instead of connections for the fourth. The system accepts the matching prefix, emits those three tokens, and discards the remainder of the proposal. It then uses the target's distribution at the first disagreement to choose the next token.
The important distinction is between sequential dependence and parallel evaluation. The target cannot decide token four without knowing tokens one through three, but those proposed tokens are already supplied as input. It can therefore calculate the predictions for all positions during one model invocation rather than making four separate invocations.
The target typically retains its attention state, often called a key-value cache or KV cache, for the already generated context. The proposed block is evaluated against that cache, and the cache is advanced only for the tokens that survive verification.
How acceptance works
There are two useful ways to understand acceptance.
For greedy decoding, where each model always chooses its highest-probability token, verification is straightforward: accept a draft token if it matches the target's choice, and stop at the first mismatch.
Sampling requires more care. A model's output is a probability distribution, not just a single choice. If the system always accepted draft tokens that happened to be plausible, it could change the target model's output distribution. Speculative decoding therefore uses an acceptance test based on both models' probabilities.
Let q(x) be the draft model's probability for candidate token x, and p(x) be the target model's probability for that same token. A common test accepts the draft token with probability:
min(1, p(x) / q(x))
A candidate that the target considers at least as likely as the draft proposed it is accepted automatically. A candidate that the draft overestimated is accepted only some of the time. If it is rejected, the system samples from a corrected distribution derived from the target and draft distributions. That correction ensures speculative decoding can produce the same distribution as ordinary sampling from the target, subject to the usual random-number and implementation details.
After a rejection, later draft tokens are invalid because they were based on a token that was not accepted. They are discarded. If every proposed token is accepted, the system can often use the target's final computed distribution to continue generation without another full target step, depending on the implementation.
Why this reduces latency
In ordinary decoding, the target might need five sequential calls to generate five tokens. With speculation, the draft first generates five candidates using its cheaper calls. The target then verifies those candidates in one call. If all five are accepted, the target has advanced the output by five tokens while paying for one target invocation.
The gain depends on several factors:
- Draft speed: a draft model that is too slow consumes the saved time.
- Acceptance rate: similar models, suitable prompts, and short draft blocks usually produce more accepted tokens.
- Target hardware utilization: the target benefits when it can process the verification block efficiently in parallel.
- Draft length: longer proposals offer more opportunity to amortize target work, but they also increase the chance of an early rejection and add draft-model work.
Speculation does not make the target model itself cheaper. It changes how its computation is scheduled: many already-proposed positions are evaluated together instead of one position at a time. When you see configuration such as a draft model, a speculation length, an acceptance rate, or a log line reporting accepted versus rejected tokens, it is describing this trade-off. The system is measuring whether the fast model can predict enough of the target's next continuation to make parallel verification worthwhile.