What Is Speculative Decoding? How LLMs Draft and Verify Tokens Faster

Speculative decoding is an LLM inference technique that speeds up token generation without changing output quality. A smaller, faster draft model proposes several candidate tokens ahead, and the larger target model checks all of them in a single parallel pass—keeping valid predictions and resampling from the first mismatch to preserve its exact output distribution.

Large language models usually generate responses one token at a time. Each new output step requires reading the model’s weights from memory, leaving accelerator compute underutilized while waiting on memory bandwidth. Speculative decoding breaks that one-token-per-step bottleneck by turning sequential checks into parallel verification.

Why Is Standard Autoregressive Decoding Slow?

Standard autoregressive decoding is slow because generating each token requires a separate forward pass through the full model, making the process memory-bandwidth bound rather than compute bound.

When a language model writes a response, every token depends on all the tokens that came before it. In standard decoding, a 70-billion-parameter model must load its weights from high-bandwidth memory to produce token one, then load those same weights again to produce token two, and repeat that cycle hundreds of times.

Yet many tokens in a response are easy to predict. Function words, closing brackets in code, common phrases, and boilerplate syntax do not require the full reasoning depth of a massive model to guess accurately, even though you still want the large model’s final judgment on the whole response. Because GPUs and TPUs can process a short block of tokens in parallel in nearly the same wall-clock time it takes to process a single token, extra compute capacity sits idle during single-token decoding. Speculative decoding puts that spare parallel capacity to work.

How Does Speculative Decoding Work Step by Step?

Speculative decoding works by running a four-step loop—drafting candidate tokens, verifying them in parallel with the target model, accepting matching tokens, and resampling at the first rejection—until the response is complete.

Rather than asking the large target model to generate every token sequentially, the inference engine pairs the target model with a much smaller, faster approximation (often called the draft model). Each cycle runs through four distinct phases:

1. Drafting candidate tokens: Starting from the current sequence, the small draft model quickly generates a short lookahead window of $K$ candidate tokens one by one, recording its probability distribution for each position.
2. Parallel verification: The large target model takes the original sequence plus all $K$ drafted tokens and runs a single forward pass. Because transformer architectures can evaluate known token sequences in parallel—just like they do during prompt processing—the target model computes its own probability distributions for every drafted position simultaneously.
3. Accepting or rejecting draft tokens: Moving from left to right across the $K$ drafted tokens, the engine compares the draft model’s probability against the target model’s probability at each step using a formal rejection sampling rule (often called speculative sampling). In greedy decoding (temperature zero), a drafted token is accepted whenever it matches the target model’s top choice. In sampling modes, tokens where the target model assigns at least as much probability as the draft model are always accepted, while lower-probability tokens are accepted proportionally to the ratio of target-to-draft probability.
4. Resampling and bonus token generation: If the first mismatch happens at position $m$, that token and all subsequent draft tokens after it are discarded, and a replacement token for position $m$ is sampled directly from an adjusted residual distribution that accounts for both models’ probabilities. If all $K$ drafted tokens are accepted, the target model already has the probability distribution for position $K + 1$ from the same parallel pass, allowing it to sample one additional bonus token for free.

Because the rejection and resampling rules mathematically cancel out the draft model’s mistakes, speculative decoding is lossless. Every multi-token step produces samples drawn from the exact same probability distribution as running the target model alone—meaning zero degradation in benchmark accuracy, reasoning ability, or tone.

How Much Faster Is Speculative Decoding in Practice?

Foundational research shows speculative decoding delivering roughly 2X to 3X wall-clock speedups on large language models when paired with a well-matched draft model and sufficient parallel compute.

The foundational papers that introduced the technique in late 2022 and early 2023 demonstrated substantial latency gains across both encoder-decoder and decoder-only transformers without altering model weights or output distributions:

Study	Target Model	Draft Setup	Reported Wall-Clock Speedup	Conditions and Tasks
Leviathan et al. (2023)	T5-XXL (11B parameters)	T5-small (77M parameters)	2X–3X (up to 2.6X–3.4X in T5-small configuration tables)	Out-of-the-box T5X implementation across translation (EN-DE) and summarization (CNN/DM) at argmax (temp 0) and standard sampling (temp 1).
Chen et al. (2023)	Chinchilla (70B parameters)	4B-parameter draft model trained for speculative sampling	2X–2.5X	Distributed setup on TPU v4 across summarization (XSum) and code generation (HumanEval).

Notice the size gap between the target and draft models in both studies: T5-small is roughly two orders of magnitude smaller than T5-XXL, and the 4B draft model is a small fraction of the 70B Chinchilla model. A draft model must be small enough that running $K$ sequential steps on it adds very little overhead compared to a single step of the target model.

What Determines the Acceptance Rate and Why Isn’t Speedup Guaranteed?

The speedup from speculative decoding depends on two main factors—how closely the draft model agrees with the target model (the acceptance rate) and how much spare parallel compute the hardware has available—so net speedups are never guaranteed.

If the draft model constantly guesses wrong, the system pays the latency cost of running $K$ sequential steps on the draft model only to have the target model reject the very first token. In that worst-case scenario, a speculative cycle produces just one resampled token while taking longer than a standard target-model step.

Several practical variables determine whether speculative decoding speeds up or slows down inference:

• Alignment between draft and target models: The draft model needs to share the exact same tokenizer and vocabulary as the target model, and its training distribution should match closely. Code completion, structured JSON output, and translation of predictable phrasing tend to yield higher acceptance rates than high-temperature creative writing where many wildly different tokens are equally plausible.
• Draft model latency overhead: Even a high acceptance rate will fail to speed up generation if the draft model itself is too large and slow. Leviathan et al. demonstrated that a smaller T5-small draft model outperformed larger T5-base and T5-large draft candidates in overall wall-clock speedup because its per-step drafting overhead was much lower.
• Lookahead window length ($K$): Picking too small a lookahead limits how many tokens can be verified in a single pass. Picking too large a lookahead wastes draft time on later tokens that get thrown away whenever an earlier token is rejected.
• Available parallel compute (batch size saturation): Speculative decoding trades idle compute for lower latency. In low-batch, latency-sensitive serving, accelerators have plenty of spare compute to verify $K$ tokens in parallel at almost no extra time cost. Under massive concurrent batch sizes where the GPU’s compute units are already fully saturated, adding $K$ extra verification positions per request increases pass time and can erase the speedup.

How Does Speculative Decoding Relate to KV Cache and Prompt Caching?

Speculative decoding only accelerates the output decode phase of generation, whereas KV caching avoids recomputing attention across tokens and prompt caching speeds up and discounts repeated input prefill.

Because these three optimizations all touch transformer inference latency, developers often confuse where each one sits in the request lifecycle:

• Prefill vs. decode: Every LLM request splits into two stages. Prefill processes all input prompt tokens in parallel to build the initial state, which is already compute-bound and fast per token. Decode generates the output tokens step by step, which is memory-bandwidth bound. Speculative decoding only helps during the decode stage; it does nothing to speed up reading a long input prompt.
• Role of the KV cache: Both the target model and the draft model rely on their own KV caches during speculative decoding. As candidate tokens are drafted and verified, their Key and Value states are appended to the cache. When the target model rejects drafted tokens after position $m$, the inference engine simply rolls back (truncates) the KV caches of both models to discard the rejected entries before starting the next cycle.
• Distinction from prompt caching: Prompt caching reuses precomputed KV cache blocks for identical prompt prefixes across separate API calls, cutting Time to First Token (TTFT) and often lowering input token prices. Speculative decoding cuts inter-token latency (ITL) while the response is streaming out.

What Does Speculative Decoding Mean for API Developers and Self-Hosters?

For API developers, speculative decoding shows up as lower end-to-end streaming latency rather than a separate billing line item, while self-hosters configure it directly inside inference engines to improve per-request responsiveness.

How you interact with speculative decoding depends on whether you call hosted model endpoints or manage your own GPU fleet:

1. Hosted API billing rarely changes per token: Hosted AI providers generally bill per million input and output tokens processed from the user’s perspective. Even if a provider uses speculative decoding under the hood, rejected draft tokens are internal to the serving engine—you are billed for the final generated output tokens at the model’s standard rate. To lower your direct API bill rather than just wait time, look at prompt engineering, batch APIs, and practical ways to reduce AI API costs or compare rates when deciding how to choose an AI model.
2. Real-time UX benefits: Where speculative decoding shines for application builders is user experience. Voice agents, IDE code assistants, and interactive production workflows—from live coding copilots to scripting pipelines paired with creative tools like AI Studio Video—feel noticeably snappier when output tokens arrive twice as fast without switching down to a weaker model.
3. Self-hosted configuration: If you run open-weight models on your own infrastructure, you configure speculative decoding directly in your serving stack. You must load both the target and draft models into GPU memory (or use a draft-free variant), ensure both share the exact same tokenizer, and tune the lookahead length against your typical concurrency level.

Do You Always Need a Separate Draft Model?

Later speculative decoding variants eliminate the need to train and deploy a separate small draft model while preserving the same draft-and-verify principle.

Managing two separate models in memory—and finding a small model with the exact same tokenizer and behavioral alignment as your target model—can be operationally awkward. To solve that, inference researchers and serving frameworks have introduced several draft-free approaches:

• N-gram and prompt lookup drafting: Instead of running a neural network to draft tokens, the engine searches the input prompt and recently generated output for matching text spans and copies the following tokens as candidates. This works particularly well when summarizing documents, editing code, or extracting quotes that already appear in the context window.
• Multi-token prediction heads and auxiliary layers: Rather than maintaining an entire standalone model, a small set of lightweight prediction heads or draft layers can be attached directly to the target model’s hidden states to propose the next few tokens in one pass.
• Self-speculative decoding: Some setups use a subset of the target model’s own layers (skipping intermediate layers during drafting) to generate candidate tokens, then run the full layer stack to verify them.

Regardless of how the candidate tokens are proposed, the core guarantee remains the same: the full target model verifies the candidates in parallel and enforces its exact probability distribution on every accepted token.

Frequently Asked Questions

Does speculative decoding change model accuracy or output quality?

No. Standard speculative decoding (speculative sampling) is mathematically lossless. For greedy decoding, it only accepts tokens that match the target model’s exact top prediction. For temperature sampling, its rejection and residual resampling rules ensure that generated sequences follow the exact same probability distribution as standard autoregressive decoding on the target model.

Does speculative decoding reduce Time to First Token (TTFT)?

No. Time to First Token measures how long the model takes to read your input prompt during the prefill stage and emit the very first output token. Because prefill already processes prompt tokens in parallel, speculative decoding only speeds up the subsequent decode steps—reducing inter-token latency and total completion time for the rest of the response.

Why can speculative decoding be slower at high batch sizes?

Speculative decoding relies on spare parallel compute on the GPU or TPU to check multiple draft tokens in roughly the same time it takes to check one token. When a server handles a very large batch of concurrent requests, the accelerator’s compute capacity is already fully utilized, so verifying extra draft tokens per request adds real execution time and can outweigh the saved memory reads.

Can any small model act as a draft model for any large model?

Not directly in standard token-level speculative decoding. The draft model and target model must share a compatible vocabulary and tokenizer so that candidate token IDs and their probability distributions map cleanly between the two models. That is why pairings usually come from the same model family, such as T5-small drafting for T5-XXL.

Sources

• Yaniv Leviathan, Matan Kalman, and Yossi Matias — Fast Inference from Transformers via Speculative Decoding (arXiv:2211.17192, ICML 2023)
• Charlie Chen, Sebastian Borgeaud, Geoffrey Irving, Jean-Baptiste Lespiau, Laurent Sifre, and John Jumper — Accelerating Large Language Model Decoding with Speculative Sampling (arXiv:2302.01318, 2023)