Speculative Decoding 2026: How AI Models Got Twice as Fast Without Getting Dumber

Ask a large language model a question and it answers one word at a time — each token waiting for the one before it. That sequential grind is the reason big models can feel slow, and it used to seem unavoidable: you either ran a smaller, faster model and lost quality, or paid in latency for the good one. Speculative decoding quietly broke that trade-off. It is the reason a frontier model can stream at the speed of a much smaller one while producing exactly the same text — and it is running underneath most of the fast APIs you used this week.

The problem: one word at a time

Text generation is autoregressive — the model predicts token 2 using token 1, token 3 using tokens 1 and 2, and so on. Every step requires a full forward pass through the whole network. For a large model, that single pass is expensive, and you cannot start the next token until this one is done. The result is that latency is bound by the size of the model times the number of tokens, no matter how much GPU you throw at it. Adding hardware makes each pass a little faster; it does not remove the fact that they happen strictly in order.

The trick: guess ahead, then verify

Speculative decoding adds a second, much smaller draft model that races ahead and proposes several tokens at once — a cheap, fast guess at what comes next. The big target model then does something clever: instead of generating those tokens itself, it checks all of the draft's guesses in a single forward pass. Wherever the draft agreed with what the big model would have said anyway, those tokens are accepted for free. At the first disagreement, the big model's own answer wins, the rest of the draft is thrown away, and the cycle repeats.

Why does this help? Because a lot of language is predictable. Boilerplate, common phrasing, the back half of a word, closing brackets, the obvious next step in a list — a tiny model gets these right most of the time. The expensive big model gets to wave them through in bulk and spend its real effort only on the hard, surprising tokens. One big-model pass can finalize several tokens instead of one.

Acceptance rate is the whole game

The speed-up depends on one number: the acceptance rate — how many of the draft's guesses the big model keeps. A well-matched draft model on predictable text might see three or four tokens accepted per round; on dense, unusual content it might get one. The art of tuning speculative decoding is choosing a draft model that is fast enough to be cheap yet accurate enough to be accepted often. Push the draft too big and it costs as much as the model it is helping; too small and its guesses get rejected and you gain nothing.

The flavours you'll hear about

"Speculative decoding" is now a family of techniques, and vendors mix and match them:

• Draft-model speculation — the classic: a separate small model proposes tokens for a big one to verify.
• Self-speculation / Medusa-style heads — the model predicts several future tokens with extra lightweight output heads, so no separate draft model is needed.
• N-gram / prompt lookup — for tasks with lots of repetition (summarizing a document, editing code), guesses are pulled straight from the input text instead of a model.
• EAGLE and successors — smarter drafting that predicts at the feature level for higher acceptance rates.

You rarely choose these by name. They are baked into serving stacks — inference engines like vLLM and TensorRT-LLM ship them, and the major API providers (Anthropic, OpenAI, Mistral and the open-weight serving world) apply them behind the scenes. When a provider's fast tier feels suspiciously snappy for the model size, speculative decoding is often part of the answer, alongside quantization and prompt caching.

What it costs, and where it doesn't help

Speculative decoding is close to a free lunch, but not entirely. Running a draft model uses extra memory and compute, so on a fully saturated server handling many parallel requests, the spare capacity that made drafting cheap may not exist — the gains shrink under heavy batching. It also helps latency per request more than raw throughput across a busy fleet. And on genuinely unpredictable output, where the draft is wrong most of the time, the overhead can even make things slightly slower. That is why it is a tunable feature, not a switch that is always on.

Why buyers should care

You will not implement speculative decoding yourself unless you self-host — but knowing it exists changes how you read a pricing page. Two providers serving the "same" model can differ wildly in latency because one has a better serving stack. When you benchmark models for a latency-sensitive product (a voice agent, a live coding assistant, an interactive chatbot), measure time-to-first-token and tokens-per-second on your own prompts, not just the model's benchmark scores. Serving technique is now as much a differentiator as the model weights. And if you do self-host for data-residency reasons, turning on speculative decoding in your inference engine is one of the cheapest speed-ups available.

The bottom line

Speculative decoding is a rare kind of optimization: it makes AI meaningfully faster with zero cost to output quality, by letting a cheap model do the easy guessing and a smart model do only the checking. It is invisible to end users and increasingly standard in production serving, which is exactly why frontier-quality models can now stream at a pace that felt impossible two years ago. When you next notice a big model answering with startling speed, this draft-and-verify dance is very often what you are watching.