A common moment in private LLM projects: the model runs on your own GPUs, quality is acceptable, and users complain that answers stream slowly. Buying more hardware raises throughput but barely changes how quickly each token appears for one user. Quantisation and smaller models change the outputs. Speculative decoding is attractive because it does neither.
How it works
Generating text one token at a time leaves a large model's GPU mostly waiting on memory, not compute. Speculative decoding uses that slack. A cheap proposer guesses the next few tokens; the large target model checks all of them in one parallel pass; accepted tokens are kept, and generation continues from the first rejection1.
The original Google Research paper showed this can sample from the target model's distribution exactly, so outputs are unchanged, and reported 2 to 3 times faster generation on T5-XXL with existing models and no retraining1. In practice, floating-point differences in batched GPU execution mean byte-identical outputs are likely rather than guaranteed.
Speculative decoding trades spare compute for latency. If your GPUs have no spare compute, there is nothing to trade.
The proposer options
| Method | How it proposes | Trade-off |
|---|---|---|
| Draft model | A smaller model from the same family | Good acceptance; extra memory and a model to manage |
| EAGLE family | A light head trained on the target model's internal features | Strong speedups; needs a head trained for your exact model |
| MTP | Multi-token prediction layers built into some models | No separate model, where the base model supports it |
| N-gram / suffix | Copies likely continuations from the prompt or earlier output | No training; helps most when output repeats input |
The current vLLM documentation summarises it this way: model-based methods such as EAGLE, MTP and draft models give the best latency reduction, while n-gram and suffix decoding give modest speedups without adding much work during peak traffic3. The EAGLE-3 paper reports speedups of up to 6.5 times in its experiments and argues its design lets the draft head benefit from more training data2. The vLLM project also maintains a Speculators library for training draft models that run in vLLM4.
When it helps, and when it doesn't
vLLM describes speculative decoding as aimed at reducing inter-token latency for memory-bound workloads at medium to low request rates3. That sentence is the whole story:
- Interactive assistants with modest concurrency: good fit. Each user sees faster streaming.
- Code editing, document revision, extraction: often good for n-gram methods, because the output repeats large parts of the input.
- High-throughput batch jobs: weak fit. The GPU is already saturated, and verification work competes with real requests.
- Peak hours on shared clusters: gains can fall as load rises, which is why the cheaper methods appeal.
Older vLLM releases also carried caveats, such as incompatibility with pipeline parallelism and inconsistent latency gains across datasets5. Check the documentation for your exact version rather than relying on a blog post's configuration.
How I'd evaluate it
- Replay real traffic, not a benchmark prompt set. Acceptance rate depends heavily on what your users ask and how your prompts are built.
- Measure the right latencies: time to first token, inter-token latency and end-to-end time, at your real concurrency, using the telemetry in LLM observability.
- Track acceptance rate and tune the number of speculative tokens; low acceptance means wasted verification.
- Check outputs on your evaluation set to confirm nothing changed that matters.
- Compare with alternatives. Prompt caching and routing easy requests to smaller models may deliver more for less effort.
- Decide per workload. Enable it for interactive endpoints and leave batch endpoints alone, using the serving patterns in vLLM inference serving.
How I can help
I design private LLM platforms that meet latency and cost targets on hardware you control, from model choice to serving configuration. See my AI architecture work, or analyse your challenge and bring the result to a call.
- 1Fast Inference from Transformers via Speculative DecodingLeviathan, Kalman and Matias (arXiv; ICML 2023) · 2022
- 2EAGLE-3: Scaling up Inference Acceleration of Large Language Models via Training-Time TestarXiv; NeurIPS 2025 · 2025
- 3Speculative decodingvLLM documentation
- 4Speculators documentationvLLM project
- 5Speculative decoding (v0.9.0)vLLM documentation