vLLM Speculative Decoding in 2026: EAGLE, P-EAGLE, MTP, DFlash2 and N-Gram Compared

Updated: November 4, 2026. A large language model can load successfully, keep its GPU busy, and still feel slow because generation happens one token at a time. Speculative decoding attacks that specific bottleneck: a cheaper proposer guesses several next tokens, then the main model verifies the guesses together.

In the latest vLLM release, this is no longer one feature with one configuration. The serving engine supports several families of speculation: ordinary draft models, EAGLE, parallel EAGLE (P-EAGLE), native multi-token prediction (MTP), n-gram and suffix lookup, plus newer block-drafting and adaptive-verification paths. Choosing the wrong method can add GPU memory and compute without improving the latency users actually notice.

This guide explains the architecture, compares the practical choices, and walks through a reproducible baseline using vLLM 0.33.0. It also shows how to measure draft acceptance instead of trusting a speedup number from somebody else's GPU.

Who this guide is for

• Engineers serving open-weight models behind an OpenAI-compatible API.
• Teams trying to reduce time-per-output-token (TPOT) or improve interactive generation latency.
• Developers comparing EAGLE, P-EAGLE, MTP, DFlash and simpler lookup-based methods.
• Operators who need to decide whether an optimization is worth extra VRAM, complexity and operational risk.

What changed in vLLM 0.33.0?

As of November 4, 2026, the vLLM release page lists v0.33.0 as the current release. It was published on November 2. The release notes include several inference changes relevant to speculative decoding:

• Support for Reflection Beam-1.0 with a DFlash2 drafter.
• An XPress refiner for block-diffusion drafters such as DFlash and DSpark.
• FLy, an opt-in approximate verification policy that is explicitly lossy.
• Adaptive budgeting and scheduling improvements for speculative tokens.
• Lookahead block hashes for EAGLE/MTP prefix-cache hits, enabled by default.
• Quantized draft language-model heads and additional MTP support through the Transformers modeling backend.

These are release-note facts, not a promise that every drafter works with every target model or GPU. The same release also moves the default CUDA wheels and images to CUDA 13.0 and drops CUDA 12.9 release artifacts. Check the exact image, driver and accelerator combination before upgrading an existing server.

Sources: vLLM releases and the v0.33.0 speculative-decoding guide.

1. Understand the decode bottleneck first

Autoregressive generation normally works like this:

1. The model reads the prompt and builds its internal key/value (KV) cache.
2. It predicts one next token.
3. That token is appended to the sequence.
4. The model runs another decode step to predict the next token.
5. The process repeats until the response is complete.

For a large model, each decode step may move a substantial amount of model weight data even though it emits only one token. In a memory-bandwidth-bound workload, the GPU can have compute capacity left while waiting for weights to be read. That is why inter-token latency can remain high even when the model is already loaded.

Speculative decoding adds a proposer. It predicts a block of candidate tokens cheaply. The target model then evaluates that block in a verification pass. The target accepts the matching prefix and corrects the first rejected position using its own distribution. The next cycle continues from the verified sequence.

The key is verification: the proposer does not get final authority over the answer. Under standard speculative sampling, the target model remains the source of truth.

2. The methods are related, but not interchangeable

vLLM's own method-selection table is deliberately qualitative: real gains depend on the model family, traffic pattern, hardware and sampling settings. It generally positions model-based methods for latency reduction and n-gram/suffix methods as lighter options that avoid loading another draft model. Treat that as a shortlist, not a benchmark result for your deployment.

3. Know what the names mean under the hood

Draft-model speculation

A separate, smaller model proposes K tokens. The target checks them in one verification step. This is conceptually straightforward, but the draft model must be compatible with the target. By default, vLLM expects matching vocabularies. The current vLLM configuration documents a Token-Level Intersection option for heterogeneous vocabularies, but that path has limitations: it currently supports greedy draft sampling only, not probabilistic draft sampling.

EAGLE and EAGLE-3

EAGLE-style drafters use intermediate hidden states from the target model rather than acting like a completely independent small language model. This can make proposals more informed, but it requires a checkpoint trained for the target architecture and the right vLLM method/configuration. A random small model is not an interchangeable EAGLE head.

P-EAGLE

Ordinary EAGLE drafts tokens autoregressively: deeper speculation can require more sequential drafter passes. P-EAGLE changes that part of the process. It uses target hidden states and learned placeholders to construct several draft positions in parallel, then predicts them in one drafter forward pass. The vLLM integration exposes this through parallel_drafting: true for compatible EAGLE/draft-model methods.

The vLLM/Amazon/NVIDIA technical post reports up to 1.69× speedup over vanilla EAGLE-3 on selected workloads on an NVIDIA B200. That is a vendor-reported result for specific models, hardware, concurrency and benchmark sets. It is not a general speedup guarantee for an H100, consumer GPU, different target model or production traffic.

MTP

Multi-token prediction is not simply “attach any assistant model”. The target family must expose a compatible MTP path. vLLM's MTP guide documents supported families and specific assistant-checkpoint handling. For example, Gemma 4 assistant checkpoints use the mtp method; treating them as generic draft models can cause initialization failures on older releases.

DFlash and DFlash2

DFlash uses a block-drafting design: a smaller diffusion-style drafter predicts a group of candidate positions from target hidden states rather than generating each proposal token sequentially. vLLM 0.33.0's release notes specifically call out Reflection Beam-1.0 with a DFlash2 drafter and an XPress refiner for block-diffusion drafters. Do not infer that a DFlash2 checkpoint can be paired with any target model. Use the exact model card and vLLM recipe for the pair you deploy.

N-gram and suffix lookup

These methods avoid loading a second neural model. N-gram lookup searches token history for matching fragments and proposes a continuation. It is attractive when the input contains repeated structures, such as code templates, structured documents or repetitive transformations. If the response is novel and there are few useful matches, it may propose very little and provide little benefit.

4. Install a pinned vLLM release before tuning

Use a clean GPU server and pin the version during evaluation. This example uses the official vLLM 0.33.0 CUDA 13.0 container. It assumes an NVIDIA GPU, a working NVIDIA Container Toolkit installation, a compatible host driver, and enough GPU memory for the target model and KV cache.

docker pull vllm/vllm-openai:v0.33.0

For a private or gated Hugging Face model, export a read token in your shell. Do not paste the token into a public script or commit it to Git.

export HF_TOKEN=your_hugging_face_read_token

Start with a target model you can already serve. The first experiment uses N-gram speculation because it does not require a separate drafter checkpoint:

docker run --rm --gpus all --ipc=host \
  -p 8000:8000 \
  -v "$HOME/.cache/huggingface:/root/.cache/huggingface" \
  -e HF_TOKEN="$HF_TOKEN" \
  vllm/vllm-openai:v0.33.0 \
  --model Qwen/Qwen3-8B \
  --speculative-config '{"method":"ngram","num_speculative_tokens":4,"prompt_lookup_min":2,"prompt_lookup_max":5}' \
  --per-request-spec-decode-metrics summary

Why begin with N-gram? It gives you a low-complexity way to validate the request path, inspect metrics and learn whether the workload has repeatable token patterns. It is not guaranteed to speed up ordinary open-ended chat.

Expected result: vLLM loads the target model and starts an OpenAI-compatible API on port 8000. The first request may take longer while model weights are downloaded and loaded. Check startup logs for the resolved speculative configuration; if the engine reports a different method or rejects a configuration key, stop and consult the docs for the exact pinned version rather than adding random flags.

5. Send a request and confirm the server works

curl http://localhost:8000/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{
    "model": "Qwen/Qwen3-8B",
    "messages": [
      {"role": "user", "content": "Explain what a database index does in three concise steps."}
    ],
    "temperature": 0,
    "max_tokens": 180
  }'

Confirm that the response contains a normal assistant completion before comparing performance. This separates model/API problems from speculative-decoding problems.

6. Measure acceptance, not just tokens per second

vLLM 0.33.0 can return per-request speculative-decoding metrics when you start the server with --per-request-spec-decode-metrics summary or detailed. The feature is marked experimental, so pin your version if downstream code parses the response schema.

Send a single-sequence request (n=1, the default) and inspect the top-level metrics.speculative_decoding object:

curl http://localhost:8000/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{
    "model": "Qwen/Qwen3-8B",
    "messages": [
      {"role": "user", "content": "Write a Python function that validates an email address and explain its edge cases."}
    ],
    "temperature": 0,
    "max_tokens": 400
  }' | python -m json.tool

Pay attention to these fields:

• mean_acceptance_length: mean tokens emitted per verification step, including the bonus token. A value of 1.0 means no draft tokens were accepted.
• draft_acceptance_rate: accepted draft tokens divided by proposed draft tokens.
• acceptance_histogram: how many verification steps accepted exactly zero, one, two, and so on draft tokens.
• num_draft_tokens and num_accepted_draft_tokens: useful totals for understanding whether the proposer is doing useful work.

These metrics explain the mechanism, but acceptance is not the final performance result. A high acceptance rate can still lose if the drafter is expensive, the verification step is slow, or the workload is compute-bound. Conversely, a lower acceptance rate can still help if drafting is extremely cheap.

For streaming requests, per-request metrics are delivered in the final usage chunk; enable usage reporting with stream_options.include_usage: true or the documented server option. Per-request metrics are only reported for single-sequence requests and may be null for n > 1.

7. Add a model-based drafter only when the baseline justifies it

Once the baseline is stable, choose a compatible model-based method. A generic draft-model configuration looks like this:

vllm serve <target-model> \
  --speculative-config '{
    "method": "draft_model",
    "model": "<compatible-draft-model>",
    "num_speculative_tokens": 5
  }'

For an EAGLE-3 or P-EAGLE checkpoint, follow the target model's official pairing instructions. The shape of the configuration is:

vllm serve <target-model> \
  --speculative-config '{
    "method": "eagle3",
    "model": "<matching-eagle3-or-p-eagle-checkpoint>",
    "num_speculative_tokens": 5,
    "parallel_drafting": true
  }'

parallel_drafting is not a universal switch for every drafter. vLLM documents it for compatible EAGLE and draft-model methods; the checkpoint must have been trained for parallel drafting. Do not set it just because a model name contains “EAGLE”.

For a model with native MTP support, vLLM's documented MiMo example is:

vllm serve XiaomiMiMo/MiMo-7B-Base \
  --tensor-parallel-size 1 \
  --speculative-config '{"method":"mtp","num_speculative_tokens":1}'

Start with the method and token count in the exact model's vLLM recipe. Then test nearby values. A value that works for one target/drafter pair may not work for another.

8. Benchmark the baseline and speculative server fairly

Do not compare a speculative server under one workload with a baseline under another. Keep the following constant:

• Target model revision, quantization, tokenizer and chat template.
• Prompt dataset and output-length distribution.
• Sampling parameters, stop conditions and maximum context length.
• GPU, tensor-parallel layout, KV-cache dtype and memory settings.
• Concurrency, request rate, warm-up procedure and number of repetitions.

For a feature-level test, vLLM documents vllm bench serve. For example, after starting the server, run an identical benchmark against the baseline and speculative variants:

export MODEL="Qwen/Qwen3-8B"
export BASE_URL="http://localhost:8000"

vllm bench serve \
  --backend openai-chat \
  --model "$MODEL" \
  --endpoint /v1/chat/completions \
  --base-url "$BASE_URL" \
  --dataset-name hf \
  --dataset-path philschmid/mt-bench \
  --num-prompts 80 \
  --max-concurrency 1 \
  --temperature 0 \
  --save-result \
  --save-detailed

Repeat at concurrency 1, 4, 16 and a level representative of your service. Record at least:

• Time to first token (TTFT).
• Time per output token (TPOT) and inter-token latency percentiles.
• Output tokens per second and requests per second.
• GPU memory use, GPU utilization and queueing delay.
• Mean acceptance length and draft acceptance rate.

Run warm-ups, repeat the test, and compare medians and tail percentiles. A single short run is not enough. vLLM's benchmark documentation notes that its CLI is useful for feature evaluation and regression testing; for production server benchmarking it recommends considering GuideLLM, which offers more flexible workload patterns and reporting.

9. Tune the method to the workload

If the service is latency-focused and traffic is light

Start with a supported EAGLE, P-EAGLE, MTP or draft-model configuration. The vLLM guide describes model-based methods as promising for low-QPS, latency-focused workloads. Measure the extra draft memory and actual TPOT improvement.

If requests repeat input text or templates

Try N-gram lookup first. It adds no separate neural drafter. It is especially worth testing on workloads where the output can reuse material from the prompt, but do not assume it helps general conversation.

If concurrency is high

Speculation is not free. Draft tokens consume compute and scheduling budget. At high concurrency, that work competes with real requests. The best method and speculation depth can change as the batch grows. vLLM's adaptive-verification work exists to allocate verification effort according to expected value rather than blindly using the same depth for every request.

If acceptance is poor

• Check that the draft checkpoint is intended for the target model and tokenizer.
• Reduce speculative depth and compare acceptance histograms.
• Use prompts from your real workload; synthetic prompts may have different repetition and token distributions.
• Check whether structured-output constraints are invalidating draft tokens.
• Try a different method rather than endlessly increasing K.

10. Lossless verification versus FLy

Standard speculative decoding uses rejection sampling to preserve the target model's output distribution in theory, subject to floating-point and hardware numerical effects. vLLM documents algorithmic validation for its standard rejection sampler and greedy equality tests.

FLy is different. vLLM explicitly labels FLy as lossy: it can defer a native rejection under certain high-entropy conditions, but it does not preserve the target distribution and may change output quality. Treat it as an opt-in quality/performance trade-off, not a drop-in lossless optimization. It is inappropriate where exact distribution preservation is a hard requirement.

11. Troubleshooting checklist

• Unknown speculative method: Confirm the method name exists in the pinned vLLM release. Use the v0.33.0 docs rather than copying a flag from a nightly build or another backend.
• Draft checkpoint fails to load: Verify target/drafter pairing, architecture, revision, tokenizer and required model-code settings from the checkpoint's official card.
• CUDA or PyTorch mismatch: vLLM 0.33.0 release artifacts use CUDA 13.0 by default. Do not assume a CUDA 12.9 container or wheel from an older deployment is interchangeable.
• Out of memory: The target, drafter, KV cache and CUDA graphs all consume memory. Reduce context length, batch limits or parallelism, and re-check the exact model's memory requirements.
• No visible speedup: Compare TPOT, not only total request duration. Short responses may not amortize drafter startup or scheduling overhead.
• Mean acceptance length is near 1.0 or draft_acceptance_rate is near zero: The proposer is rarely accepted. Check compatibility and workload, then disable speculation if the cost exceeds the benefit.
• Metrics missing: Enable per-request metrics, use a single output sequence, and for streaming include the final usage chunk. The response field is experimental.
• Hybrid attention/KV-cache errors: Some model cards document version-specific cache-grouping fixes for particular target/drafter pairs. Read the model card and linked vLLM issue/PR; do not assume a generic upgrade or a copied workaround fixes every hybrid model.

12. Security and operational considerations

• Keep the vLLM API private while testing. Do not expose port 8000 directly to the public internet without authentication, TLS and network controls.
• Use a read-only Hugging Face token where possible. Keep it in a secret manager or runtime environment, not in a Dockerfile or repository.
• Pin container tags and model revisions for reproducibility. Avoid silently moving model branches in production.
• Download model and drafter checkpoints only from trusted publishers. Verify licenses and model-card requirements before commercial use.
• Apply request limits and authentication at the gateway. Speculative decoding does not replace quotas, tenant isolation or abuse controls.
• Monitor GPU memory, queue depth, request latency and acceptance metrics together. An optimization that helps one tenant's short requests may hurt another tenant's long-context workload.
• Keep a tested non-speculative configuration so you can roll back quickly if a drafter or new vLLM release introduces a regression.

13. A practical decision tree

1. Need a low-risk first experiment? Start with N-gram lookup. It requires no separate draft model.
2. Does the target model ship native MTP support in your vLLM version? Try its documented MTP path before adding another model.
3. Do you have a matching EAGLE/P-EAGLE checkpoint? Test it at low concurrency and verify that your exact target/drafter pair is supported.
4. Is the target explicitly supported with DFlash/DFlash2? Use the model-specific vLLM recipe and measure it independently.
5. Does your production workload saturate the GPU at high concurrency? Compare adaptive verification and no-speculation baselines; more draft tokens are not automatically better.
6. Does the method improve TPOT and tail latency without hurting throughput or quality? Only then consider rolling it out gradually.

FAQ

Does speculative decoding make the model smarter?

No. It changes how candidate tokens are proposed and verified. The target model remains responsible for the final output under standard verification.

Will it always increase tokens per second?

No. It is most useful when decode latency is the bottleneck and the proposer is cheap enough. It can lose at high concurrency, on short outputs, or when acceptance is poor.

Is P-EAGLE the same as MTP?

No. Both can predict multiple positions, but P-EAGLE is a trained EAGLE-style drafter that uses target hidden states and parallel drafting. MTP uses a native multi-token prediction capability supported by the target model family or its designated assistant path.

Can I use any small model as the draft model?

Not safely by assumption. The default draft-model path expects compatible vocabularies. The heterogeneous-vocabulary path has documented restrictions, and EAGLE/P-EAGLE/MTP checkpoints have architecture-specific requirements.

Does FLy preserve the exact target distribution?

No. vLLM labels FLy as lossy and warns that it may degrade outputs. Standard rejection sampling is the lossless-oriented path.

What should I optimize first: acceptance rate or throughput?

Measure both, along with TPOT and tail latency. Acceptance explains how well the proposer guesses; end-to-end measurements show whether those guesses make the service faster.

Conclusion

Speculative decoding is not a magic “make every LLM faster” switch. It is a family of scheduling and verification techniques with different hardware costs and compatibility rules. Start with a pinned vLLM release, establish a baseline, and try N-gram or a model-native MTP path before adding a more complex drafter. Move to EAGLE, P-EAGLE or DFlash only with an explicitly compatible checkpoint, and measure the same real workload at multiple concurrency levels.

The most useful number is not a headline speedup from a B200 benchmark. It is the improvement you can reproduce on your own prompts, hardware and traffic—with quality, tail latency and GPU memory still inside your service's limits.

Further reading on GyanAangan

• vLLM installation, OpenAI API and GPU-memory serving guide — get the serving foundation right before tuning decoding.
• Ollama vs llama.cpp vs LM Studio vs vLLM — choose the right inference runtime for the deployment.
• Benchmark local GGUF models properly — build repeatable performance comparisons.
• Trace local AI applications with OpenTelemetry and Phoenix — connect inference tuning to observability.
• Build a local LLM gateway with LiteLLM — put a consistent API layer in front of local inference engines.

Primary sources and further documentation

• vLLM release notes — v0.33.0 date, release artifacts and speculative-decoding changes.
• vLLM 0.33.0 speculative-decoding guide — methods, configuration schema, compatibility notes and lossless guarantees.
• Per-request acceptance metrics — metric definitions and response shape.
• MTP guide — supported target and assistant-checkpoint paths.
• vLLM benchmark CLI — repeatable server benchmarking.
• P-EAGLE technical post by the vLLM, Amazon and NVIDIA teams — architecture, supported checkpoints and vendor benchmark methodology.
• Amazon GPT-OSS-20B P-EAGLE model card — checkpoint-specific serving instructions and compatibility caveats.
• Adaptive verification guide — dynamic draft-budget behavior and current limitations.
• FLy verification guide — approximate, explicitly lossy verification.

Editorial note: This article was prepared from the official vLLM release notes, versioned documentation and model cards checked on November 4, 2026. Commands are documentation-based examples, not claims of a live benchmark run. Model support and performance must be verified on the exact target/drafter pair and hardware before production rollout.