vLLM 0.34.0 Upgrade Guide: Model Runner V2, Extensible KV Cache and Breaking Changes (2026)
Updated November 18, 2026 · Based on vLLM 0.34.0, released November 16, 2026.
A vLLM upgrade can change more than the Python package version. In v0.34.0, the old GPU Model Runner V1 is removed, the default KV-cache block size changes from 16 to 64, and extensible KV-cache sizing becomes the default on CUDA and ROCm. A deployment that worked yesterday can fail at startup—or quietly behave differently under load—if its old flags and memory assumptions are carried forward unchanged.
This guide walks through a controlled upgrade from v0.33.x to v0.34.0, starts a pinned OpenAI-compatible server, tests it with a real request, and explains how to decide whether to keep the new memory defaults or preserve your previous configuration.
Who this guide is for
- Developers running vLLM as an OpenAI-compatible API server.
- Teams upgrading CUDA or ROCm inference hosts.
- Operators who maintain Docker Compose, systemd, or Kubernetes deployments.
- Engineers who use custom logits processors, quantized checkpoints, or tuned KV-cache settings.
This is not a beginner guide to training models, and it is not a promise that every model will fit on every GPU. The example uses Qwen3-0.6B as a low-cost smoke-test checkpoint; replace it only after confirming that your chosen model, precision, and context length fit your hardware.
What changed in vLLM 0.34.0?
The v0.34.0 release was published on November 16, 2026. The official release notes report 1,061 commits from 412 contributors and list ten new model integrations. The most operationally important changes are:
| Area | v0.34.0 change | Upgrade action |
|---|---|---|
| GPU execution | Model Runner V1 is removed; V2 is the only GPU runner. | Remove VLLM_USE_V2_MODEL_RUNNER=0. Port V1-only custom logits processors. |
| KV cache blocks | Default block size changes from 16 to 64. | Set --block-size 16 only when you intentionally need the previous behavior. |
| KV cache sizing | Extensible KV cache is enabled by default on CUDA and ROCm; default gpu_memory_utilization becomes 1.0 with this mode. | Review shared-GPU and external-allocation assumptions. |
| CUDA artifacts | Release wheels and images are CUDA 13.0; CUDA 12.9 release artifacts are no longer published. | Check the host driver and runtime before replacing the image. |
| ROCm artifacts | ROCm 7.2 release artifacts are dropped; ROCm 10.0 artifacts are provided. | Use the matching ROCm 10.0 wheel or image. |
| Python dependencies | Python 3.10 is dropped; newer Hugging Face dependencies are required. | Use a supported Python environment and resolve dependency pins. |
| Quantization | QuTLASS / fp_quant checkpoints and Marlin W4A8 support are removed. | Check the checkpoint format and migrate supported quantization configuration. |
These are release facts. Whether the new defaults improve your own workload is an operational question: the release notes describe larger KV capacity and better throughput in some high-concurrency tests, but also report higher per-token decode latency on some H200 workloads. Measure with your own model, prompt lengths, concurrency, and latency target.
Prerequisites and preparation
- A Linux host with a GPU/runtime combination supported by the vLLM 0.34.0 artifact you plan to use.
- Docker with NVIDIA Container Toolkit for the container path below, or a clean Python environment for the wheel path.
- Enough GPU memory for the model weights, runtime workspaces, CUDA graphs, and KV cache.
- A model checkpoint supported by vLLM 0.34.0. The official support table lists
Qwen/Qwen3-0.6Bas a Qwen3 example. - A maintenance window and a rollback image or environment.
Do not assume that a CUDA 13.0 container can run on every older GPU driver. Check NVIDIA's compatibility guidance for your exact host, and verify that nvidia-smi works before changing vLLM.
Step 1: Record the current deployment before changing it
Capture the package versions, container image, startup command, model revision, environment variables, and recent logs. Do not paste secrets or full environment dumps into tickets.
python -c "import vllm; print(vllm.__version__)" 2>/dev/null || true
python -m pip freeze > requirements-before-vllm-034.txt
nvidia-smi
docker ps --format 'table {{.Names}}\t{{.Image}}\t{{.Ports}}'
Search your deployment files for settings that changed. Adjust the path to match your repository:
grep -RInE 'VLLM_USE_V2_MODEL_RUNNER|block-size|gpu-memory-utilization|mamba-cache-mode|VLLM_MARLIN_INPUT_DTYPE|fp_quant' ./deploy/
Keep a copy of the old image tag and exact command. If you use Helm or Compose, save the rendered manifest as well as the source values file.
Step 2: Choose a reproducible installation path
Option A — use the pinned CUDA 13.0 Docker image
For a first upgrade test, a versioned image avoids mixing the vLLM package with unrelated host Python packages. The v0.34.0 release publishes vllm/vllm-openai:v0.34.0 for CUDA 13.0.
docker pull vllm/vllm-openai:v0.34.0
Generate a temporary API key and start a local-only test server:
export VLLM_API_KEY="$(python -c 'import secrets; print(secrets.token_urlsafe(32))')"
docker run --rm --gpus all --ipc=host \
-p 127.0.0.1:8000:8000 \
-v "$HOME/.cache/huggingface:/root/.cache/huggingface" \
-e VLLM_API_KEY \
vllm/vllm-openai:v0.34.0 \
--model Qwen/Qwen3-0.6B \
--served-model-name qwen3-0.6b \
--host 0.0.0.0 \
--port 8000 \
--max-model-len 8192
The container listens on all interfaces inside its network namespace, but Docker publishes the port only on host loopback because the mapping uses 127.0.0.1. Do not change that mapping to a public interface unless a firewall and authenticated reverse proxy are already in place.
If you need a gated Hugging Face model, pass an access token through a secret manager or a protected environment variable; do not bake it into the image or command history. The public Qwen3 smoke-test checkpoint does not require a token.
Option B — install into an isolated Python environment
Use the CUDA 13.0 PyPI wheel path documented for this release. A fresh environment makes rollback and dependency comparison much easier.
uv venv --python 3.12 --seed
source .venv/bin/activate
uv pip install "vllm==0.34.0" --torch-backend=auto
python -c "import vllm; print(vllm.__version__)"
The release notes also publish separate ROCm 10.0 and XPU installation paths. Do not use the CUDA command on an AMD or Intel GPU host. For ROCm, use the exact v0.34.0 wheel index shown in the release notes; for XPU, use the matching v0.34.0 index and PyTorch XPU index.
Step 3: Verify the server before tuning it
Wait for the model-loading logs to report that the API server is ready. In a second terminal, check the model list:
curl -s http://127.0.0.1:8000/v1/models \
-H "Authorization: Bearer $VLLM_API_KEY"
Then send a minimal chat-completion request:
curl http://127.0.0.1:8000/v1/chat/completions \
-H "Authorization: Bearer $VLLM_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"model": "qwen3-0.6b",
"messages": [
{"role": "user", "content": "Explain what a KV cache does in two sentences."}
],
"max_tokens": 120
}'
Expected result: a JSON response containing a model identifier, a choice with an assistant message, and usage information. Exact wording and token counts vary. This is a connectivity smoke test, not a performance benchmark.
Step 4: Remove settings that no longer work
4.1 Remove the Model Runner V1 switch
If your deployment sets VLLM_USE_V2_MODEL_RUNNER=0, remove it. v0.34.0 no longer includes Model Runner V1, so the old switch causes startup failure. If you maintain a custom V1 logits processor, the release notes point to vllm.v1.worker.gpu.sample.logits_processor as the V2 migration location. Treat that as a code migration, not a simple environment-variable change.
4.2 Decide whether to keep block size 64
The new default is 64 tokens instead of 16. For most new deployments, begin with the default and benchmark. If a workload or backend integration was explicitly tuned around the old block size, test compatibility with:
--block-size 16
This is a compatibility choice, not a universal optimization. The release notes state that ROCm paged attention still uses 16.
4.3 Understand extensible KV-cache sizing
With extensible KV cache, vLLM reserves a virtual address range and commits physical pages after warmup and CUDA graph capture. This lets it size the cache from memory actually free at that point rather than relying only on an earlier estimate. It is enabled by default on CUDA and ROCm when supported.
On the new default path, vLLM uses a gpu_memory_utilization default of 1.0. That does not mean every host should blindly allocate all memory: the release documentation specifically calls out memory used by other processes, later allocations, unified-memory devices, weight-transfer buffers, and KV connectors as reasons to leave headroom.
If another process shares the GPU, test an explicit limit such as:
--gpu-memory-utilization 0.90
If you need to compare against the previous estimate-based behavior, test:
--no-enable-extensible-kv-cache
Do not add both options by habit. Change one variable at a time, record the resolved cache size from startup logs, and compare throughput, time-to-first-token, inter-token latency, and out-of-memory frequency.
4.4 Check dependency and quantization changes
v0.34.0 requires huggingface_hub 2.0.0 or newer, tokenizers 0.23.3 or newer, and timm 1.0.30 or newer. Python 3.10 is dropped. A clean install should resolve compatible versions; custom constraints files may need updating.
Also audit your model artifacts:
- Checkpoints relying on the removed QuTLASS /
fp_quantpath no longer load through that implementation. - Marlin W4A8 support is removed. The release notes describe the Humming alternative using
--quantization hummingandVLLM_HUMMING_INPUT_QUANT_CONFIG. - ROCm 7.2 release wheels and images are no longer published for v0.34.0; use the ROCm 10.0 artifacts supported by this release.
Do not rename a quantization flag and assume the checkpoint is compatible. Validate the exact model format and backend against the release documentation.
Step 5: Run a meaningful upgrade test
After the smoke test passes, run a short staging matrix using the same model revision and representative prompts as the old deployment:
- Cold start: record model-load time, GPU memory after warmup, and any kernel compilation.
- Single request: measure time-to-first-token and inter-token latency.
- Representative concurrency: replay the normal number of simultaneous requests, not an arbitrary maximum.
- Long context: test the longest prompt your application actually sends.
- Mixed load: include other GPU processes if production shares the device.
- Restart and recovery: restart the container and verify that model cache, configuration, health checks, and client retries behave as expected.
Compare the same metrics on v0.33.x and v0.34.0. The release notes report that extensible KV cache can increase cache capacity and throughput at high concurrency, but also report a 6–18% increase in TPOT on some H200 models at concurrency 128. Those are release-reported observations, not a prediction for your GPU or model.
Step 6: Roll out gradually and keep a rollback path
Use a canary instance or a separate port. Route a small percentage of internal traffic to v0.34.0, compare error rates and latency, and only then move production traffic. Keep the v0.33.x image and configuration available until the new version has passed your normal soak period.
For Docker, rollback means stopping the v0.34.0 container and starting the previous pinned image with its saved command. For Python, recreate the previous virtual environment from the saved lock or requirements file. Avoid trying to downgrade a mutated environment in place.
Troubleshooting
| Symptom | Likely cause | What to check |
|---|---|---|
| Startup fails immediately after upgrade | Old V1 runner environment variable remains. | Remove VLLM_USE_V2_MODEL_RUNNER=0 and inspect custom V1-only extensions. |
| Different cache capacity or OOM behavior | Extensible KV cache and the new memory default changed sizing. | Check other GPU allocations; test a lower utilization limit or disable extensible cache for comparison. |
| Previously tuned cache layout behaves differently | Default block size changed from 16 to 64. | Test --block-size 16 in staging and compare results. |
| Dependency resolver reports conflicts | Old Python or pinned Hugging Face dependencies. | Use a fresh supported Python environment and update the minimum dependency pins. |
| Quantized model no longer loads | Removed quantization backend or unsupported checkpoint format. | Identify the checkpoint's quantization method; migrate only to a documented supported path. |
| ROCm image or wheel cannot be found | ROCm 7.2 artifacts are no longer published for this release. | Use the v0.34.0 ROCm 10.0 artifact if your hardware and host stack support it. |
| Browser app cannot call the API | Cross-origin access is restricted by default. | Configure only the exact trusted origin if browser access is required; CORS is not authentication. |
Security and privacy checklist
- Bind test deployments to loopback or a private network. Do not expose the inference port directly to the public internet.
- Use TLS and a reverse proxy or API gateway for production authentication, rate limiting, request-size limits, and access logging.
- Do not rely on
--api-keyalone. vLLM documents that its key protects supported API path prefixes but does not protect every route;/invocationsremains an important example. Restrict routes at the proxy and firewall. - Keep distributed communication and KV-transfer ports reachable only by trusted nodes.
- Do not enable development-only endpoints in production. Keep model downloads pinned to trusted repositories and revisions.
- Define retention and deletion rules for prompts, completions, uploaded files, and traces. Local inference does not automatically make application logs private.
- Keep Hugging Face tokens and API keys in a secret manager; rotate them if they appear in shell history, CI logs, or container inspection output.
Trade-offs: should you upgrade immediately?
Upgrade in staging now if you need one of the new model integrations, want the Model Runner V2-only path, or can test the new KV-cache behavior on your hardware.
Schedule a controlled migration if you use custom logits processors, quantized checkpoints, ROCm, or heavily tuned memory flags. These are the deployments most likely to need code or configuration changes.
Wait for a maintenance window if your current service is stable, the host is tied to CUDA 12.9 or ROCm 7.2 artifacts, or you cannot reproduce the current environment. A release being available is not a reason to skip a rollback plan.
Frequently asked questions
Is vLLM 0.34.0 the same as a new model release?
No. It is an inference-engine release that adds support for several model architectures and changes runtime behavior. Model weights and licenses remain separate.
Does v0.34.0 require CUDA 13.0?
The release's published NVIDIA wheels and default Docker image are CUDA 13.0 artifacts. Check the official compatibility documentation for your driver and GPU before upgrading. ROCm and XPU have separate artifacts.
Can I keep block size 16?
Yes. The release notes say to pass --block-size 16 to retain the previous behavior. Validate that setting on the exact platform and workload you use.
Should I set gpu-memory-utilization to 0.9?
Not automatically. With extensible KV cache enabled, vLLM measures memory after warmup. Leave the value unset unless other processes or allocations need headroom. Use a lower value only after testing the actual memory-sharing conditions.
Does the new release make every model faster?
No. Performance depends on architecture, GPU, precision, context length, concurrency, and scheduler settings. Treat release benchmarks as vendor-reported results for their stated test conditions.
Can I run vLLM 0.34.0 on an Apple Silicon Mac?
The standard CUDA server instructions in this guide are for Linux GPU deployments. vLLM documents a separate community-maintained vLLM-Metal path for Apple Silicon; do not assume the CUDA container or wheel instructions apply to it.
Related GyanAangan guides
- vLLM speculative decoding: EAGLE, MTP, DFlash2 and n-gram methods — understand draft-token strategies before tuning a new serving runtime.
- Build a local LLM gateway with LiteLLM — put a consistent API layer in front of multiple inference backends.
- Run Ollama in Docker with an NVIDIA GPU — compare container operations for another local inference runtime.
- Benchmark local GGUF models properly — build a repeatable measurement habit rather than relying on headline numbers.
Official references
- vLLM v0.34.0 release notes and full changelog
- vLLM v0.34.0 GPU installation guide
- vLLM v0.34.0 serve CLI reference
- vLLM v0.34.0 engine arguments
- vLLM v0.34.0 security guidance
- vLLM v0.34.0 supported models
- Qwen3-0.6B model card
- Official vLLM Docker deployment guide
Editorial note: Version-specific statements in this guide were checked against the vLLM v0.34.0 release notes and official documentation available on November 18, 2026. Commands are documentation-based examples and were not executed on a live GPU host. The article does not claim independent reproduction of vLLM's performance figures.
