SGLang 1.0 Release Tracker Explained: RadixAttention, HiCache, sgl-router and the 0.5.x Migration Plan (2026)
Updated: November 18, 2026. If you run an LLM endpoint in production, a major version number is not a reason to upgrade by itself. The important question is whether the new release makes serving more predictable without breaking the parts of your stack that already work.
That is the central story behind SGLang 1.0. The project's public release tracker describes 1.0 as a stability and compatibility release: harden existing serving paths, stop silently changing unsupported settings, validate requests more strictly, and define what operators can rely on across the 1.x series. The tracker is still marked in progress, so this guide treats 1.0 as an upcoming release—not as a version you can install today.
This deep-dive explains the architecture that makes SGLang interesting, what the 1.0 tracker says will change, and how to prepare a controlled migration from the current 0.5.x line without turning a production inference service into an experiment.
Who should read this
- Platform engineers operating GPU-backed model APIs.
- Teams serving long-context, multi-turn or tool-using workloads where prompts repeat across turns.
- Operators evaluating prefix caching, prefill/decode disaggregation or multi-replica routing.
- Teams using SGLang Diffusion alongside text or multimodal model serving.
- Developers currently using SGLang 0.5.x who want a migration checklist before 1.0 ships.
This is not a beginner guide to running a small local model on a laptop. SGLang is primarily a GPU inference and serving framework. Its official quickstart targets Linux with an NVIDIA GPU and a CUDA 13-compatible driver; other hardware paths have their own installation requirements.
1. Establish what is released—and what is not
As checked on November 18, 2026, the SGLang GitHub releases page lists v0.5.23 as the latest stable release. The separate SGLang 1.0 Release Tracker, opened on November 12, remains open and labels its work in progress. Its release-validation phase is still pending the release branch.
That distinction matters:
- Released today: use a published 0.5.x release such as v0.5.23 for a reproducible production baseline.
- Planned for 1.0: the tracker describes intended removals, router migration, stricter validation and compatibility policy. These are not all final release artifacts yet.
- Not a promise: the tracker does not provide a guaranteed ship date. Do not schedule a production cutover around an assumed date.
Primary references: official releases and the live 1.0 tracker.
2. Understand SGLang's serving architecture
SGLang is an open-source inference framework for language and multimodal models. The official documentation describes a runtime designed around RadixAttention, prefix caching and multi-GPU parallelism. Think of the system as several cooperating layers rather than a single model process:
- Request/API layer: accepts native SGLang requests or supported OpenAI-compatible requests.
- Scheduler and runtime: batches work, coordinates prefill and decode, and schedules GPU execution.
- Attention and KV cache: stores key/value state so the server does not have to recompute every token for every request.
- Parallel execution: distributes work across GPUs using supported parallelism strategies such as tensor, data, expert and pipeline parallelism.
- Routing and deployment: sends requests to workers, and in advanced topologies can coordinate separate prefill and decode pools.
The useful mental model is that serving quality is not just model tokens per second. It is also how much repeated context can be reused, how the scheduler handles concurrent requests, where cached state lives, and whether routing sends a returning conversation to a worker that still has its prefix.
3. RadixAttention and prefix reuse: why repeated prompts matter
Many production prompts contain a large shared prefix: a system prompt, policy text, tool definitions, a codebase summary, or the beginning of a long conversation. If every request recomputes that prefix, the server spends prefill compute on work it has already performed.
SGLang's RadixAttention and prefix-sharing runtime are designed to reuse matching prefix state. Conceptually:
- A request arrives with a token sequence.
- The runtime looks for an already-cached matching prefix.
- It can reuse the KV state for the shared portion when the relevant cache entry is available.
- Only the uncached suffix needs new prefill work.
- The resulting state may remain useful for later requests, subject to cache capacity and eviction.
This is particularly relevant to coding agents and other multi-turn tools. A repository map or a large system prompt may be repeated on every turn, while only the user's latest instruction changes.
Important: prefix caching does not make arbitrary prompts interchangeable. Tokenization, exact prefix content, model configuration and cache lifecycle matter. A small change near the start of a prompt can reduce the amount of reusable prefix. Measure your own traffic rather than assuming a fixed cache-hit rate.
4. HiCache: move useful KV state beyond GPU memory
GPU memory is valuable and finite. SGLang's HiCache provides a hierarchical KV-cache path so evicted prefixes can be retained outside the GPU pool instead of being discarded immediately. The current documentation and model cookbooks describe host-memory caching and, for selected validated configurations, an external storage tier such as Mooncake.
A practical way to think about the tiers is:
- GPU-resident KV: fastest access, constrained by accelerator memory.
- Host DRAM: a larger cache tier, but moving data back to the GPU has transfer and latency costs.
- External cache/storage: can be useful when a deployment needs a shared or larger cache tier, but it adds another service, network path and operational failure mode.
Do not enable every cache layer at once. Start with a workload that has repeated long prefixes, measure GPU memory, cache hit rate, time to first token and inter-token latency, then add one tier at a time. The SGLang cookbook's HiCache and Mooncake examples are model- and hardware-specific; copy their topology only when your model, image and accelerator match the validated recipe.
5. Aggregated serving versus prefill-decode disaggregation
In an aggregated deployment, the same worker handles both prompt prefill and token decode. This is simpler to operate and is usually the right baseline for a single-node service.
In prefill-decode (PD) disaggregation, prefill and decode run on separate SGLang server pools. The prefill pool processes the input prompt and produces KV state; that state is transferred to the decode pool, which generates output tokens. The tracker lists aggregated and PD-disaggregated serving as existing SGLang capabilities that 1.0 intends to harden.
| Topology | Useful when | Main trade-off |
|---|---|---|
| Aggregated | Single node, moderate traffic, simple operations | Prefill and decode compete for the same worker resources |
| PD disaggregated | Large sustained traffic where prefill and decode need separate scaling | More services, KV transfer networking, routing and failure handling |
PD disaggregation is not a free latency optimization. It adds transfer cost and distributed-systems complexity. It is worth evaluating when your measurements show that the two phases need different resource pools—not simply because a diagram looks more advanced.
6. Routing: why sgl-router is a major 1.0 migration item
The 1.0 tracker says sgl-router, a Rust router that distributes requests across SGLang workers (including PD deployments), is intended to replace sgl-model-gateway. The tracker also says the old gateway source is to be removed from the 1.0 repository and that SGLang Docker images will ship sgl-router instead.
This is a migration, not a guaranteed drop-in binary rename. The tracker explicitly notes that some features available only in sgl-model-gateway are not yet available in sgl-router. The flag mapping, before-and-after deployment examples and unsupported-feature list are still part of the migration work.
For multi-replica long-context workloads, cache-aware routing is important: a returning session is more likely to benefit when it reaches a worker that holds its reusable prefix. The official MiniMax M3.1 cookbook demonstrates a cache-aware router in a specific validated deployment. Treat that as an example of the design, not as a universal command for every model.
Migration rule: inventory your gateway flags and routes now. Do not replace sgl-model-gateway with an unpinned router build in production until the official migration guide, feature parity notes and release artifacts are published.
7. Run a reproducible SGLang 0.5.23 baseline
Before testing any 1.0 candidate, capture a known-good baseline. The following quickstart uses the published v0.5.23 Docker image and a small Qwen3 model for an API smoke test. It is a connectivity test, not a performance benchmark or a recommendation for production model sizing.
Prerequisites
- Linux host with a supported NVIDIA GPU and compatible CUDA 13 driver.
- Docker Engine and NVIDIA Container Toolkit configured for GPU containers.
- Enough free GPU memory for the selected model and runtime overhead.
- A Hugging Face token only if the chosen model is gated or private.
Step 1 — record the host and image
nvidia-smi
docker version
docker pull lmsysorg/sglang:v0.5.23
For reproducibility, record the image digest as well as the tag in your deployment notes. Tags are convenient labels; a digest identifies the exact image content.
Step 2 — start the server on a loopback-only host port
export SGLANG_API_KEY='replace-with-a-long-random-local-secret'
docker run --rm --gpus all --ipc=host --shm-size 32g \
-p 127.0.0.1:30000:30000 \
-e SGLANG_API_KEY \
lmsysorg/sglang:v0.5.23 \
sglang serve Qwen/Qwen3-0.6B \
--host 0.0.0.0 \
--port 30000 \
--api-key "$SGLANG_API_KEY"
The container listens on its interface, while Docker publishes the host port only on 127.0.0.1. This keeps the smoke-test endpoint local to the machine. If your GPU or shared-memory budget is smaller, use the installation guide's supported settings rather than blindly copying the 32 GB shared-memory allocation.
Step 3 — send an OpenAI-compatible request
curl http://127.0.0.1:30000/v1/chat/completions \
-H "Authorization: Bearer $SGLANG_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"model": "Qwen/Qwen3-0.6B",
"messages": [
{"role": "user", "content": "Explain KV caching in two sentences."}
],
"temperature": 0.2,
"max_tokens": 120
}'
Expected result: a JSON response containing a completion under choices[0].message.content. The first start may take longer while the model is downloaded and initialized. The exact startup time depends on the network, storage, model and hardware.
Step 4 — save the baseline
docker ps
docker logs --tail 200 <container-id>
docker inspect <container-id>
nvidia-smi
Record the image digest, model revision, launch arguments, GPU model, driver, CUDA environment, request payload, output correctness, time to first token, inter-token latency, throughput and peak memory. Keep the request set identical when you compare versions.
8. Prepare a 0.5.x-to-1.0 migration without upgrading blindly
The public tracker already identifies several areas that operators should prepare for. Because the final upgrade guide is still in progress, treat this as a planning checklist, not a final flag-by-flag migration document.
- Freeze the current environment. Save the Docker image digest or Python lockfile, model revision, container command, environment variables and GPU driver version.
- Inventory every launch option. Save the full command and environment. Search deployment manifests, Compose files, Helm values and scripts for deprecated SGLang flags and
SGLANG_*variables. - Inventory the gateway. Record every sgl-model-gateway flag, route, health check, authentication rule, PD route and cache-aware policy. Mark features that are gateway-only.
- Separate API and admin credentials. The tracker says sensitive administrative routes—such as weight updates, cache management and profiling—will require
--admin-api-key. It says--api-keywill no longer unlock those routes in 1.0. Plan separate secrets and access policies. - Test strict request validation. The 1.0 work aims to reject invalid request fields earlier with 4xx responses. Check clients that currently send undocumented or misspelled fields.
- Test streaming tool calls. The tracker lists work on parser correctness across stream chunk boundaries. If your agents depend on streamed function calls, test multiple calls, fragmented chunks, final-chunk arguments and reconnect behavior.
- Wait for official release artifacts. Use the published release tag, versioned docs, router migration guide and release notes—not a random main-branch container—as your candidate.
- Canary before cutover. Run the same model, traffic replay and evaluation set against the old and new versions. Keep a rollback image and configuration ready.
9. What changes in 1.0 according to the tracker?
The following items are explicitly described in the public tracker. Their status can change, so re-check the tracker before acting.
- Stricter configuration: unsupported settings should be rejected with an error that names the setting, instead of being silently adjusted or ignored.
- Earlier request validation: malformed fields should return a 4xx response at the entry point, with tests intended to ensure the server stays healthy.
- Gateway replacement: sgl-router is planned to replace sgl-model-gateway, but migration details and feature gaps remain under work.
- Admin API separation: sensitive admin endpoints are planned to require a dedicated admin API key.
- Deprecated option removal: flags and environment variables already deprecated in 0.5.x are planned for removal, subject to the tracker’s open decisions.
- Compatibility policy: the project is defining which CLI flags, environment variables, HTTP fields and Python Engine APIs are stable across 1.x and which remain experimental.
The 1.0 tracker also lists streaming tool-call parsing, request validation, router fuzzing, diffusion API stability and versioned documentation among release work. The point of the release is operational predictability—not a blanket claim that every workload will become faster.
10. Where SGLang Diffusion fits
SGLang's current documentation describes SGLang Diffusion as a built-in engine in the same repository and Python package, using the shared sglang generate and sglang serve commands. It covers image, video, audio and action generation, with OpenAI-compatible media APIs and a ComfyUI plugin.
This is useful if a team wants one serving project for autoregressive LLMs and selected media-generation pipelines. However, do not assume every diffusion model, API route or plugin is stable just because it appears in the documentation. The 1.0 tracker specifically calls for model-loading CI coverage and clear stable-versus-experimental labels for diffusion interfaces.
For a media deployment, choose a model from the current supported-model catalogue, follow its exact hardware recipe, and verify the API path and checkpoint license before building a product around it.
11. Trade-offs: when SGLang is a good fit
| Workload | Why consider SGLang | What to validate |
|---|---|---|
| Long multi-turn agents | Prefix reuse and hierarchical cache options can reduce repeated prefill work | Cache hit rate, memory pressure, session routing and cache-transfer cost |
| High-volume GPU serving | Scheduling, parallelism and disaggregated serving offer deployment controls | Model support, GPU topology, batch mix and tail latency |
| Separate prefill/decode pools | PD disaggregation lets operators scale phases independently | RDMA/network fabric, KV transfer, router behavior and recovery |
| Multimodal generation | SGLang Diffusion shares the project and package ecosystem | Model-specific hardware support and API stability |
| Small local experiments | Quickstart can expose an OpenAI-compatible endpoint | Installation footprint and GPU availability; lighter local runtimes may be simpler |
SGLang is not automatically better than vLLM for every model. Compare the exact model, quantization, hardware, concurrency, prompt lengths and API behavior. GyanAangan's vLLM speculative-decoding guide is a useful companion when your decision depends on draft-model strategies rather than serving architecture alone.
12. Troubleshooting checklist
Container starts but cannot see the GPU
- Run
nvidia-smion the host. - Verify NVIDIA Container Toolkit is installed and configured for Docker.
- Run a minimal GPU container before debugging SGLang itself.
CUDA or FlashInfer installation errors
- Check the SGLang installation guide for the supported CUDA lane. Current docs say CUDA 13 is required for current images and wheels; v0.5.19 is listed as the last CUDA 12 lane.
- For source or Python installs, verify
CUDA_HOME, compiler/toolkit and PyTorch compatibility. - If FlashInfer fails on a supported GPU, consult the official troubleshooting page before switching attention backends.
Model loads but the request fails
- Confirm the request model name matches the server's loaded model identifier.
- Check the server logs for tokenizer, model architecture or memory errors.
- Start with a short prompt and small output budget, then increase one variable at a time.
Long conversations become slow
- Inspect whether the repeated prefix is actually identical across turns.
- Measure cache hit rate and KV memory before enabling HiCache.
- Check whether cache-aware routing is configured and whether the session is reaching the expected worker.
Upgrade breaks a launch command
- Compare the exact CLI and environment variables against the versioned docs.
- Do not assume a warning means a flag is still supported in 1.0.
- Keep the 0.5.x image and manifest available for rollback until the new version passes canary tests.
13. Security and privacy checklist
- Bind development endpoints to loopback or a private network; do not publish port 30000 directly to the public internet.
- Use a reverse proxy or private service network with authentication, TLS and rate limits for shared endpoints.
- Store model-hub tokens and API keys in a secret manager, not in images, shell history, Git or logs.
- Use separate inference and administrative credentials. Prepare for the dedicated admin-key behavior described in the 1.0 tracker.
- Restrict profiling, cache-management and weight-update endpoints to operators.
- Redact prompts, tokens, cache credentials and user data from logs. The tracker includes work to redact HiCache storage credentials from server information, dumps, logs and gRPC.
- Apply request-size, output-token and concurrency limits. Validate tool calls in the application rather than treating model-generated arguments as trusted commands.
- For external cache tiers, protect credentials and network traffic, and define retention and deletion behavior for cached user context.
14. Frequently asked questions
Has SGLang 1.0 been released?
Not according to the official release page and tracker checked on November 18, 2026. The latest stable release shown is v0.5.23; the 1.0 tracker remains in progress.
Should I install the 1.0 branch now?
Do not use an unpinned development branch as a production replacement. If you need to evaluate ongoing work, isolate it from production, pin the commit, and expect interfaces to change before the release branch is validated.
Is SGLang only for DeepSeek?
No. The project supports multiple model families. The exact supported architectures and optimized recipes depend on the release, backend and hardware.
Is HiCache the same as ordinary GPU KV caching?
No. It adds hierarchical cache options beyond the GPU-resident pool, including host memory and selected external storage integrations. More tiers can improve reuse opportunities but add transfer and operational costs.
Do I need PD disaggregation?
No. Start with aggregated serving. Consider PD only when measured prefill and decode resource needs justify separate pools and the added networking and routing complexity.
Will sgl-router support every sgl-model-gateway feature?
The tracker explicitly says some gateway-only features are not yet available in sgl-router. Wait for the official migration guide and feature mapping.
Does SGLang run on Apple Silicon?
The project's platform documentation evolves and includes multiple hardware backends, but the quickstart used in this article targets Linux with NVIDIA GPUs. Check the current hardware-platform and installation pages for the exact supported path before choosing Apple Silicon.
Conclusion
SGLang 1.0 is worth watching because it is treating reliability, configuration predictability, API validation and routing migration as first-class release work. The underlying serving ideas—prefix reuse, hierarchical KV caching, PD disaggregation and model-aware routing—are already part of the 0.5.x ecosystem.
The safest operator strategy is straightforward: pin v0.5.23, capture a reproducible baseline, inventory gateway and deprecated options, separate administrative credentials, and build a canary test plan. Upgrade only after the official 1.0 tag, versioned docs, router migration guide and release-validation results are available.
Further reading on GyanAangan
- vLLM speculative decoding: EAGLE, MTP, DFlash2 and N-Gram — compare inference acceleration strategies.
- E2E Networks Blaze and NVIDIA Dynamo — understand another model-serving and routing architecture.
- Kimi K3.1 open weights and serving options — model and hardware considerations.
- DeepSeek DeepInfer — related inference-system developments.
Official sources and version references
- SGLang 1.0 Release Tracker #50045 — release status, upgrade notes, open decisions and migration work.
- SGLang official releases — stable tags and release notes.
- SGLang documentation — runtime, RadixAttention, supported models and platform overview.
- Official quickstart — Docker and API smoke test.
- Official installation guide — supported CUDA environment, Python, Docker and deployment options.
- SGLang Diffusion documentation — image, video, audio and action generation interfaces.
- MiniMax M3.1 cookbook — a model-specific example of long-context deployment, HiCache, Mooncake and cache-aware routing.
Editorial note: release status and tracker items in this article are a snapshot checked on November 18, 2026. Revisit the official tracker and release notes before executing a production upgrade. The commands are documentation-based examples; no live SGLang server or GPU benchmark was run for this article.
