llama.cpp 0.9.0 Released: FP8 E4M3, Faster KV Cache, New Models and Local Audio (November 2026)

Updated: November 7, 2026. A local model runtime can look like a simple executable, but every release can change the part that matters most: how model weights are represented, how requests share the KV cache, how devices move tensors, and whether the server can handle audio as well as text. The llama.cpp 0.9.0 release is worth reading for more than its version number. It combines new model-family support with low-level inference and API changes that affect both people running GGUF files and developers embedding ggml.

Released on November 4, 2026, llama.cpp v0.9.0 adds OCP FP8 E4M3 support, a new row-oriented Flash Attention operation for unified KV-cache workloads, batched backend tensor-copy APIs, four model architecture additions, Gemma 4 prefill optimizations, and expanded speech support. This guide separates what the release actually documents from what you should measure on your own hardware, then walks through a pinned build, a local API smoke test, and a safe upgrade checklist.

Who should read this

• Local-AI users: people running GGUF models on Linux, Windows, or Apple Silicon who want to know whether a new engine build changes their model options.
• Self-hosters: developers exposing llama-server to private applications through an OpenAI-compatible API.
• Inference engineers: people investigating KV-cache behavior, FP8 formats, multi-sequence serving, or backend copies.
• Application maintainers: teams shipping llama.cpp or ggml libraries inside desktop apps, model runners, and internal inference services.

Prerequisites

• Git, CMake, and a C/C++ toolchain supported by your operating system.
• Enough RAM or GPU memory for the GGUF model and the context you intend to run. There is no universal minimum: model size, quantization, context length, batch size, and offload settings all matter.
• A compatible GGUF model and any required multimodal projector or companion files, obtained from a source whose licence you have reviewed.
• For NVIDIA acceleration, a compatible CUDA toolkit and driver. For Apple Silicon, the project documents Metal as enabled by default on macOS builds.

The upstream build guide pinned to the v0.9.0 release commit is the authority for platform-specific compiler and backend requirements. Do not copy a CUDA command onto a machine without the matching toolkit, or assume a model that loads on a workstation will fit on a laptop.

1. Understand what changed in v0.9.0

The release notes group the work across ggml, model support, audio, server behavior, and the user interface. The most useful changes are not all user-facing toggles:

OCP FP8 E4M3 tensor support

llama.cpp adds an OCP FP8 E4M3 tensor type in saturating mode, conversion helpers, and the LLAMA_FTYPE_MOSTLY_F8_E4M3 file type. FP8 is a floating-point representation; it is not simply another name for a Q4 or Q8 integer quant. Whether it is useful depends on the checkpoint, conversion path, backend kernels, and hardware. Treat this as format support—not a promise that every existing GGUF will become smaller or faster when you upgrade.

A row-oriented Flash Attention operation for unified KV

The new flash_attn_ext_rows operation lets each query slice attend to its own list of key/value rows. The release describes it as making unified KV-cache handling more efficient for multi-sequence inference while restoring non-unified KV prefill performance. This matters most to engine developers and serving setups that process multiple sequences together. It does not mean every single-user prompt will automatically become faster.

Batched backend tensor-copy APIs

The release adds ggml_backend_tensor_copy_n, ggml_backend_tensor_set_n, and ggml_backend_tensor_get_n, implemented for Metal and CUDA. They supersede the older _2d API family. If your application calls ggml backend copy helpers directly, inspect the header and release diff, update your wrappers, and rebuild against the new library rather than assuming binary compatibility.

New model architectures and Gemma 4 work

The release lists Clover-35B-A3B-Preview, Limite 1B (Violetto), Metis-1.6, and Mistral Large 4 support. It also includes graph optimizations for Gemma 4 E2B and E4B prefill. “Supported” means the engine has implementation work for those architectures; it does not supply the model weights, grant a licence, or guarantee that every quantization and hardware backend is equally mature.

GyanAangan already has a separate Mistral Large 4 open-weights overview. Use that for model-level context; this article focuses on what the llama.cpp runtime release adds.

Audio generation and transcription support

Audio support expands in three distinct directions:

• Chatterbox TTS: a text-to-speech path is added to the multimodal subsystem.
• Nemotron 3.5 Transcribe: the release notes describe offline transcription support for 40 languages, with prompt context and speaker attribution.
• Pocket TTS: predefined voices are supported without voice cloning.

The server release notes also say GET /props reports predefined voices and the TTS endpoint accepts a voice name. Voice names and language support are model-dependent; do not assume that one model's options apply to another.

2. Decide whether you need to upgrade

Use the following decision guide before replacing a working installation:

3. Build the exact v0.9.0 source release

For a reproducible test, do not build an unpinned master checkout. Clone the upstream repository and check out the release tag:

git clone https://github.com/ggml-org/llama.cpp.git
cd llama.cpp
git checkout v0.9.0
git rev-parse --short HEAD

The final command should identify the checked-out commit. The official release page identifies the v0.9.0 tag commit as cd959ba. Keep that value in your deployment notes so you can reproduce the build later.

Build a CPU version

cmake -B build -DCMAKE_BUILD_TYPE=Release
cmake --build build --config Release -j

This follows the upstream CMake build pattern. On systems with limited RAM, reduce the parallel job count instead of using every logical CPU.

Build with NVIDIA CUDA

cmake -B build-cuda \
  -DGGML_CUDA=ON \
  -DCMAKE_BUILD_TYPE=Release

cmake --build build-cuda --config Release -j

Use a separate build directory when changing backends. The official build guide requires the CUDA toolkit and documents additional architecture-specific options. If CMake cannot find CUDA, fix the toolkit/compiler path first; do not interpret a CPU-only build as evidence that GPU acceleration is working.

Build on Apple Silicon

The upstream build guide says Metal is enabled by default on macOS. The standard CMake build is therefore the starting point:

cmake -B build -DCMAKE_BUILD_TYPE=Release
cmake --build build --config Release -j

Confirm the build output and runtime device list on the actual Mac. Available unified memory is shared by the operating system, model weights, KV cache, and other applications. A model's file size alone is not a complete memory estimate.

4. Verify the build before loading a large model

Check that the server binary exists and reports its build information:

./build/bin/llama-server --version
./build/bin/llama-server --help

If you built into build-cuda, use that directory instead. Confirm that the binary you execute is the one you just compiled—not an older package-manager copy earlier in your shell's PATH.

For a first smoke test, choose a small, known-compatible GGUF. Download it from the model publisher or a trusted GGUF conversion repository, read the model card, and preserve any required companion projector file. Avoid starting with a large MoE checkpoint simply because it appears in the release notes.

5. Start a local OpenAI-compatible server

With a model file already downloaded, start the server on loopback:

./build/bin/llama-server \
  --model /absolute/path/to/model.gguf \
  --host 127.0.0.1 \
  --port 8080 \
  --ctx-size 8192

Replace the model path and context size with values appropriate for your checkpoint and machine. The server's documented default port is 9931; this example deliberately chooses 8080. Binding to 127.0.0.1 keeps the service on the same machine.

In a second terminal, wait for the model to finish loading and check health:

curl http://127.0.0.1:8080/health

A ready server returns an OK status. A temporary unavailable response can mean the model is still loading; wait and retry rather than repeatedly launching duplicate server processes.

6. Send a chat-completions request

The server documents OpenAI-compatible chat-completions routes. Test with a small request before connecting an application:

curl http://127.0.0.1:8080/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "local-model",
    "messages": [
      {
        "role": "user",
        "content": "Explain the difference between FP8 and 4-bit integer quantization in three points."
      }
    ],
    "temperature": 0.2,
    "max_tokens": 160
  }'

Check that the response contains a completion and that the server log shows the expected model. Compatibility is not identical to every hosted provider: clients may rely on optional fields, tool-call conventions, or streaming details that need separate testing. Consult the server README for the endpoints and request fields supported by the exact build.

7. Try the new TTS route only with a compatible audio model

The v0.9.0 release adds audio-related model support, but a regular text-only GGUF does not become a speech model just because the server was upgraded. Load a model documented for speech generation, then inspect its properties:

curl http://127.0.0.1:8081/props

If the response includes a voices list, those are the predefined voice names the model exposes. The server documents a POST /tts route that returns audio bytes. For a model with predefined voices, a request has this shape:

curl -X POST http://127.0.0.1:8081/tts \
  -H "Content-Type: application/json" \
  -d '{
    "input": "This is a local text to speech test.",
    "voice": "VOICE_FROM_PROPS"
  }' \
  -o output.wav

Replace VOICE_FROM_PROPS with a real value returned by that model. If the model has no predefined voice list, omit the field or follow its model-specific speaker-reference instructions. Do not use a voice-cloning reference unless you have permission to use that recording.

For transcription, follow the exact Nemotron 3.5 Transcribe model and multimodal input instructions. The release establishes architecture support; it does not mean every text-generation command is a valid transcription workflow.

8. Review the API changes if you embed ggml

Application maintainers should treat the new tensor-copy functions as an integration task, not just a runtime update. Search your codebase for calls to the older _2d copy helpers and inspect the v0.9.0 headers and release diff. Then:

1. Build your application against the v0.9.0 headers and libraries in a clean build directory.
2. Update wrappers to the new batched copy APIs where appropriate.
3. Run tests for host-to-device, device-to-host, and device-to-device paths used by your application.
4. Test each backend you ship. The release notes explicitly identify Metal and CUDA implementations for the new batched APIs; do not infer identical behavior on every backend.
5. Keep a known-good previous build until your packaged application passes its regression suite.

The release also adds FP8 file-type declarations and audio-generation API entries in llama.h and mtmd.h. If you wrap those interfaces, update bindings and test their lifetime, error handling, and data-format assumptions rather than merely adding enum names.

9. Measure the changes instead of assuming a speedup

The release notes describe the unified KV-cache change as more efficient, but they do not publish a single speedup that applies to all models and machines. A useful before/after test keeps the following constant:

• Same model file, quantization, tokenizer, and chat template.
• Same prompt set, context size, output-token limit, and sampling settings.
• Same hardware, driver, backend, GPU offload, and power mode.
• Separate measurements for prompt processing, time to first token, decode tokens per second, total request latency, and concurrent-request throughput.
• Repeated runs with warm-up, plus a record of memory use and any errors.

Use the existing GyanAangan guide to llama-bench and SPEED-Bench for a more complete benchmark methodology. Do not compare a single-user run on v0.8.0 with a multi-user run on v0.9.0 and call the difference a release speedup.

10. Troubleshooting

The server binary is missing

Make sure the build completed and that you are using the correct build directory. You can explicitly build the target with:

cmake --build build --config Release -t llama-server

CUDA is not detected

Confirm the NVIDIA driver and CUDA toolkit are installed and visible to CMake. Reconfigure in a clean CUDA build directory with -DGGML_CUDA=ON, then inspect the configure output. A successful compile does not prove that the model is actually offloading layers to the GPU; check runtime logs and device information.

The model fails to load

Check that the GGUF is complete, the architecture is supported by this build, and all required split shards or multimodal companion files are present. Re-download corrupted files and compare hashes when the publisher provides them. Do not rename an unrelated model file to make it appear compatible.

The server reports an unsupported audio operation

Check that you loaded a speech-capable model, not a text-only checkpoint. Query /props, inspect the model's documented input/output modalities, and use the model's expected audio format. A voice value from a different model may be rejected.

Memory errors or very slow inference

Reduce context size, choose a smaller or more aggressively quantized compatible model, reduce batch/concurrency settings, or offload fewer layers to the GPU. Leave memory headroom for the KV cache and the operating system. Avoid treating swap as a performance solution.

The API works in curl but not in an SDK

Check the SDK's expected base URL, endpoint, model field, timeout, streaming parser, and authentication header. Test one non-streaming request first, then add streaming or tool calls. Different clients interpret optional OpenAI-compatible fields differently.

11. Security and privacy checklist

• Keep the listener private: use 127.0.0.1 for local development. Do not bind to 0.0.0.0 unless you intentionally need remote access and have added network controls.
• Authenticate remote clients: the server supports API-key configuration. Use a strong secret, store it outside source control, and rotate it if exposed.
• Use TLS across networks: place remote access behind a correctly configured reverse proxy or supported TLS setup. Do not send API keys over untrusted plaintext networks.
• Restrict CORS: allow only the frontend origins that need access. CORS is not authentication.
• Be cautious with tools and MCP: the server has optional tool and MCP-related features. Do not enable shell, file-write, or agent capabilities for untrusted users or arbitrary web pages. Use a separate runtime or container boundary and least-privilege filesystem mounts.
• Review model licences: engine support does not change the licence or usage restrictions of model weights, voice samples, or datasets.
• Protect logs and prompts: local inference reduces the need to send prompts to a hosted provider, but local logs, crash dumps, browser history, and backups can still contain sensitive information.

12. Trade-offs and limitations

• New support is not automatic optimization: FP8, unified KV improvements, and Gemma prefill changes help only where the model, workload, and backend use the relevant path.
• Architecture support is not a hardware recommendation: a supported large model may still require memory and multi-GPU capacity beyond a typical workstation.
• Audio support is model-specific: transcription, TTS, voice cloning, and predefined voices are separate capabilities with different model requirements.
• API changes affect downstream projects: applications using ggml headers directly should rebuild and test rather than assuming all old interfaces remain interchangeable.
• llama.cpp is an engine, not a complete product: model discovery, access management, monitoring, user quotas, and production operations remain the deployer's responsibility.

13. Frequently asked questions

When was llama.cpp 0.9.0 released?

The official GitHub release page records the v0.9.0 release on November 4, 2026.

Does v0.9.0 make every GGUF model faster?

No. The release includes targeted improvements, including a new unified-KV attention operation and Gemma 4 prefill work. Actual results depend on model, context, backend, and concurrency.

Does FP8 E4M3 replace Q4_K_M?

No. FP8 E4M3 is a floating-point tensor format, while Q4_K_M is an integer quantization scheme. They have different storage, accuracy, conversion, and hardware characteristics.

Can I run Mistral Large 4 on any laptop now?

No. Runtime support does not remove the model's memory and compute requirements. Check the model card, quantization, split files, and your available RAM/VRAM before downloading.

Can llama.cpp now generate speech?

The release adds support for specific TTS paths, including Chatterbox and Pocket TTS voice handling. You still need a compatible speech model and its required files.

Is the TTS voice parameter universal?

No. Query /props for the loaded model's available voices. Voice names and supported options are model-dependent.

Should I upgrade production immediately?

Not without a staged test. Pin the release, reproduce your current workload, run compatibility checks, and keep a rollback build until your application passes regression tests.

Where can I learn about routing several local models?

See the GyanAangan guide to llama.cpp router mode for multi-model serving behind one local API.

Official sources and further reading

1. llama.cpp v0.9.0 release notes — release date, feature list, API changes, model additions, and backend notes.
2. Build guide pinned to the v0.9.0 commit — CMake and backend build instructions.
3. Official llama-server documentation — server flags, API routes, authentication, health checks, and TTS.
4. OCP FP8 E4M3 support — implementation change.
5. flash_attn_ext_rows — unified KV-cache attention work.
6. Batched backend tensor-copy API — copy/set/get API change.
7. Audio generation API changes — Chatterbox and predefined-voice API work.
8. Nemotron 3.5 Transcribe support — transcription integration.
9. Pocket TTS predefined voices — voice support.
10. Hugging Face's Mistral Large 4 inference overview — model-specific serving context.

Final takeaway

llama.cpp 0.9.0 is a meaningful engine update because it touches several layers at once: tensor formats, multi-sequence attention, backend data movement, new model architectures, and audio serving. For a casual local-model user, the safest move is to build the release beside the current version and test one compatible model. For an inference engineer, the most interesting work is the KV-cache and backend API layer. For an application maintainer, the batched-copy API and model/audio changes deserve explicit regression coverage. Upgrade because a documented change solves a real problem in your workload—not because a new version number guarantees a speedup.