Benchmark Local GGUF Models Properly: llama-bench vs SPEED-Bench (2026)

A model that generates 90 tokens per second in a microbenchmark can still feel slow in a real application. Long prompts, model loading, context size, request queuing, and the time required to produce the first visible token all affect the experience. Comparing models using one headline number hides those differences.

This guide builds a repeatable benchmark workflow for local GGUF models with llama.cpp. You will use llama-bench to measure prompt-processing and token-generation throughput, then use the project’s SPEED-Bench client against llama-server to measure a more realistic request path. A small Python streaming probe adds time-to-first-token (TTFT) and total request latency measurements.

Publication/update date: October 30, 2026. The command examples were checked against the official llama.cpp documentation available on this date. The project changes frequently, so confirm flags with the version you have installed.

Who this guide is for

• Developers choosing between GGUF model sizes or quantizations for a laptop or workstation.
• Teams deciding whether CPU inference, GPU offload, or a different context size is the better trade-off.
• Engineers who need reproducible performance records before changing a local inference stack.

Prerequisites

• A working llama.cpp build that includes llama-bench and llama-server, or a compatible prebuilt package.
• At least one local GGUF model file that fits your available memory.
• Python 3 and requests for the optional streaming-latency probe.
• Enough free memory for the model, its context/KV cache, and the operating system. A model file fitting on disk does not guarantee it will fit in RAM or VRAM at the selected context size.

Official references: llama.cpp repository, official releases and platform builds, and the llama-bench README.

1. Understand which metric answers which question

The official llama-bench documentation describes prompt processing, generation, and combined tests, and notes that its measurements do not include tokenization or sampling time. Treat it as a controlled engine-level benchmark, not a complete simulation of your product. For application quality, pair this workflow with a separate regression suite such as our Promptfoo and Ollama evaluation guide.

2. Record the test environment before comparing anything

Create a small run sheet before launching the benchmark. Record at least:

• Operating system and CPU model.
• GPU model, VRAM, and inference backend (for example, CUDA, Metal, Vulkan, or CPU).
• llama.cpp version or commit and how it was installed.
• Exact model repository, file name, quantization, and file checksum if the result must be reproducible.
• Number of CPU threads, GPU layers offloaded, context size, batch settings, and KV-cache settings.
• Whether the machine was on battery, connected to power, thermally throttled, or running other heavy tasks.

Do not compare two models if one run used a different build, context depth, or offload configuration unless that difference is the variable you are intentionally testing. A result without its environment details is difficult to reproduce.

3. Confirm the benchmark binary and model files

Use the executable names from your installation. In a source build, they are commonly in build/bin; packaged builds may place them elsewhere.

./build/bin/llama-bench --version
./build/bin/llama-bench --help
./build/bin/llama-server --version
./build/bin/llama-server --help

On Windows, run the corresponding .exe files. If your package does not include llama-bench, use the official llama.cpp build/release instructions rather than downloading an executable from an unknown mirror. Keep model files from sources you trust and review the model’s license before redistribution or commercial use.

Set a model path in your shell. Example for macOS/Linux:

MODEL="/absolute/path/to/model-Q4_K_M.gguf"
test -f "$MODEL" && echo "Model file found"

For PowerShell:

$Model = "C:\models\model-Q4_K_M.gguf"
Test-Path $Model

Use a real path on your machine; these are placeholders, not a downloadable model recommendation.

4. Run a baseline with llama-bench

Start with a consistent, manageable test: a 512-token prompt-processing test and a 128-token generation test, repeated five times. Adjust those sizes to represent your actual workload later.

./build/bin/llama-bench \
  -m "$MODEL" \
  -p 512 \
  -n 128 \
  -r 5 \
  -o csv > baseline.csv

Open baseline.csv and preserve it with the run notes. The output contains build and hardware fields as well as the test configuration and throughput statistics. If you prefer a readable console table, omit -o csv or select the documented output format supported by your build.

In the results, look for separate prompt-processing and generation rows. Higher tokens-per-second is useful only when the test settings are equivalent. A result with a longer prompt, different context depth, or different GPU offload is not a fair apples-to-apples comparison.

5. Test prompt length and context depth

Real applications rarely send exactly 512 input tokens. A coding assistant may read a file or a long conversation; a document workflow may send retrieved passages. Test multiple prompt lengths and, when relevant, a deeper context:

./build/bin/llama-bench \
  -m "$MODEL" \
  -p 512,2048 \
  -n 128 \
  -r 5 \
  -o csv > prompt-sweep.csv

./build/bin/llama-bench \
  -m "$MODEL" \
  -p 512 \
  -n 128 \
  -d 4096 \
  -r 5 \
  -o csv > depth-4096.csv

The project’s current benchmark documentation describes -p for prompt length, -n for generated tokens, and -d for pre-filling the KV cache to a specified context depth. Confirm these options with llama-bench --help if your installed build differs.

Keep prompt length and context depth conceptually separate. The prompt is the input being processed; context depth tests a position within a longer context. Both can affect performance, but they represent different scenarios.

6. Compare CPU and GPU offload carefully

If your build supports your hardware backend, compare a CPU-oriented run with a GPU-offloaded run. For example, the following illustrates a layer-offload sweep; verify the option syntax and supported values for your build:

./build/bin/llama-bench \
  -m "$MODEL" \
  -p 512 \
  -n 128 \
  -ngl 0,20,40,99 \
  -r 5 \
  -o csv > offload-sweep.csv

Here, -ngl controls the number of model layers offloaded to the GPU in builds that expose this option. A value such as 99 is commonly used to request extensive offload, but it does not guarantee every layer fits or that every backend behaves identically. Watch for out-of-memory errors and confirm the backend reported in the output.

Do not assume GPU offload always wins. Some models or devices are constrained by memory bandwidth, memory capacity, transfer overhead, or backend support. The correct setting is the one that meets your latency and memory goals on your actual machine.

7. Add a real server to measure application-facing latency

Microbenchmarks are only one layer. Start llama-server on loopback so it is not exposed to your network while testing:

./build/bin/llama-server \
  -m "$MODEL" \
  --host 127.0.0.1 \
  --port 8080 \
  -c 4096 \
  -ngl 99

Use -ngl 0 for a CPU-only comparison, if supported by your build. The selected context size must be suitable for the model and available memory. Some model chat templates may require additional configuration; consult the server README and the model card rather than assuming every GGUF has identical chat behavior.

In another terminal, check that the server responds:

curl http://127.0.0.1:8080/health
curl http://127.0.0.1:8080/v1/models

The exact response and endpoint availability can vary with the version and build. If health checks fail, read the server logs first: model loading, unsupported architecture, insufficient memory, and port conflicts are common causes.

The official llama-server documentation describes its OpenAI-compatible endpoints and deployment options.

8. Use SPEED-Bench for a repeatable request workload

The llama.cpp repository also documents a SPEED-Bench client that sends requests to an already-running llama-server. It reports prompt throughput, generation throughput, average client-observed latency, and—when relevant—speculative-decoding acceptance metrics. Its benchmark dataset and options are documented in the repository; follow that setup rather than substituting random prompts between runs.

From a checkout of llama.cpp that contains the tool, install its documented requirements and review its help:

python tools/server/bench/speed-bench/speed_bench.py --help

Then run a small, controlled coding workload as a smoke test. The following follows the documented option pattern; use the dataset/configuration supplied by the version you checked out:

python tools/server/bench/speed-bench/speed_bench.py \
  --url localhost:8080 \
  --bench qualitative \
  --category coding \
  --osl 256 \
  --concurrency 1 \
  --limit 8 \
  --output run-coding.json

For a meaningful comparison, run the same benchmark category, sample limit, output length, and concurrency against each candidate. Save the JSON output. Increase the sample count only after the smoke test succeeds and the selected model can handle the intended context.

Do not treat a tiny smoke test as a statistically conclusive result. It verifies the path and configuration; use more samples and repeated runs for decisions. If the script or dataset layout has changed in your checkout, use the current official SPEED-Bench README as the source of truth.

9. Measure TTFT with a small streaming probe

Average request latency does not reveal how long a user waits before seeing the first text. The following optional probe sends one streaming chat request and records time to the first non-empty text chunk plus total duration. It measures the client-observed request path, not a pure model-kernel metric.

python -m pip install requests

import json
import time
import requests

URL = "http://127.0.0.1:8080/v1/chat/completions"

payload = {
    "model": "local-model",
    "messages": [
        {"role": "user", "content": "Explain what a Python context manager does in three sentences."}
    ],
    "temperature": 0,
    "max_tokens": 128,
    "stream": True,
}

started = time.perf_counter()
first_text_at = None
text_parts = []

with requests.post(URL, json=payload, stream=True, timeout=600) as response:
    response.raise_for_status()

    for raw_line in response.iter_lines(decode_unicode=True):
        if not raw_line:
            continue

        line = raw_line.strip()
        if not line.startswith("data:"):
            continue

        data = line[5:].strip()
        if data == "[DONE]":
            break

        try:
            event = json.loads(data)
        except json.JSONDecodeError:
            continue

        choices = event.get("choices") or []
        if not choices:
            continue

        delta = choices[0].get("delta") or {}
        chunk = delta.get("content", "")
        if isinstance(chunk, str) and chunk:
            if first_text_at is None:
                first_text_at = time.perf_counter()
            text_parts.append(chunk)

ended = time.perf_counter()
print("TTFT seconds:", None if first_text_at is None else round(first_text_at - started, 3))
print("Total seconds:", round(ended - started, 3))
print("Output preview:", "".join(text_parts)[:300])

Set the model field to whatever name or alias your server accepts. This probe assumes ordinary text chunks in an OpenAI-compatible streaming response; multimodal responses or a different server version may need a parser adjustment. Run it several times with the same prompt and compare the median, not only the best result. Python’s time.perf_counter() is designed for measuring elapsed durations; see the official Python time documentation.

10. Build a comparison table and make a decision

Do not publish a table containing only one tokens-per-second figure. Keep a record like this for each candidate:

Choose a candidate based on the workload. For interactive chat, TTFT and decode speed may dominate. For long-context summarization, prompt processing and memory use can matter more. For a multi-user service, concurrency and queueing become important, so a single-request benchmark is not enough.

If you are deciding between local inference and a hosted API, add electricity, hardware cost, utilization, and maintenance to the calculation. See our cloud GPU vs LLM API cost guide. If the choice is between runtimes rather than models, the Unsloth vs Ollama comparison explains their different roles.

11. Troubleshooting checklist

• Model does not load: verify the path, file integrity, supported architecture, and available RAM/VRAM. Check server logs rather than assuming the benchmark command is wrong.
• GPU results look like CPU results: inspect the reported backend and GPU-layer count; confirm your binary was built with the intended backend.
• Results vary widely: close other workloads, keep power and thermal conditions stable, allow warm-up, and increase repetitions. Record standard deviation instead of selecting the best run.
• Long prompts fail: lower context or prompt length, check KV-cache memory, and confirm the server context is large enough for the test.
• TTFT probe returns no text: inspect the HTTP response and server logs, confirm the endpoint and model field, and check whether the response schema differs from the assumed streaming format.
• SPEED-Bench fails to start: check the current tool README, Python dependencies, dataset setup, URL, and server availability. The benchmark tool may evolve with the repository.

12. Security, privacy, and benchmark integrity

• Bind a local benchmark server to 127.0.0.1 unless remote access is deliberately configured. Do not expose an unauthenticated inference endpoint to the public internet.
• Use synthetic or non-sensitive prompts. Logs, benchmark datasets, and saved JSON results can contain prompt text or model output.
• Do not run unknown model-conversion scripts or binaries with elevated privileges. Download releases from the official repository and review the license and provenance of model files.
• Keep benchmark prompts and settings identical when comparing candidates. Label any configuration change explicitly.
• Never present a result from one machine as a universal speed ranking. Hardware, build flags, quantization, context, and workload materially affect performance.

FAQ

Is llama-bench enough to choose a local model?

It is a good engine-level starting point, but not sufficient by itself. Add server-level latency and memory observations, then test output quality on representative tasks.

Why can prompt-processing speed be high while chat still feels slow?

Prompt processing and output generation are different phases. The user also experiences model loading, time to first token, network/client overhead, and the duration of the full response.

Can I compare Q4 and Q8 quantizations directly?

Yes, if you record the exact files and use the same hardware, build, context, and benchmark settings. Compare quality as well as speed and memory use; a throughput result does not measure answer quality.

Should I use the fastest result from five runs?

No. Use the reported average and variability, or summarize repeated end-to-end trials with a median and range. The fastest run may simply benefit from a favorable transient condition.

Does this prove which model is most accurate?

No. Performance benchmarks measure speed under specified conditions. Use a separate, representative quality evaluation and inspect failures before choosing a model for production.

Conclusion

A trustworthy local-LLM benchmark is a small experiment, not a screenshot of tokens per second. Use llama-bench to isolate prompt processing and generation, use SPEED-Bench to test a realistic server request path, and measure TTFT when responsiveness matters. Record the build, hardware, model file, quantization, context, and settings every time. That makes model selection repeatable—and makes your published comparisons useful instead of misleading.

Primary sources

• llama.cpp llama-bench documentation
• llama.cpp SPEED-Bench documentation
• llama-server documentation
• Official llama.cpp releases
• Python time module documentation