Run One GGUF Across Multiple GPUs with llama.cpp: Layer vs Tensor Split and Server Setup (2026 Guide)
Two graphics cards do not automatically make an AI model twice as fast. They can, however, let you load a model that will not fit on one card, keep more of its weights and KV cache on accelerators, or improve throughput when the split matches the workload. The key is choosing the right parallelism mode instead of guessing at a tensor-split ratio.
This guide is an evergreen technical deep-dive, updated November 19, 2026. In the official llama.cpp release feed checked today, the newest listed item is the b12532 pre-release build, dated November 19, with an OpenCL change for adjusting KV padding in decomposed Flash Attention. That is a narrow development build, not a new stable model release or confirmation that llama.cpp 1.0.0 has shipped. Rather than turn that small change into an exaggerated launch story, this guide explains a practical multi-GPU workflow using the project's current documentation.
Who this guide is for
- Developers with two or more supported GPUs who want to serve one GGUF model across them.
- People with mixed VRAM capacities who need a deliberate layer split.
- Teams deciding between pipeline-style layer splitting and experimental tensor parallelism.
- Anyone exposing a local llama.cpp model through an OpenAI-compatible API.
What you will build
You will inspect the devices visible to llama.cpp, start with the default layer split, assign a custom ratio to GPUs with different memory sizes, compare the experimental tensor split where it is supported, and expose the resulting model through llama-server. You will also benchmark prompt processing and token generation separately so a configuration is not declared faster based on one convenient number.
Prerequisites
- A machine with at least two GPUs supported by the backend you intend to use, such as a compatible CUDA, ROCm, Metal, Vulkan or other supported build.
- A llama.cpp build that includes the correct backend for those devices. A CPU-only binary will not become a CUDA build just because the machine has NVIDIA cards.
- A local GGUF model file that llama.cpp supports, plus enough combined accelerator and system memory for model weights, runtime buffers and the KV cache.
- Recent GPU drivers and a working vendor runtime. Confirm the backend before tuning model flags.
- Basic shell access. The examples below use NVIDIA device names such as
CUDA0andCUDA1; use the exact names reported by your own binary on other backends.
Multi-GPU is not a magical pooled-memory device. llama.cpp assigns model layers or tensor segments to participating devices. The model still has to fit the actual placement, and context length, batch size, concurrency, KV-cache type and compute buffers consume memory too.
1. Check which GPUs llama.cpp can actually see
Do not start by guessing device numbers. Run the device-list command from the same build you plan to use for serving:
llama-cli --list-devices
On a CUDA machine, you can cross-check driver visibility with:
nvidia-smi
Record the device names, total VRAM and currently free VRAM. If llama-cli --list-devices shows only CPU, fix the build or runtime first. Changing --split-mode cannot enable a backend that was not compiled into the binary.
2. Understand the split modes before choosing one
| Mode | What it does | Practical guidance |
|---|---|---|
none | Uses one selected GPU. | Useful for a baseline or to isolate a device. |
layer | Pipeline-style split: each GPU owns a contiguous set of layers and the KV cache for those layers. | The default and most compatible starting point. Usually the first choice when the main goal is fitting a larger model or increasing batch throughput. |
tensor | Experimental tensor parallelism: splits weights and KV across participating GPUs, with cross-GPU communication during layer computation. | Consider when token-generation latency matters and the GPUs have a fast interconnect. Validate architecture support and output quality before relying on it. |
row | Legacy row-split path. | The current multi-GPU guide marks it deprecated; do not choose it for a new deployment. |
The difference matters. Layer splitting sends activations from one GPU's layer range to the next. It reduces repeated cross-GPU communication and can work well when cards are connected by ordinary PCIe, but a small batch may not keep every stage equally busy. Tensor parallelism divides work inside layers, which can help latency-sensitive generation, but it performs more cross-GPU reductions. Slow PCIe links, mismatched cards or unsupported model architectures can erase the benefit.
3. Start with the default layer split
For a first multi-GPU run, use layer mode and let llama.cpp discover the visible devices. The official guide's simplest recipe is just to launch the model with the normal CLI or server command; layer splitting is the default and automatic fitting is enabled by default.
llama-server -m /models/your-model.gguf --host 127.0.0.1 --port 8080
Watch the startup log. Confirm that buffers or layers are being assigned to more than one GPU and that the model finishes loading. If only one device is used, inspect the device list, build backend and any device-filtering environment variables before changing split ratios.
4. Set an explicit ratio for GPUs with different VRAM
Suppose the first card has 24 GB of VRAM and the second has 16 GB. A starting ratio of 3,2 represents 60% and 40% of the split, matching the cards' nominal memory ratio. The values follow the order of devices in --device; they are proportions, not percentages that must add to 100.
llama-server -m /models/your-model.gguf --device CUDA0,CUDA1 --split-mode layer --tensor-split 3,2 --host 127.0.0.1 --port 8080
Do not treat 24 GB plus 16 GB as 40 GB of usable model memory. Leave headroom for the KV cache, compute buffers, display usage, other processes and the driver's allocations. If the smaller card fills first, reduce its share or reduce the workload's context/concurrency. If your devices have equal memory, the default automatic split may be sufficient; only add an explicit ratio when the logs or memory use show that it is needed.
5. Choose devices explicitly when the machine has more accelerators
If the host has an integrated GPU, a display GPU or another accelerator you do not want llama.cpp to use, specify the devices by their reported names:
llama-server -m /models/your-model.gguf --device CUDA0,CUDA1 --split-mode layer --host 127.0.0.1 --port 8080
For a single-GPU baseline, choose one device and use none mode:
llama-server -m /models/your-model.gguf --device CUDA1 --split-mode none --host 127.0.0.1 --port 8080
Use the device index semantics of your installed build and confirm the startup log. Device ordering can change when environment variables such as CUDA_VISIBLE_DEVICES filter or reorder devices.
6. Test tensor parallelism only when the model and hardware qualify
Tensor mode is not a universal “faster” switch. The current multi-GPU documentation labels it experimental, requires Flash Attention, and limits KV-cache types to non-quantized formats such as f16, bf16 or f32. Quantized KV cache is not implemented for this mode in the documented path. Auto-fit is also not supported with tensor mode, so plan memory explicitly.
llama-server -m /models/your-model.gguf --device CUDA0,CUDA1 --split-mode tensor --tensor-split 1,1 --n-gpu-layers all --flash-attn on --cache-type-k f16 --cache-type-v f16 --fit off --host 127.0.0.1 --port 8080
This is a test configuration, not a production recommendation. It may fail if the architecture has no tensor-split implementation. The current allow-list excludes a number of MoE, hybrid, state-space and other architectures; examples in the documentation include DeepSeek2, GLM-DSA, Nemotron-H, Mamba, MiniCPM3 and T5. Check the current list before selecting this mode. If it fails with an unsupported-architecture error, return to --split-mode layer.
7. Tune memory pressure without changing five variables at once
After the model loads, increase the workload gradually. Context length and parallel request slots can materially change KV-cache use. For a server that runs out of memory during prefill, try a smaller context size or lower concurrency before changing the model quantization.
llama-server -m /models/your-model.gguf --device CUDA0,CUDA1 --split-mode layer --tensor-split 3,2 --ctx-size 8192 --parallel 1 --host 127.0.0.1 --port 8080
The exact context and concurrency values depend on the model, architecture, quantization and available memory. Treat 8192 and one parallel slot as a conservative starting example, not a minimum or a promise that every model will fit. If your model needs a larger context, measure the KV-cache cost and increase it in controlled steps.
8. Verify the OpenAI-compatible server endpoint
Once the server reports that it is listening, inspect the model list:
curl http://127.0.0.1:8080/v1/models
Then send a small chat-completions request:
curl http://127.0.0.1:8080/v1/chat/completions -H "Content-Type: application/json" -d '{"model":"gpt-3.5-turbo","messages":[{"role":"user","content":"Reply with OK and one sentence describing your role."}]}'
The model field in the documented llama.cpp example uses an OpenAI-style model string; applications should use the model identifier or alias expected by their server configuration. A successful response confirms that the API path works. It does not prove the chosen split is faster or that every OpenAI API feature is implemented identically.
9. Benchmark the workload you actually care about
Use llama-bench to compare prompt processing and generation separately. Keep the model file, quantization, prompt length, generated-token count, batch settings, device set and build constant. Run the same benchmark with layer mode and, only if supported, tensor mode.
llama-bench -m /models/your-model.gguf -sm layer -p 512 -n 128 -r 5 -o json
llama-bench -m /models/your-model.gguf -sm tensor -fa on -ctk f16 -ctv f16 -p 512 -n 128 -r 5 -o json
Record prompt-processing tokens per second and generation tokens per second separately. Also note peak VRAM per device, load time, context length and whether the run stayed entirely on GPU. llama-bench measurements do not include every part of a real application path, such as tokenization and sampling, so follow them with an end-to-end server test using representative prompts and concurrency.
Do not copy a benchmark number from another machine and assume it applies to yours. PCIe generation, lane width, NVLink, GPU architecture, driver, model architecture and quantization all affect the result. The goal is to find the best configuration for your own workload, not to produce a universal ranking.
Layer split vs tensor split: which should you choose?
- Choose layer mode first when the model does not fit on one card, the GPUs have different memory sizes, or you want a stable starting point with modest interconnect requirements.
- Test tensor mode when the model architecture is supported, Flash Attention is available, the GPUs communicate quickly, and single-request generation latency matters more than simplicity.
- Keep one GPU when the model already fits and multi-GPU communication would add overhead without a demonstrated benefit.
- Do not use row mode for a new deployment; the current guide marks it deprecated.
Troubleshooting
| Symptom | What to check |
|---|---|
| Only one GPU is used | Run --list-devices, confirm the correct backend build, inspect CUDA_VISIBLE_DEVICES, and pass the exact device names with --device. |
| CUDA out of memory during load | Reduce the share assigned to the smaller card, lower context or concurrency, close other GPU workloads, or use layer mode instead of tensor mode. |
| Tensor mode says Flash Attention is required | Use --flash-attn on only if the model/backend supports it, or switch back to layer mode. |
| Tensor mode rejects the architecture | The architecture is not implemented for this mode. Use the default layer split. |
| Tensor mode rejects a quantized KV cache | Use a documented non-quantized KV type such as f16, bf16 or f32, or use layer mode if quantized KV is important. |
| Multi-GPU is slower than one GPU | Check interconnect bandwidth, batch size and synchronization overhead. Try layer mode and compare prompt and generation performance independently. |
| NCCL warning on CUDA tensor mode | Confirm the CUDA build found NCCL. The project notes that missing NCCL can make cross-GPU performance suboptimal; NCCL is not automatically bundled with CUDA. |
| Crashes after enabling CUDA P2P | Unset GGML_CUDA_P2P. The project warns that P2P can be unstable on some motherboard, BIOS and IOMMU configurations. |
Security and privacy considerations
- Keep a development server bound to
127.0.0.1. Do not expose an unauthenticated inference port directly to the public internet. - If other machines need access, use a private network or VPN and an authenticated reverse proxy with firewall rules and request limits. Do not assume that a private LAN is automatically trusted.
- Use model files from sources you trust, keep the model revision and file hash with your deployment record, and scan downloaded artifacts according to your organization's policy.
- Do not place secrets, customer records or production credentials in benchmark prompts. Logs and request traces can contain sensitive prompt content.
- Restrict who can change model paths, server flags and the process environment. A local inference process may have access to files and resources beyond the model itself.
What about using GPUs on another computer?
llama.cpp also has an RPC backend that can expose accelerator devices on remote hosts. However, the official RPC README explicitly labels it proof-of-concept, fragile and insecure, and warns against running the RPC server on an open network or in a sensitive environment. This guide therefore focuses on multi-GPU operation inside one host. Treat remote RPC as experimental lab infrastructure only, not as a safe production cluster recipe.
FAQ
Does llama.cpp combine two GPUs into one large VRAM pool?
No. It places layers or tensor segments across devices. Each device still has its own memory budget, and KV cache and runtime buffers must fit the chosen placement.
Can I use two GPUs with different VRAM sizes?
Yes, for supported backends and models. Start with layer mode and use --tensor-split to express a proportional allocation, then adjust based on actual free memory and startup logs.
Is tensor parallelism always faster than layer splitting?
No. Tensor mode communicates across GPUs more frequently and is experimental. It can lose to layer mode when the interconnect is slow, the model is unsupported or the workload is not latency-sensitive.
Can I use quantized KV cache with tensor mode?
The current multi-GPU guide says quantized KV cache is not implemented for tensor split. Use a non-quantized cache type or choose layer mode.
Can I expose the server to another device?
Yes, but do not expose the port directly to the public internet. Put remote access behind a VPN or private network, authentication, firewall rules and request limits.
Does this guide guarantee a model will fit?
No. Model weights, KV cache, context length, batch size, compute buffers and other GPU processes all affect memory use. Validate with your exact model and hardware.
Related GyanAangan guides
- llama.cpp 0.9.0: FP8, KV cache and local audio changes — release context and model/backend support.
- Benchmark local GGUF models properly — compare llama-bench and broader serving metrics.
- Build a local LLM gateway with LiteLLM — put local inference endpoints behind one API layer.
- Access a local AI API remotely with Tailscale Serve — private remote access patterns for local inference.
Official sources and further reading
- llama.cpp: Using multiple GPUs — split modes, flags, recipes and architecture limitations.
- llama.cpp server documentation — server flags and OpenAI-compatible endpoints.
- llama-bench documentation — benchmark parameters and output formats.
- llama.cpp RPC backend README — remote-device architecture and its explicit security warning.
- Official llama.cpp releases — stable releases and pre-release builds. Checked November 19, 2026.
- llama.cpp build guide — backend-specific build options.
Final takeaway
Start with --split-mode layer, verify both GPUs are active, and use --tensor-split only when the memory distribution needs to be controlled. Treat --split-mode tensor as an experiment that must pass architecture, memory, interconnect and quality checks. Then benchmark the exact prompt and generation workload your application will serve. A measured, stable layer split is more useful than a theoretical speedup that your hardware cannot deliver.
