Ollama GPU Not Being Used in 2026: How to Diagnose and Fix CPU Fallback

When Ollama suddenly uses your CPU instead of your NVIDIA, AMD, or other supported GPU, the frustrating part is that the model may still work. You can get a response, watch RAM climb, and assume Ollama is “using the GPU a little.” The more useful question is different: Did Ollama actually discover a usable GPU, select a GPU inference backend, load model layers onto it, and keep enough VRAM available for the context?

This guide walks through that chain in order. It is designed for the common symptoms: ollama ps shows CPU instead of GPU, nvidia-smi shows no Ollama process or almost no VRAM use, Ollama reports zero VRAM, a Docker deployment cannot see the GPU, or an update causes a previously working GPU to fall back to CPU.

First: prove whether Ollama is actually using the GPU

Do not start by changing environment variables. First establish the baseline.

NVIDIA: check the GPU outside Ollama

nvidia-smi
nvidia-smi -L

You want the driver to report the GPU normally. If nvidia-smi itself fails, Ollama is not the first problem to fix.

Then start a model in one terminal:

ollama run qwen3:8b

In another terminal, watch the device while the model is actively generating:

nvidia-smi -l 1

Look for an Ollama process and a meaningful increase in GPU memory while inference is active. GPU utilization can be bursty, so a momentary 0% reading does not prove that inference is CPU-only.

Use `ollama ps` as a second check

ollama ps

The output can show whether a loaded model is on CPU or GPU and can expose the practical placement of the model. Treat it as complementary evidence rather than replacing OS-level monitoring.

Observation	What it usually tells you	Next test
`nvidia-smi` cannot see the GPU	Driver or host problem	Fix NVIDIA installation first
GPU is visible, Ollama uses CPU	Ollama discovery/backend/configuration problem	Inspect Ollama logs
Some VRAM is used but model is mostly CPU	Partial offload or insufficient usable VRAM	Check model size, context and placement
GPU memory rises but utilization looks low	Could still be GPU inference	Observe during active generation and inspect `ollama ps`
Works on host, fails in Docker	Container GPU passthrough problem	Test GPU access from the container

Read Ollama's startup and inference logs

Ollama's own troubleshooting documentation recommends debug logging when GPU discovery or library selection is not behaving correctly. Start with:

OLLAMA_DEBUG=1 ollama serve

On Windows, if you use the desktop application, the official troubleshooting documentation recommends quitting the running app from the tray and launching it with OLLAMA_DEBUG=1 so the diagnostic output is captured.

Look for messages describing inference compute, the selected library, GPU name, available VRAM, or a fallback to library=cpu. A particularly useful distinction is between “GPU exists on the machine” and “Ollama initialized a GPU backend.”

What a useful log tells you

A healthy discovery path should contain evidence that Ollama initialized a CUDA, ROCm, Vulkan, Metal, or another supported accelerator backend appropriate to the platform. A CPU-only line such as inference compute id=cpu library=cpu means you need to investigate discovery/backend initialization before tuning model parameters.

Ollama's troubleshooting documentation also notes that it bundles multiple LLM libraries and normally chooses one based on the capabilities it detects. If autodetection fails, the documentation describes forcing a specific library as a troubleshooting workaround. This is an advanced step: do not jump to it before checking drivers, permissions and logs.

NVIDIA GPU: fix the host before changing Ollama

1. Verify the NVIDIA driver

nvidia-smi

If the command reports a healthy device, note the GPU model, driver version and reported CUDA compatibility. You do not need the CUDA toolkit merely because Ollama uses CUDA libraries, but the installed NVIDIA driver must be compatible with the backend Ollama is trying to use.

2. Check the NVIDIA UVM driver on Linux

Ollama's official troubleshooting documentation specifically calls out the NVIDIA UVM driver as a possible discovery problem. On Linux, inspect and, when appropriate, reload it:

lsmod | grep nvidia_uvm
sudo nvidia-modprobe -u

If necessary, the documented troubleshooting path includes reloading the module:

sudo rmmod nvidia_uvm
sudo modprobe nvidia_uvm

Do this only when you understand the impact on other GPU workloads. A reboot is often the simpler diagnostic if the machine is not hosting important GPU jobs.

3. Check kernel errors

sudo dmesg | grep -i nvrm
sudo dmesg | grep -i nvidia

Driver errors here are evidence that the problem is below Ollama. Fix the driver or kernel integration before trying model-specific settings.

Docker: the GPU must work inside the container

A common mistake is proving that the host has a GPU and then assuming a container automatically inherits it. It does not. Ollama can only use a GPU that its container runtime exposes.

For NVIDIA, Ollama's troubleshooting documentation recommends this direct test:

docker run --gpus all ubuntu nvidia-smi

If this test cannot see the GPU, stop debugging Ollama. Fix the NVIDIA container runtime, GPU passthrough, or Docker configuration first.

Once the test succeeds, inspect your Ollama container configuration. Depending on your deployment, that may mean using Docker's GPU device support or the appropriate NVIDIA Container Toolkit configuration. Avoid adding random CUDA environment variables until the basic container test works.

AMD GPU: check permissions and device access

On Linux, Ollama's official troubleshooting documentation notes that AMD GPU access commonly depends on permission to access /dev/kfd, typically through the appropriate video and/or render groups.

ls -l /dev/kfd
ls -l /dev/dri
groups

If the Ollama service runs under a dedicated user, check that that service user has the required device access. Testing as your interactive login user is not enough if systemd launches Ollama as another account.

System service problems: your shell environment may not be the service environment

This is one of the most common reasons a manual test appears to work while the normal Ollama service still uses CPU.

For example, this may work in an interactive shell:

CUDA_VISIBLE_DEVICES=0 ollama serve

But if Ollama is already running as a systemd service, your shell variable does not retroactively change that process.

Check the service:

systemctl status ollama
systemctl cat ollama

For a deliberate service-level environment override on a systemd installation:

sudo systemctl edit ollama

Then configure only the variables you actually need, for example:

[Service]
Environment="CUDA_VISIBLE_DEVICES=0"

Reload and restart:

sudo systemctl daemon-reload
sudo systemctl restart ollama

Then re-check the logs. Do not assume the variable was applied merely because it appears in your terminal.

Be careful with CUDA_VISIBLE_DEVICES

CUDA_VISIBLE_DEVICES can be useful for selecting which NVIDIA devices a process can see, but it is not a universal “make Ollama use my GPU” switch. If the GPU is not initializing, hiding devices can make diagnosis harder.

There have also been Ollama GitHub reports involving GPU selection, mixed NVIDIA/AMD systems, multi-GPU scheduling and changes between releases. These reports are useful evidence for version-specific behavior, but they should not be treated as proof that every installation has the same bug.

For a simple single-GPU machine, first let Ollama autodetect the hardware. Restrict device visibility only when you have a concrete reason, such as reserving one GPU for another workload.

Check whether the model simply does not fit comfortably in VRAM

“GPU not being used” and “model does not fit entirely in VRAM” are different situations.

Ollama can use CPU/GPU placement when the workload cannot fit completely on the accelerator. A large model, long context, multiple parallel requests, or a large KV cache can consume memory beyond the raw model file size.

Before changing anything, estimate the memory requirement using the model's actual quantization and context settings. Our VRAM calculation guide covers the practical calculation in detail.

Situation	Likely result	What to do
Model comfortably fits in VRAM	Mostly or fully GPU-resident	Verify with logs and `ollama ps`
Model is close to VRAM limit	Partial offload or allocation failure	Lower context, use a smaller quantization/model, or add memory headroom
Model is much larger than VRAM	CPU/GPU split or inability to load	Choose a smaller model/quantization or a system with more memory
Several models are already loaded	Less free VRAM	Unload unused models and retest

Context length can look like a GPU problem

Model weights are only part of inference memory. KV cache and runtime buffers also consume memory, and the requirement grows with context and concurrency.

If a model worked at a small context but begins falling back or failing at a much larger context, test a conservative context first. This is especially important on GPUs with limited VRAM.

Our Local RAG vs long-context guide explains why simply increasing context is not always the best solution for a document-heavy workflow.

Do not confuse low GPU utilization with CPU inference

GPU utilization is not a binary indicator. During token generation, utilization may fluctuate because the model is repeatedly executing small steps. A machine can also show low utilization while the model remains resident in VRAM.

Use three signals together:

Ollama logs: did it initialize a GPU backend?
ollama ps: where is the model placed?
OS GPU tools: is VRAM allocated and is the device active during inference?

If all three indicate GPU execution, do not “fix” the system just because a graph is not pinned at 100%.

Update regressions: prove the version changed the behavior

Local AI stacks change quickly. If a configuration worked yesterday and stopped using the GPU after an update, record the versions before making more changes:

ollama --version
nvidia-smi

Then compare the smallest reproducible test with the previous known-good release when practical.

Ollama's GitHub issue tracker contains examples of version-specific GPU discovery regressions. For example, a reported 2026 issue described a prerelease where an NVIDIA GPU stopped being detected while an earlier release worked on the same system. Another report involved NVIDIA MIG discovery in specific Docker/runtime combinations. These are reported cases, not universal behavior, but they demonstrate why version pinning and reproducible tests matter.

If the regression is confirmed, check the relevant Ollama release notes and issue tracker before changing drivers, CUDA libraries, models and configuration simultaneously.

Mixed NVIDIA and AMD GPUs need extra care

Systems containing an NVIDIA dGPU plus an AMD integrated GPU are more complicated than a single-vendor machine. Both devices can be discoverable, and backend/device-selection behavior can vary by operating system and Ollama version.

If you are diagnosing a mixed-GPU system, start with the simplest configuration: verify each vendor's device independently, remove unnecessary visibility overrides, and inspect Ollama's inference-compute lines. Only after that should you experiment with backend-specific selection.

Recent GitHub reports have documented cases where device visibility variables did not produce the expected scheduling behavior on mixed NVIDIA/AMD systems. Treat those reports as clues for reproducing a specific configuration, not as a reason to copy every environment variable from a forum post.

Useful diagnostic sequence

If you want a clean checklist, follow this order:

Run ollama --version.
Run the vendor's GPU diagnostic, such as nvidia-smi.
Run ollama list and select a known-working model.
Run the model while watching GPU memory.
Run ollama ps during active inference.
Start Ollama with OLLAMA_DEBUG=1 and inspect backend discovery.
If NVIDIA on Linux, check UVM and kernel messages.
If Docker is involved, run docker run --gpus all ubuntu nvidia-smi.
If AMD on Linux, verify /dev/kfd, /dev/dri and service-user permissions.
Check whether the model, context and concurrency fit the available VRAM.
Only then test visibility/backend overrides.
If the failure started after an update, compare the exact versions.

Common mistakes that waste time

Mistake	Why it is misleading	Better approach
Installing the CUDA toolkit immediately	Ollama's packaged GPU libraries and the system toolkit are not the same thing	Verify the NVIDIA driver and Ollama backend first
Setting every GPU environment variable from a forum post	Conflicting variables can make discovery harder	Change one variable at a time
Watching only GPU utilization	Utilization can fluctuate during generation	Check logs, placement and VRAM together
Ignoring the service account	Interactive shell permissions may differ from systemd	Test the actual Ollama service environment
Assuming a model file size equals VRAM use	KV cache and runtime allocations add memory requirements	Account for context and runtime overhead
Blaming the GPU when Docker cannot see it	The container may not have GPU passthrough	Test `nvidia-smi` inside the container

Security and reliability considerations

Do not expose Ollama's API publicly just to make a remote application “see the GPU.” Keep the inference service on a trusted interface or private network unless you have deliberately designed authentication and network controls around it.

GPU troubleshooting also deserves the same caution as any production service: changing drivers, unloading kernel modules, restarting inference services, or modifying systemd units can interrupt unrelated workloads. On a workstation, a reboot may be harmless. On a shared server, schedule the change.

When CPU inference is actually the right choice

GPU is not automatically better for every workload. If the GPU is occupied by another application, has very little available memory, or is an unsupported/unstable device, CPU inference can be a deliberate fallback. A small model on a capable CPU may also be perfectly adequate for lightweight automation.

The goal is not “make the GPU graph move.” The goal is to choose the inference path that gives you the required responsiveness without making the machine unstable.

FAQ

Why is Ollama using CPU instead of my NVIDIA GPU?

Common causes include driver problems, failed CUDA backend initialization, service-environment differences, Docker GPU passthrough issues, insufficient usable VRAM, or a version-specific regression. Start with nvidia-smi, Ollama debug logs and ollama ps.

How do I check whether Ollama is using my GPU?

On NVIDIA, watch nvidia-smi while generating and compare that with ollama ps and Ollama's inference-compute log lines. No single utilization percentage is sufficient evidence by itself.

Does Ollama need the CUDA toolkit installed?

Do not assume that installing the full CUDA toolkit is the fix. First verify the NVIDIA driver and Ollama's own CUDA backend detection. The official troubleshooting guidance focuses on driver, UVM, container access and backend diagnostics.

Why does Ollama use only part of my GPU?

The model may be partially offloaded because the complete workload does not fit in available VRAM. Context length, KV cache, runtime buffers and other GPU applications can reduce usable memory.

Why does Ollama work manually but not as a system service?

Your interactive shell and the system service can have different environment variables, users and permissions. Check systemctl cat ollama and configure the service explicitly when required.

Should I force CUDA with OLLAMA_LLM_LIBRARY?

Only as a targeted troubleshooting step when logs show that backend selection is the problem. Ollama's documentation describes library overrides as a workaround for autodetection problems. Keep the override minimal and remove it once the underlying issue is understood.