Run Ollama in Docker with an NVIDIA GPU: Docker Compose, Persistent Models and Troubleshooting (2026)

Updated: 31 October 2026. A local model can be easy to run on a laptop and surprisingly fiddly to deploy as a repeatable service. The usual problems are not the prompt: they are GPU passthrough, model files disappearing when a container is replaced, and accidentally exposing an API that has no built-in local authentication.

This guide builds a small, reproducible Ollama deployment with Docker Compose on Linux and an NVIDIA GPU. It keeps model data in a named volume, binds the API to the host loopback interface, tests the server with a real request, and shows how to diagnose GPU fallback. It is an infrastructure guide, not a performance benchmark; results depend on your GPU, driver, model, quantization, context size, and workload.

Who this is for

Use this setup if you want a local model endpoint for development, an internal tool, or a service that can be recreated from a Compose file. It is especially useful when you already run Dockerized applications and want Ollama separated from your application containers.

This walkthrough targets Ubuntu or another supported Linux distribution, Docker Engine with the Compose plugin, and an NVIDIA GPU with a working host driver. AMD has a different container path, described briefly below. Docker Desktop on macOS does not provide NVIDIA GPU passthrough, and this setup does not make an Apple Silicon GPU available inside a Linux container.

Prerequisites

• A supported Linux host and an NVIDIA GPU.
• A working NVIDIA driver; nvidia-smi must work on the host before you troubleshoot Docker.
• Docker Engine and Docker Compose v2.
• Enough disk space for the model and enough system RAM/VRAM for the chosen model, context window, and concurrent requests.
• Basic familiarity with a terminal, YAML, and HTTP APIs.

The example uses qwen3:8b as a reproducible starting point. The Ollama library currently lists its Q4_K_M package at roughly 5.2 GB. That is the download size, not a promise that 5.2 GB of VRAM is sufficient: runtime buffers, context/KV cache, and other processes need memory too. If resources are tight, choose a smaller model variant.

1. Confirm the host GPU and Docker installation

First, check the host itself:

nvidia-smi
docker --version
docker compose version

Expected result: the first command shows your GPU and driver information; the Docker commands print installed versions. If nvidia-smi fails on the host, fix the driver installation before proceeding. A container cannot repair a missing or broken host driver.

2. Install and configure the NVIDIA Container Toolkit

Docker needs the NVIDIA Container Toolkit to pass supported GPU devices into containers. Follow NVIDIA's current installation guide for your distribution rather than copying a package version from an old blog post. On Ubuntu/Debian, the official repository setup and installation commands are documented by NVIDIA.

After installing the toolkit, configure Docker and restart the daemon:

sudo nvidia-ctk runtime configure --runtime=docker
sudo systemctl restart docker

This updates Docker's runtime configuration. If this is a production host, review the change and plan the daemon restart because it can interrupt running containers.

Test GPU passthrough independently of Ollama:

docker run --rm --gpus all nvidia/cuda:12.9.0-base-ubuntu22.04 nvidia-smi

Docker's official GPU guide uses a CUDA base image to demonstrate this pattern. The image tag must be compatible with the host driver; if the test fails with a CUDA/driver compatibility message, select a compatible documented image tag rather than changing Ollama settings at random.

Expected result: the container prints NVIDIA device information. If the host command succeeds but this container test fails, the fault is in the Docker/NVIDIA runtime path, not the model.

3. Create a project directory

mkdir -p ~/ollama-compose
cd ~/ollama-compose

Keep the Compose file in version control if appropriate, but do not commit secrets or private prompts. This example needs no API key because the local Ollama API is unauthenticated and will remain bound to loopback.

4. Define the Ollama service in Compose

Create a file named compose.yaml with the following content:

services:
  ollama:
    image: ollama/ollama:latest
    container_name: ollama
    restart: unless-stopped
    ports:
      - "127.0.0.1:11434:11434"
    volumes:
      - ollama:/root/.ollama
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: all
              capabilities: [gpu]

volumes:
  ollama:

Why these settings matter:

• ollama/ollama:latest follows the current image tag. For a reproducible production deployment, test an approved version and pin a version tag or image digest according to your update policy.
• The named volume mounted at /root/.ollama keeps downloaded model data outside the container's writable layer.
• The GPU device reservation asks Compose to expose the NVIDIA GPU. The capabilities: [gpu] field is required by Docker Compose's GPU reservation syntax.
• 127.0.0.1:11434:11434 publishes the host port only on loopback. It is intended for clients on the same host, not other machines on your LAN.
• restart: unless-stopped restarts the service after a Docker restart unless you intentionally stopped it.

Validate the file before starting anything:

docker compose config

Then start the service:

docker compose pull
docker compose up -d
docker compose ps
docker compose logs --tail=100 ollama

Expected result: the service appears as running, and the logs show the server starting. The initial image pull requires internet access. Model downloads also require internet access unless you have separately prepared a cache or offline workflow.

5. Download a model and run a first prompt

Pull a known model into the persistent volume:

docker compose exec ollama ollama pull qwen3:8b

Run a short interactive test:

docker compose exec ollama ollama run qwen3:8b

Type a small prompt and confirm that the model responds. Exit the interactive session using the CLI's supported exit method. The first request can take longer because the model must be loaded; do not interpret one cold start as steady-state performance.

List the installed models:

docker compose exec ollama ollama list

Because the model directory is in a named volume, recreating the container should not require downloading the model again. The volume is separate from the container lifecycle.

6. Test the HTTP API from the host

Ollama exposes a local REST API. Send a non-streaming chat request so the response is easy to inspect:

curl -sS http://127.0.0.1:11434/api/chat \
  -H 'Content-Type: application/json' \
  -d '{
    "model": "qwen3:8b",
    "messages": [
      {
        "role": "user",
        "content": "Explain why a persistent Docker volume matters in one sentence."
      }
    ],
    "stream": false
  }'

Expected result: a JSON response containing a message with generated content and metadata. Wording is model-generated, so it will not be identical on every run. If the connection is refused, check docker compose ps, the container logs, and the host port mapping before changing firewall rules.

7. Verify that the model is actually using the GPU

A running container does not prove GPU acceleration is working. While the model is loaded, run:

docker compose exec ollama ollama ps
nvidia-smi

Ollama's ps output includes a processor field indicating whether a model is on GPU, CPU, or split between them. On the host, nvidia-smi can show GPU memory and activity while inference is running. A partial CPU/GPU split is not necessarily an error; it can happen when the model and runtime state do not fit entirely in VRAM.

Do not compare two setups using only a model's parameter count or file size. Record the exact model tag, quantization, GPU, driver, context size, concurrent workload, warm-up policy, and the metric you care about. For a broader evaluation workflow, see our guide to regression-testing local LLMs with Promptfoo and Ollama.

8. Keep model data across container updates

To recreate the container after changing the Compose file or pulling a newer image:

docker compose down
docker compose pull
docker compose up -d

This leaves the named volume intact. Do not add -v to docker compose down unless you intentionally want to remove the named volume and its downloaded models. Before upgrading a production service, back up the volume or otherwise confirm that the models can be downloaded again and that your rollback plan is workable.

To see the volume name created by Compose, use:

docker volume ls
docker volume inspect ollama-compose_ollama

Compose prefixes named volumes with the project name by default. If you changed the directory/project name, inspect the actual name shown by docker volume ls instead of assuming it is exactly the example above.

9. Troubleshooting common failures

Container starts, but Ollama uses CPU

• Confirm the host driver with nvidia-smi.
• Run the standalone CUDA container test from Step 2. If it fails, fix the NVIDIA Container Toolkit/runtime first.
• Inspect docker compose logs ollama for GPU discovery or initialization errors.
• Run docker compose exec ollama ollama ps while a model is loaded. Some models may be split across CPU and GPU because of memory limits.
• Update to a supported driver/runtime combination and restart Docker only after reviewing the impact.

Port 11434 is already in use

Check the existing listener and containers before stopping anything:

sudo ss -ltnp | grep 11434
docker ps --format 'table {{.Names}}\t{{.Ports}}'

If another local Ollama instance is already using that port, either stop the duplicate service intentionally or choose a different host port, such as 127.0.0.1:11435:11434. In that case, send host API requests to port 11435.

The model download fails or the disk fills up

Check available disk space with df -h, review container logs, and confirm that the Docker host can reach the model registry. Model files can be large. Do not delete the named volume as a first troubleshooting step; that can remove all downloaded models.

The API works in the container but not from the host

Check the Compose port mapping and use curl http://127.0.0.1:11434/api/tags on the Docker host. Remember that binding the published port to 127.0.0.1 intentionally prevents other machines from connecting through that host port.

It worked once and later stopped using the GPU

Inspect fresh logs rather than relying on old output. Ollama documents cases where GPU discovery can fail after startup; the cause can involve driver, runtime, or system configuration. Avoid changing Docker's cgroup driver blindly. Collect the error, driver version, container logs, and toolkit test result first, then follow the current official troubleshooting guide.

10. Security and privacy checklist

• Keep the API private by default. Ollama's local API does not require authentication. Do not publish port 11434 to 0.0.0.0 or expose it directly to the public internet.
• For remote access, use a controlled path. Prefer a VPN or private network. If a reverse proxy is required, put authentication, TLS, rate limits, and access controls in front of Ollama and keep the backend unreachable directly. A proxy alone is not authentication.
• Review host firewall and Docker networking. Docker port publishing can interact with firewall rules in ways that are not obvious. Test reachability from a second machine and review the Docker security documentation for your platform.
• Protect the volume. Model files may be redistributable under their own license, but prompts, logs, and adjacent application data can be sensitive. Limit host access and back up only what you intend to retain.
• Do not assume all requests are local. This setup uses a local model tag, but cloud model features or other connected applications may send data outside the host. Review the configuration and model choice for your privacy requirements.
• Control resource use. Restrict access to trusted clients, monitor VRAM/RAM and disk space, and consider request limits at the application or proxy layer for multi-user deployments.

11. AMD, Windows, and macOS notes

This Compose file is for NVIDIA GPUs on a supported Linux host. Ollama documents a different Docker image tag and device configuration for AMD ROCm, so do not copy the NVIDIA reservation block and assume it will work unchanged. Windows GPU passthrough for Linux containers depends on the WSL 2 backend and supported drivers. Docker Desktop on macOS does not provide GPU passthrough for Ollama's Linux container; use a native runtime or a different supported deployment path if you need Apple Silicon acceleration.

Trade-offs and limitations

• Compose is repeatable, not magically production-ready. You still need an upgrade policy, backups, monitoring, capacity planning, and access control.
• The image tag is mutable. latest is convenient for a first run but not ideal for reproducible deployments. Pin and test a specific image before production use.
• GPU availability does not guarantee full GPU residency. Model size, context length, quantization, VRAM, and concurrent workloads all affect placement and speed.
• The local API has no built-in local authentication. Loopback binding is a safer default for this single-host example, but it is not a replacement for a deliberate network security design.

Frequently asked questions

Will my models disappear when I recreate the container?

Not when the named volume remains attached. The Compose file stores Ollama data in the ollama volume. Removing the volume, changing to a different volume, or mounting the wrong path can make the models appear missing.

Does a 5.2 GB model need exactly 5.2 GB of VRAM?

No. The listed size is the model package size. Inference also needs memory for runtime state and context, and some layers may be placed in system RAM. Leave headroom and test the model on the target hardware.

Can another computer on my LAN use this API?

Not with the default port mapping in this guide, because it binds to host loopback. If remote access is required, deliberately design a private network or authenticated proxy path instead of simply opening port 11434.

Can I use this on a MacBook with Apple Silicon?

This NVIDIA GPU recipe does not apply. Docker Desktop on macOS does not pass the Apple GPU through to the Ollama Linux container. Use a supported native runtime for Apple Silicon acceleration.

Where should I go next?

If you want one API interface across multiple local runtimes, read the LiteLLM local LLM gateway guide. If your next goal is to add AI-assisted checks to a repository, see the Ollama and Git code-review hook tutorial. For basic model lifecycle commands, use the Ollama CLI reference.

Official references

• Ollama official Docker instructions — NVIDIA and AMD container launch paths.
• NVIDIA Container Toolkit installation guide — installation and Docker runtime configuration.
• Docker Compose GPU support — device reservation syntax and requirements.
• Docker Engine GPU access — host prerequisites and the --gpus test pattern.
• Ollama FAQ — GPU placement checks, local API exposure, and model storage behavior.
• Ollama troubleshooting documentation — container logs and GPU discovery diagnostics.
• Ollama Qwen3 8B model page — model tag, listed package size, and license information.

Verification note: Commands and configuration in this guide were checked against the official documentation linked above on 31 October 2026. This article was not executed on a live NVIDIA host during authoring, so treat the commands as a documented deployment recipe, not as a claim of hands-on testing. Validate driver compatibility, Compose behavior, model fit, and security controls on your own host before production use.