Docker Model Runner + Compose Models: Build a Portable Local LLM API with FastAPI (2026 Guide)
Most local-LLM tutorials start by asking you to run a model server, copy its URL into an application, and then maintain the connection yourself. That works, but it leaves model setup outside the application definition. Docker Compose now has a model-aware models section: your service declares the model it needs, and a compatible platform such as Docker Model Runner provisions the model endpoint and injects the connection details.
In this tutorial, you will build a small FastAPI chat service that uses a local model through Docker Model Runner. The app, Python dependencies, model identifier, context setting, and model connection will be described in one Compose project. You will also learn how to test the API, inspect the injected environment, change models, and avoid exposing an unauthenticated inference endpoint.
What you are building
The finished project has one application container and one model dependency managed by Docker Model Runner:
- FastAPI: accepts a chat message at
POST /chat. - Docker Model Runner (DMR): downloads and serves the model locally.
- Compose model binding: injects the model endpoint and model identifier into the FastAPI container.
- OpenAI Python SDK: calls DMR through its OpenAI-compatible chat-completions API.
This is not an Ollama container and does not require a separate model service in your Compose file. Docker Model Runner is the model provider in this example. If you want an Ollama-specific deployment, see our Ollama in Docker with NVIDIA GPU guide.
1. Check the prerequisites
You need:
- Docker Desktop with Docker Model Runner support enabled, or Docker Engine with the Docker Model Runner plugin installed.
- Docker Compose v2.38.0 or later. Compose model dependencies were introduced in this version.
- Python knowledge is helpful, but Python will run inside the container.
- Enough system memory for the model you select. The sample uses a small quantized model to keep the first run approachable.
Docker's current Model Runner documentation lists Docker Desktop 4.40 or later on macOS and 4.41 or later on Windows. The documented macOS path is Apple Silicon. On Windows x86-64, the documented GPU path requires a supported NVIDIA GPU and driver 576.57 or later. Docker Engine on Linux supports CPU, NVIDIA CUDA, AMD ROCm, and Vulkan backends; NVIDIA use requires driver 575.57.08 or later. Check the official platform requirements before choosing a machine.
Check your installed versions:
docker version
docker compose version
Make sure Compose reports v2.38.0 or newer. If it does not, update Docker Desktop or the Compose plugin before continuing.
2. Enable Docker Model Runner
On Docker Desktop:
- Open Docker Desktop settings.
- Open the AI section.
- Enable Docker Model Runner.
- On a supported Windows NVIDIA setup, enable GPU-backed inference if that option is available and you want to use the GPU.
- Apply the settings and wait for the feature to start.
You do not need to enable host-side TCP access just to use the Compose model binding in this tutorial. Enable TCP only if a process on the host, such as an IDE or a command-line client, needs to call the Model Runner API directly.
Verify that the CLI can see Model Runner:
docker model status
docker model version
If you are using Docker Engine on Ubuntu or Debian, Docker documents installation through its package repository:
sudo apt-get update
sudo apt-get install docker-model-plugin
docker model version
For RPM-based distributions, the documented package command is:
sudo dnf update
sudo dnf install docker-model-plugin
On Docker Engine, TCP access is enabled by default on port 12434. Keep that endpoint private; the security section explains why.
3. Create the project
Create a folder and enter it:
mkdir compose-local-llm
cd compose-local-llm
Create these three files:
compose-local-llm/
├── app.py
├── requirements.txt
├── Dockerfile
└── compose.yaml
The model itself is not stored in this project directory. Docker Model Runner manages the model artifact and its local cache.
4. Add the FastAPI application
Create requirements.txt:
fastapi
uvicorn[standard]
openai
For a production project, pin and lock the dependency versions you have tested. The open version ranges above keep this tutorial focused on the integration rather than asserting a particular SDK release.
Create app.py:
import os
from fastapi import FastAPI, HTTPException
from openai import OpenAI
from pydantic import BaseModel, Field
MODEL_URL = os.environ["LLM_BASE_URL"].rstrip("/")
MODEL_ID = os.environ["LLM_MODEL"]
# The placeholder key satisfies OpenAI-compatible client configuration.
# Docker Model Runner does not use it as authentication.
client = OpenAI(
base_url=MODEL_URL,
api_key="not-needed",
)
app = FastAPI(title="Local Compose LLM API")
class ChatRequest(BaseModel):
message: str = Field(min_length=1, max_length=4000)
@app.get("/health")
def health():
# This is an application liveness check, not a model inference test.
return {"status": "ok", "model": MODEL_ID}
@app.post("/chat")
def chat(request: ChatRequest):
try:
result = client.chat.completions.create(
model=MODEL_ID,
messages=[
{
"role": "system",
"content": "You are a concise and helpful technical assistant.",
},
{"role": "user", "content": request.message},
],
temperature=0.2,
max_tokens=300,
)
except Exception as exc:
# Do not return internal connection details to API callers.
raise HTTPException(
status_code=502,
detail="The local model request failed. Check the Model Runner logs.",
) from exc
answer = result.choices[0].message.content or ""
return {"answer": answer}
There are two important details:
- The application reads
LLM_BASE_URLandLLM_MODELfrom its environment. Compose will inject both values from the model binding. - The OpenAI SDK uses the base URL for the compatible API. The Model Runner OpenAI-compatible API exposes chat completions under
/engines/v1/chat/completions; do not append the endpoint path yourself when passing the base URL to the SDK.
5. Build the application image
Create a Dockerfile:
FROM python:3.12-slim
WORKDIR /app
COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt
COPY app.py .
EXPOSE 8000
CMD ["uvicorn", "app:app", "--host", "0.0.0.0", "--port", "8000"]
This image contains only the API application and its Python dependencies. It does not bundle model weights, CUDA libraries, or a second inference server.
6. Declare the model in compose.yaml
Create compose.yaml:
services:
api:
build: .
ports:
- "127.0.0.1:8000:8000"
models:
local_llm:
endpoint_var: LLM_BASE_URL
model_var: LLM_MODEL
models:
local_llm:
model: ai/smollm2:360M-Q4_K_M
context_size: 2048
Read the file from the bottom up:
- The top-level
models.local_llmentry identifies the model artifact and sets a modest context size for this demonstration. - The
apiservice explicitly requests that model through itsmodelsattribute. endpoint_varandmodel_varchoose the environment variable names that Compose injects into the API container.- The port mapping binds the FastAPI service to loopback on the host, rather than exposing it on every network interface.
The sample model identifier is a small quantized SmolLM2 artifact documented in Docker's own getting-started examples. It is useful for checking the wiring, not a recommendation that a small model will match a larger coding or reasoning model. You can replace it with another compatible artifact that fits your hardware.
Compose model support requires v2.38.0 or later and a platform that implements Compose models, such as Docker Model Runner. The platform provisions the model and supplies its endpoint; the application does not need a hard-coded localhost address or a manually maintained container IP.
7. Validate the Compose configuration and start it
First, ask Compose to parse the project:
docker compose config
Review the rendered configuration for indentation mistakes or an incorrectly named model. Then build and start the API:
docker compose up --build
On the first run, expect a model download and initialization delay. The model artifact is cached locally after it is pulled. Docker Model Runner loads models into memory when requests need them, so the first inference can take longer than later requests.
In another terminal, check the application:
docker compose ps
curl -sS http://127.0.0.1:8000/health
The health endpoint should return JSON containing "status":"ok" and the model identifier. This proves that FastAPI started and received the Compose-injected model name; it does not yet prove that inference works.
8. Inspect the injected model connection
Compose's model binding should provide the two variables declared in the file. Inspect them inside the running service:
docker compose exec api sh -lc 'printf "LLM_BASE_URL=%s\nLLM_MODEL=%s\n" "$LLM_BASE_URL" "$LLM_MODEL"'
The model name should match the identifier in compose.yaml. The endpoint should be the OpenAI-compatible base URL supplied by the platform. If either value is empty, stop here and check the Compose version and platform support rather than hard-coding a guessed URL.
Docker's API reference documents the OpenAI-compatible host endpoint as http://localhost:12434/engines/v1 when host-side TCP access is enabled. From a container, the hostname and routing depend on the Docker platform. Compose model binding is useful precisely because it injects the endpoint appropriate to the provider.
9. Send a real chat request
Call the FastAPI endpoint:
curl -sS http://127.0.0.1:8000/chat \
-H "Content-Type: application/json" \
-d '{
"message": "Explain Docker Compose models in two short sentences."
}'
A successful request returns a JSON object with an answer field. The wording will vary by model and sampling behavior; do not expect a fixed sentence.
You can also send a longer coding-oriented prompt:
curl -sS http://127.0.0.1:8000/chat \
-H "Content-Type: application/json" \
-d '{
"message": "Write a Python function that validates an email-like string. Include two edge cases."
}'
If this request succeeds, the complete path is working: host request, FastAPI, injected Compose model endpoint, Docker Model Runner, and local model inference.
10. Change the model and tune context
To try a different model, change the artifact identifier in the top-level models section. For example, Docker's current documentation uses identifiers such as ai/qwen3, ai/llama3.2, and ai/qwen2.5-coder in its examples and integrations. Confirm the artifact exists in the registry and that its format and runtime are supported by your installation.
For a coding-focused model, you might use:
models:
local_llm:
model: ai/qwen2.5-coder
context_size: 4096
This is an example configuration, not a performance guarantee. A model's maximum trained context and the context configured for inference are different things. Larger context windows can consume substantially more RAM or VRAM. Start small, inspect the model metadata, and increase the context only when your workload needs it.
You can also provide llama.cpp runtime flags through Compose when the selected model uses that engine:
models:
local_llm:
model: ai/qwen2.5-coder
context_size: 4096
runtime_flags:
- "--temp"
- "0.2"
- "--top-p"
- "0.9"
Runtime flags are passed to the inference engine, so do not copy llama.cpp flags into a configuration using a different engine without checking that engine's documentation. See Docker's configuration reference for context size and engine-specific options.
After changing the model definition, apply the project again:
docker compose up -d
11. Troubleshooting
Compose rejects the models section
Check docker compose version. The Compose models feature requires v2.38.0 or later. Also confirm that the selected Compose platform supports model provisioning. Updating Docker Desktop or the Compose plugin may be required.
The model endpoint variables are empty
Check that the service has a models binding and that its key matches the top-level model key. In this example, both are named local_llm. Confirm that Docker Model Runner is enabled and that the platform supports Compose models. Do not substitute a guessed IP address as a permanent fix.
The API returns a model-not-found error
Use the full model identifier, including its namespace, such as ai/qwen2.5-coder. Check the model list with:
docker model list
If you changed the model name in Compose, make sure the application is using the injected LLM_MODEL rather than a stale hard-coded identifier.
The request fails with a connection error
Check the Model Runner status and inspect the API container's injected endpoint:
docker model status
docker compose exec api sh -lc 'echo "$LLM_BASE_URL"'
If you are bypassing Compose model binding and connecting directly from a host process, Docker Desktop requires host-side TCP access to be enabled. The documented host URL is http://localhost:12434; the OpenAI SDK base URL includes /engines/v1. Docker Engine on Linux has different container routing; consult the API reference for its host and container addresses.
The first request is very slow or times out
The model may still be downloading or loading into memory. Check the Docker Model Runner logs and wait for initialization. If the machine is short on memory, choose a smaller quantized model or reduce context_size. A large context setting is not automatically better for a small local workload.
The application starts but inference returns an error
The /health endpoint only checks the FastAPI process. Use POST /chat to test the actual model path. Inspect the API container logs with:
docker compose logs -f api
Also inspect Model Runner logs in Docker Desktop or with the available Model Runner CLI logging command. Check that the selected model supports the request format and parameters being sent.
12. Security and privacy checklist
- Do not expose the Model Runner API publicly. Docker documents that its API is not authenticated. Any client that can reach it may submit inference requests and interact with model management endpoints.
- Keep host access local. Do not publish port
12434to a public interface or forward it through a router. Enable host-side TCP only when needed and restrict access with host firewall rules. - Keep the demo API on loopback. The Compose file binds port 8000 to
127.0.0.1. If you change that for a team deployment, add authentication, authorization, request limits, and network controls first. - Do not treat the placeholder API key as protection. Docker's OpenAI-compatible API does not require an API key and ignores the Authorization header. The value
not-neededis only there because the SDK expects a key argument. - Use trusted model artifacts. Model files are executable inference inputs, not automatically trustworthy just because they are available in a registry. Use a known publisher and a specific model tag for repeatable deployments.
- Understand the network boundary. Inference can run locally, but pulling a model requires downloading it. Docker's documentation also describes limited model-name and request-origin telemetry behavior; review your Docker Desktop usage-statistics settings and your organization's policy.
- Protect prompts in application logs. Avoid logging full user prompts, generated responses, or secrets unless your retention and access policies explicitly allow it.
13. Stop the stack and clean up
Stop the application:
docker compose down
This stops and removes the Compose application container. The model artifact managed by Docker Model Runner may remain in its local cache; stopping the API is not the same as deleting downloaded models. Review the model list before removing anything:
docker model list
Only remove a model if you are sure no other local project depends on it.
Where to go next
You now have a small local-LLM API whose model dependency is declared alongside the application. That makes the project easier to share with another developer who has a compatible Compose platform, and it avoids hard-coding a model server address into the Python source.
Continue with these related GyanAangan guides:
- Run Ollama in Docker with NVIDIA GPU — for a separate Ollama-based container deployment.
- Ollama structured outputs with Pydantic — validate model output in Python applications.
- Build a read-only local codebase agent — add carefully scoped tools to a local model workflow.
- Incremental local RAG with Ollama and SQLite — add document retrieval without re-embedding every file on each run.
FAQ
Is Docker Model Runner the same as Ollama?
No. Both can serve local models and expose compatible APIs, but they are separate runtimes with different management and integration workflows. This tutorial uses Docker Model Runner as the provider behind Compose model declarations.
Do I need a separate model container in compose.yaml?
Not for the Compose model-provider workflow shown here. You declare the model under the top-level models key and grant it to the application service. A compatible platform provisions the model endpoint and injects the connection details.
Does the application need an OpenAI API key?
No real OpenAI key is needed for local Docker Model Runner. The SDK example supplies not-needed as a placeholder. That placeholder does not authenticate requests or secure the endpoint.
Can I use this with a different OpenAI-compatible framework?
Yes, if the framework supports a configurable OpenAI-compatible base URL and the model supports the API operations you use. Docker also documents integrations for tools such as Continue, Cline, Aider, LangChain, and LlamaIndex.
Can I move the same Compose file to a cloud provider?
The Compose models specification is designed for platforms that implement it, but each platform decides how to provision the model. Portability does not mean every provider uses the same runtime, model catalog, or configuration extensions.
Is local inference automatically private?
It keeps inference on the local runtime in this setup, but it does not eliminate model downloads, application logging, or access risks. Keep the API private, review telemetry settings, and apply your own controls for prompts and outputs.
Official documentation and references
- Docker Model Runner overview and platform requirements
- Docker Model Runner installation and getting started
- Define AI models in Docker Compose applications
- Docker Compose models reference
- Docker Model Runner REST API reference
- Docker Model Runner configuration options
- Docker Model Runner IDE and tool integrations
- Docker's hello-genai sample repository
