Ollama 0.40.16 Responses API Tutorial: Build a Local Tool-Calling Agent with Python (2026)
Updated November 13, 2026. If your application already speaks the OpenAI Responses API, you can point it at a local Ollama server instead of rewriting your integration around a different SDK. The important catch is state: Ollama implements a non-stateful Responses endpoint, so your application must keep and resend the conversation items that matter.
That distinction is more important than the URL change. It affects tool-call loops, retries, context limits, and how you store agent sessions. This guide builds a small, read-only Python agent against Ollama's /v1/responses endpoint, explains the differences from Chat Completions, and covers the truncation behavior clarified in Ollama 0.40.16.
What changed in Ollama 0.40.16?
Ollama's OpenAI compatibility documentation records support for POST /v1/responses from version 0.13.3 onward. It supports a subset of the Responses API, including input, instructions, tools/function calling, streaming, and reasoning summaries for compatible thinking models. It does not implement the stateful conversation features that depend on previous_response_id or a server-side conversation object.
Ollama 0.40.16, released November 9, 2026, made a separate but useful correction: /v1/responses now respects the request's truncation setting and no longer silently truncates by default. The endpoint itself is not new in 0.40.16; the release improves how context overflow is handled. Version 0.40.17-rc0 is listed as a pre-release, while 0.40.16 is the latest stable release shown in the official release list as of this guide's update date.
Practical implication: you can reuse the OpenAI Python client and the Responses-style tool-call item format, but you must own conversation history and test the exact features your app needs. Do not assume every OpenAI-hosted Responses feature exists in Ollama.
Who this guide is for
- Python developers migrating an OpenAI Responses client to local inference.
- Teams building tool-using assistants that need a local model and application-controlled tools.
- Developers comparing
/v1/responseswith Ollama's native/api/chatand OpenAI-compatible/v1/chat/completions. - Engineers who want explicit context-overflow behavior instead of silently losing older prompt items.
This is a local development tutorial. It does not claim that every model will follow tool schemas reliably, or that a local model is a drop-in replacement for every hosted Responses model.
Prerequisites
- Ollama installed and running locally. Use version 0.40.16 or later for the truncation behavior discussed below.
- Python 3.10 or newer and a terminal.
- The OpenAI Python SDK installed in an isolated virtual environment.
- A model with tool-calling support. The official Ollama tool-calling guide uses Qwen3; this walkthrough uses
qwen3:8bas a reproducible starting point. - Enough RAM or unified memory for the selected model and the context you intend to send.
1. Install or update Ollama and verify the version
Install Ollama using the official download instructions for your operating system. On Linux, the official installer is:
curl -fsSL https://ollama.com/install.sh | sh
On macOS or Windows, use the official installer from ollama.com/download. Do not run a second server if the desktop app or system service is already managing one.
Check the CLI version and confirm the local server responds:
ollama --version
curl -s http://localhost:11434/api/version
Expected result: the CLI prints an installed version, and the HTTP request returns a JSON version value. If your version is older than 0.40.16, update before relying on the release's truncation correction.
2. Download a tool-capable model
Pull Qwen3 8B:
ollama pull qwen3:8b
ollama list
The model page lists Qwen3 variants and identifies tool use as a capability of the family. The Ollama tool-calling documentation also uses qwen3 in its function-call examples. A model tag being available does not guarantee that it will choose the correct tool for every prompt; evaluate it with your own tasks.
If the model is too large for your machine, choose a smaller supported variant and retest. Do not assume the same tool-call reliability across model sizes or quantizations.
3. Make a basic Responses API request
Create a clean Python environment and install the OpenAI SDK:
python -m venv .venv
source .venv/bin/activate
# Windows PowerShell:
# .venv\Scripts\Activate.ps1
python -m pip install --upgrade pip openai
Create responses_smoke_test.py:
from openai import OpenAI
client = OpenAI(
base_url="http://localhost:11434/v1",
api_key="ollama", # required by the SDK; local Ollama ignores this value
)
response = client.responses.create(
model="qwen3:8b",
input="Explain in two sentences what a REST API is.",
max_output_tokens=160,
)
print(response.output_text)
Run it:
python responses_smoke_test.py
Expected result: the SDK receives a Responses object and prints its text using output_text. If you receive a 404, first check the installed Ollama version and confirm the base URL ends with /v1. If the model is missing, pull it and verify the exact tag with ollama list.
4. Understand the stateless request contract
Hosted Responses integrations may use server-managed state through fields such as previous_response_id or conversation. Ollama's documented local Responses implementation is non-stateful: do not depend on either field. Your application should retain the input items for each session and send the required history again with the next request.
A simple mental model is:
- Your application creates a list of input items.
- It sends that list to
client.responses.create(). - The response contains output items, which may include a function call.
- Your application appends the model's output items to its local history.
- Your application executes only the approved function and appends a
function_call_outputitem with the matchingcall_id. - It sends the expanded input list again to obtain the next model response.
This gives you explicit control over retention and pruning, but it also means you are responsible for storing session history, limiting its size, and preventing one user's items from being mixed with another user's session.
5. Define a safe, read-only tool
We will build a small issue lookup tool backed by in-memory sample data. It does not connect to GitHub, Linear, a production database, or any external service. This keeps the example safe while showing the Responses API tool-call format.
Create local_agent.py:
import json
from openai import OpenAI
MODEL = "qwen3:8b"
client = OpenAI(
base_url="http://localhost:11434/v1",
api_key="ollama",
)
ISSUES = {
"APP-104": {
"title": "Login form returns a validation error",
"status": "In Progress",
"priority": "High",
"summary": "The API rejects a valid email when the address contains a plus sign.",
},
"APP-118": {
"title": "Add request IDs to API logs",
"status": "Open",
"priority": "Medium",
"summary": "Support needs a stable request identifier for incident investigation.",
},
}
TOOLS = [
{
"type": "function",
"name": "lookup_issue",
"description": "Look up one issue by its exact issue key. This tool is read-only.",
"parameters": {
"type": "object",
"properties": {
"issue_key": {
"type": "string",
"description": "Issue key such as APP-104",
}
},
"required": ["issue_key"],
"additionalProperties": False,
},
}
]
def dispatch_tool(name: str, arguments_json: str) -> str:
if name != "lookup_issue":
return json.dumps({"error": "Tool is not allowed"})
try:
args = json.loads(arguments_json)
except json.JSONDecodeError:
return json.dumps({"error": "Arguments were not valid JSON"})
if not isinstance(args, dict) or set(args) != {"issue_key"}:
return json.dumps({"error": "Expected only the issue_key argument"})
issue_key = args["issue_key"]
if not isinstance(issue_key, str) or len(issue_key) > 32:
return json.dumps({"error": "Invalid issue key"})
issue = ISSUES.get(issue_key.upper())
if issue is None:
return json.dumps({"error": "Issue not found"})
return json.dumps({"issue_key": issue_key.upper(), **issue})
def run_agent(question: str) -> str:
input_items = [{"role": "user", "content": question}]
# A small hard limit prevents an accidental endless tool-call loop.
for _ in range(4):
response = client.responses.create(
model=MODEL,
input=input_items,
tools=TOOLS,
tool_choice="auto",
max_output_tokens=500,
truncation="disabled",
)
calls = [
item for item in response.output
if item.type == "function_call"
]
if not calls:
return response.output_text
# Preserve the model's call items because the next request is stateless.
input_items.extend(response.output)
for call in calls:
result = dispatch_tool(call.name, call.arguments)
input_items.append({
"type": "function_call_output",
"call_id": call.call_id,
"output": result,
})
return "Stopped after the tool-call limit. Review the model output and history."
if __name__ == "__main__":
print(run_agent("Look up APP-104 and summarize its status and priority."))
Run the program:
python local_agent.py
Expected result: the model should request lookup_issue, your Python code should return the sample issue record, and the next Responses request should produce a natural-language summary. The precise wording and whether the model chooses the tool are model-dependent; the code is an implementation example, not a claim of a live test.
6. Why the tool loop is structured this way
- Tool schemas describe capability, not permission. The application still decides which function names can run.
- Arguments are untrusted input. Parse JSON, validate types and bounds, and reject unexpected fields.
- Call IDs correlate outputs. The
call_idinfunction_call_outputmust match the call returned by the model. - History is explicit. The model output is appended before the tool result, so the next request contains the call and its corresponding output.
- Loops need a limit. The example caps the number of tool rounds to avoid runaway inference or repeated calls.
For a production application, replace the sample dictionary with a read-only service layer. Keep the model-facing tool interface small and make the service enforce user authorization independently of the model's arguments.
7. Test the endpoint directly with cURL
You can bypass the Python SDK to isolate HTTP and server issues:
curl -sS http://localhost:11434/v1/responses \
-H "Content-Type: application/json" \
-d '{
"model": "qwen3:8b",
"input": "Return one short sentence confirming that the local Responses endpoint works.",
"max_output_tokens": 100
}'
Look for a JSON response object with an output array. A text result is represented as an output message; tool use produces a function-call output item. Do not parse the response by assuming that the first array item is always plain text.
8. Handle context limits and truncation deliberately
Ollama 0.40.16's release notes state that /v1/responses now respects the supplied truncation setting and no longer silently truncates by default. The example explicitly sends truncation="disabled" so an oversized request is surfaced as a failure instead of letting the application assume all prior context was preserved.
For a long-running agent:
- Store conversation items per session in a database or durable session store.
- Estimate or count tokens before sending very large histories.
- Keep system instructions, current task details, and recent tool results.
- Summarize older turns in application code when appropriate, and preserve important facts with provenance.
- Record when history was pruned so debugging does not rely on an incomplete transcript.
- Test both normal requests and deliberately oversized requests against the Ollama version you deploy.
Do not treat automatic truncation as a memory strategy. Removing old tool results can change what the agent believes it has already checked.
9. Choose the right Ollama API
| Interface | Best fit | Important distinction |
|---|---|---|
/api/chat | Native Ollama applications | Ollama-specific request and response format. |
/v1/chat/completions | Clients built around OpenAI Chat Completions | Message-list conversation flow; tool calls use the Chat Completions format. |
/v1/responses | Clients built around OpenAI Responses input/output items | Tool-call items and function-call outputs; Ollama's implementation is non-stateful. |
Use the native Ollama SDK when you want Ollama-specific features and a native type system. Use Chat Completions when your existing client expects messages and tool-call IDs in that format. Use Responses when your application already uses the Responses item model or needs to share a compatible client interface. Compatibility does not mean feature parity with OpenAI-hosted models.
10. Troubleshooting
- 404 on
/v1/responses: verify Ollama is at least 0.13.3 and that the URL includes/v1. Update to 0.40.16 or later for the truncation behavior in this guide. - Connection refused: check whether the Ollama service is running and listening on
localhost:11434. Do not assume a Docker container'slocalhostis the host machine. - Model not found: run
ollama listand pull the exact model tag used in the request. - No tool call is returned: verify the model supports tools, make the tool description concrete, simplify the schema, and test with a prompt that clearly requires the tool. The model may answer directly if it thinks the tool is unnecessary.
- Invalid function-call arguments: parse the JSON and validate every field. Do not pass raw model arguments into shell commands, SQL, file paths, or privileged APIs.
- Context-length error: reduce history, shorten tool output, lower the context-heavy prompt, or create a deliberate summary. Do not silently discard important prior tool results.
- Different behavior between SDKs: capture the actual request body and compare the endpoint, model tag, tools schema, and supported fields. A client may send options that Ollama does not implement.
11. Security and privacy checklist
- Keep Ollama private. The local OpenAI-compatible examples use a placeholder API key that Ollama ignores. That is not authentication. Do not expose port 11434 directly to the public internet.
- Use a protected access layer for remote clients. If a remote application must reach the model, use a VPN or an authenticated HTTPS reverse proxy with access controls and rate limits. See the internal Tailscale guide below.
- Never let the model authorize itself. Enforce user and tenant permissions in your application before a tool reads or changes data.
- Start with read-only tools. Require a separate approval step for writes, deployments, refunds, account changes, or shell execution.
- Protect logs. Prompt histories and tool outputs may contain source code, customer data, credentials, or internal identifiers. Redact secrets and set retention limits.
- Limit resource use. Set request timeouts, maximum output tokens, bounded tool loops, per-user quotas, and queue limits.
- Review model output. Tool calls are proposals from an untrusted model, not trusted commands or proof that an operation succeeded.
12. FAQ
Is the Responses API new in Ollama 0.40.16?
No. Ollama's documentation says /v1/responses was added in 0.13.3. Version 0.40.16 clarified truncation handling for that endpoint.
Does Ollama support previous_response_id?
Not in the documented local Responses implementation. It is non-stateful, so keep the input items in your application and send the required history again.
Can I use the official OpenAI Python SDK?
Yes, for the supported subset. Set the client base URL to http://localhost:11434/v1 and supply a placeholder API key value as shown in Ollama's documentation.
Does every Ollama model support tool calling?
No. Tool support and reliability depend on the model and its template. Check the model's documentation and run a test with the exact model tag you plan to deploy.
Can I use this with Codex or Claude Code?
This tutorial demonstrates the Responses endpoint directly, not a guaranteed Codex or Claude Code integration. Those tools have their own provider, model, and feature requirements. Use a tool-specific integration guide and validate compatibility separately.
Is local inference automatically private?
Inference can stay on your machine, but privacy also depends on network exposure, application logs, backups, telemetry, remote access, and the tools you connect. Audit the full data path.
Conclusion
Ollama's Responses endpoint gives developers a familiar OpenAI-style interface for local models, and the 0.40.16 truncation correction makes context handling more explicit. The key engineering choice is to treat the endpoint as stateless: keep the transcript yourself, append function-call items and outputs in order, validate every tool request, and test the exact subset of the API your application uses.
Start with a read-only tool and a small model, verify the request/response flow, then add persistence, authentication, observability, and carefully approved write operations. That approach produces a much safer local agent than simply giving a model unrestricted access to your machine.
Further reading on GyanAangan
- Ollama Tool Calling with Python: Build a Safe Read-Only Local Codebase Agent — a separate walkthrough using Ollama's native tool-calling workflow.
- Build a Local LLM Gateway with LiteLLM — route Ollama and LM Studio behind a shared API.
- Access Your Local Ollama API Remotely with Tailscale Serve — private remote access without exposing port 11434.
- Run Claude Code and Codex with Local Models on Apple Silicon — a different runtime and agent integration path.
Official references
- Ollama v0.40.16 release notes — stable release dated November 9, 2026, including the Responses truncation change.
- Ollama OpenAI compatibility documentation — local Responses endpoint, supported fields, and statefulness limitations.
- Ollama tool-calling documentation — supported tool-call flow and Qwen3 examples.
- Ollama Qwen3 model page — available model variants and model information.
- OpenAI function-calling guide — Responses function-call and function-call-output item structure.
- OpenAI Python SDK repository — client installation and API usage.
