Ollama + LLM CLI: Build a Private Terminal AI Workflow with Saved Prompts and Searchable History (2026)

Most local-AI tutorials stop once a model answers one question. A useful developer workflow needs more: a repeatable way to ask questions from the terminal, reuse prompts across files, continue conversations, and find an answer you generated last week. You can build that workflow with Ollama and LLM, Simon Willison’s command-line tool for working with language models.

This guide sets up a local-first terminal workflow. You will run an Ollama model, invoke it through the LLM CLI, save reusable prompt templates, pipe files into prompts, inspect conversation history, and turn logging off when a prompt contains sensitive data. The examples are designed for macOS and Linux shells; most commands also work in Windows WSL.

Privacy note: running inference locally does not automatically make every part of a workflow private. LLM’s prompt/response history is logged to a local SQLite database by default, and remote model endpoints or plugins can introduce network calls. We will inspect those settings rather than assume.

What you are building

• Ollama: downloads and runs a model on your machine and exposes a local API.
• LLM CLI: sends prompts to models, supports reusable templates and interactive chats, and records history in SQLite.
• A repeatable workflow: pipe a file or command output into a named prompt, then search the saved conversation when you need it again.

This is not a new model or a benchmark. It is a practical terminal interface around a model you already run locally.

Prerequisites

• A working installation of Ollama.
• Python tooling through uv, or use pipx if that is already part of your setup.
• Enough RAM for the model you choose. Start with a smaller model if your machine has limited memory.
• A terminal and a shell such as Bash, Zsh, or a WSL shell.

For this walkthrough, we use the example model llama3.2. Model availability, quality, memory use, and speed vary by hardware; choose another model if it fits your machine better.

Step 1: Start Ollama and pull a model

Install Ollama from its official download page, then open the application or start its service according to your operating system. Pull the example model:

ollama pull llama3.2
ollama list

The first command downloads the model. The second lists models available to your local Ollama installation. If you already have a suitable model, you can skip the download and use its exact name in the commands below.

Check that Ollama responds locally:

curl http://localhost:11434/api/tags

You should receive JSON describing the local model library. If the connection is refused, Ollama may not be running, or it may be listening on a different host or port. Do not expose port 11434 to the public internet just to make this test work.

Step 2: Install the LLM CLI and Ollama plugin

Install LLM as an isolated command-line tool with uv:

uv tool install llm

Confirm the command is available:

llm --version
llm models

Now install the Ollama provider plugin into the same LLM environment:

llm install llm-ollama

The plugin discovers models available from Ollama. List the discovered models and capabilities:

llm ollama models

Current versions of the plugin identify Ollama-backed models with the ollama/ prefix. For example, a local llama3.2 model is addressed as ollama/llama3.2. Use the exact name shown by your own model list if it differs.

Official references: LLM documentation, plugin installation guide, and the llm-ollama plugin package.

Step 3: Run a prompt from your terminal

Send a one-shot prompt to the local model:

llm -m ollama/llama3.2 "Explain the difference between a process and a thread in three concise bullets."

The CLI prints the response as it is generated. This is useful for quick explanations, summarising logs, drafting shell commands for review, or asking questions about a configuration file without opening a separate chat interface.

For a multi-turn conversation, run:

llm chat -m ollama/llama3.2

Type questions at the prompt and enter exit or quit when finished. The CLI records the conversation in its local history unless logging has been disabled. You can also continue the most recent conversation with llm -c, where supported by the selected model.

Step 4: Save a reusable prompt template

Typing the same instructions repeatedly is easy to avoid. LLM can save a prompt, system instruction, model choice, and related settings as a named template.

Create a template that explains configuration files and calls out risks without changing them:

llm --system "You are a careful senior developer. Explain the configuration, identify likely operational risks, and suggest checks. Do not claim to have executed commands or changed files." \
  -m ollama/llama3.2 \
  --save explain-config

List saved templates:

llm templates

Now pipe a file into the template:

cat nginx.conf | llm -t explain-config

Replace nginx.conf with a file you are comfortable sharing with the local model. The input is added to the saved prompt template. If you prefer, open a template for editing with:

llm templates edit explain-config

Templates make repeated work more consistent, but they do not guarantee correct output. Review any suggested configuration changes manually and validate them with the relevant tool before deploying.

Step 5: Use the CLI in ordinary shell pipelines

One of the biggest benefits of a CLI is that it can consume the output of existing developer tools. For example, ask the model to explain a service status without granting it permission to operate the service:

systemctl status nginx --no-pager | llm -m ollama/llama3.2 \
  -s "Summarize the status output. Separate observed facts from possible causes. Suggest read-only diagnostic commands only."

Or summarise a test report:

pytest -q 2>&1 | llm -m ollama/llama3.2 \
  -s "Summarize the failures, group likely root causes, and quote the exact failing test names."

These commands pass text to the model; they do not make the model a reliable substitute for reading the original output. For long logs, trim the input to the relevant time range or error section so the useful evidence is not lost in noise.

Before piping output, check whether it contains access tokens, database credentials, customer data, private source code, or other secrets. Local inference reduces the need to send content to a hosted model, but it does not remove the need to protect the input and local logs.

Step 6: Find earlier answers in the local history

LLM stores prompts and responses in a SQLite database by default. Check the logging state and database location:

llm logs status
llm logs path

Display recent entries:

llm logs -n 5

Search prompt and response text for a term:

llm logs -q "nginx"

You can inspect a recent conversation with llm logs -c, or request JSON output for processing with another command using llm logs --json. The available flags can vary with the installed version, so use llm logs --help if a command differs on your system.

The history is useful for recovering a command explanation or comparing how a prompt behaved after you revised it. It is not a secure secrets vault. Anyone who can read the database or your account files may be able to read the saved prompts and responses.

Step 7: Turn off logging for sensitive prompts

Because logging is enabled by default, decide deliberately what should be retained. To disable logging for future prompts:

llm logs off
llm logs status

To re-enable it later:

llm logs on

For a single prompt that should not be logged, use the documented no-log option:

llm -m ollama/llama3.2 -n "Explain this generic error message: connection refused"

Check llm --help if your installed version uses different option placement. Disabling logging prevents future entries from being saved through this feature; it does not erase older records. Treat deletion and retention of the existing SQLite database as a separate data-management task, and back it up before making manual changes.

Step 8: Troubleshoot common problems

The model is missing from LLM

• Run ollama list to confirm the model is installed.
• Run ollama pull llama3.2 if you intend to use that example and it is missing.
• Run llm ollama models to refresh your understanding of what the plugin can see.
• Use the full model identifier shown by the plugin, including the ollama/ prefix.

Ollama is not reachable

• Confirm the Ollama application or service is running.
• Test curl http://localhost:11434/api/tags.
• If Ollama intentionally runs on another host, configure OLLAMA_HOST according to the plugin documentation and secure that connection. Do not assume a remote endpoint is private just because it is on a LAN.

The first response is slow

The first request may include model loading. Large models and long prompts also need more memory and processing time. Try a smaller model, shorten the input, and compare subsequent requests before diagnosing a performance regression. This guide does not claim benchmark results for your hardware.

The output is incorrect or invents details

Ask the model to quote the relevant lines and label uncertainty, then verify those lines yourself. Keep the source output available. For production decisions, add deterministic checks and tests rather than relying on an unverified natural-language summary.

Security and privacy checklist

• Inspect logging: check llm logs status and disable logging before handling sensitive content.
• Keep Ollama local by default: avoid public exposure of its API port; use authenticated, encrypted access controls if a remote endpoint is necessary.
• Protect templates and history: prompts may include proprietary details, internal paths, stack traces, or snippets of source code.
• Do not execute model suggestions blindly: treat generated shell commands and configuration edits as untrusted proposals.
• Review plugin dependencies: install only the plugins you need, from their official project or package pages, and update them deliberately.
• Remember the limits of “local”: the model inference can be local while other integrations, plugins, or explicitly configured endpoints still use the network.

Useful next steps

If you want to expand this workflow, start with GyanAangan’s Ollama CLI commands cheat sheet for model management. For application code that needs predictable JSON, see Ollama structured outputs with Pydantic. If you are comparing local runners rather than using a terminal interface, read the Unsloth vs Ollama guide.

FAQ

Does this require an API key?

Not for a local Ollama model on the default local endpoint. Remote or hosted providers may require credentials and can have different privacy and billing implications.

Does the LLM CLI download a model itself?

No. Ollama manages the model download in this walkthrough. The llm-ollama plugin makes Ollama-hosted models available to the LLM CLI.

Are prompts stored only in memory?

No. By default, LLM saves prompts and responses to a local SQLite database. Use the logging controls above if you do not want future prompts recorded.

Can I use another Ollama model?

Yes. Pull or install the model in Ollama, list the models discovered by the plugin, and replace ollama/llama3.2 with the model identifier shown on your machine.

Is this a replacement for an IDE assistant?

It solves a different problem. The CLI is useful for quick prompts, shell pipelines, reusable templates, and searchable history. An IDE assistant may be better for inline edits and interactive project navigation.

Official references

• LLM CLI documentation
• LLM usage and interactive chat
• LLM prompt templates
• LLM SQLite logging and search
• llm-ollama package documentation
• llm-ollama source repository
• Ollama API documentation