Regression-Test Local LLMs with Promptfoo and Ollama: A Practical Evaluation Suite (2026)

A local model can answer five hand-picked prompts well and still break the moment you change its system prompt, switch quantization, update Ollama, or try a real edge case. Manual spot-checking is useful, but it is a poor regression test: people forget the exact input, overlook small failures, and struggle to compare two model versions fairly.

This guide shows how to build a repeatable evaluation suite for local language models using Promptfoo and Ollama. You will write a small set of test cases, run them against a local model, inspect pass/fail results, and expand the suite before changing a model or prompt in an application. The example uses a lightweight Qwen3 model as a smoke-test target; it is not a benchmark or a recommendation that one model is best for every task.

Documentation checked: 23 October 2026. CLI commands, provider identifiers, and model tags can change, so verify them against the linked official documentation if your installed version behaves differently.

What you will build

• A version-controlled set of prompts and test cases for a local model.
• Simple, deterministic checks for required words and output formats.
• A repeatable command you can run after changing a prompt, model, or runtime.
• An optional side-by-side comparison between two locally installed models.
• A privacy checklist that distinguishes local model inference from telemetry and optional cloud features.

This is a small evaluation harness, not proof that a model is safe, factual, or production-ready. It helps you catch known failure patterns early and gives you a consistent starting point for human review.

Who this guide is for

Use this workflow if you build a local AI feature, maintain a prompt used by an application, compare small models for a constrained task, or want a basic regression check in a developer workflow. It is especially useful when you need to ask, “Did this change make our known examples worse?” without relying on memory or a few screenshots.

If you only want to chat with a model, you do not need an evaluation framework. If you need statistically meaningful benchmark results, safety certification, or a production-grade evaluation program, this starter suite is only one component of that work.

Prerequisites

• Ollama: installed and running. Use the official download page and the Ollama documentation.
• Node.js and npm: needed to install and run the Promptfoo CLI. Use a currently supported Node.js release from the official Node.js website.
• Disk space and memory: enough for the model you choose and the other applications you run. A smaller model is easier for a first smoke test.
• A test project folder: keep evaluation files separate from production secrets and customer data.

Step 1: Download a model and verify Ollama first

Start with the small qwen3:0.6b tag listed in the official Ollama model library. It is suitable for checking that the wiring works; its small size does not mean it will be accurate enough for your real application.

ollama --version
ollama pull qwen3:0.6b
ollama list
ollama run qwen3:0.6b

At the interactive prompt, ask a simple question, then exit the chat. The model should appear in ollama list. If the download fails or the model cannot answer directly, fix that before introducing Promptfoo.

Official reference: Ollama's Qwen3 model library. If the exact tag is unavailable in your environment, select a tag that is currently listed in the library and use the same name in the Promptfoo provider configuration.

Step 2: Create an isolated Promptfoo project

Make a new folder so the configuration and result files do not get mixed with application code:

mkdir local-llm-evals
cd local-llm-evals
npm init -y
npm install --save-dev promptfoo
npx promptfoo --version

The installation is local to this project. Keeping Promptfoo in the project's development dependencies makes it easier to record the version used by a team or CI job than relying indefinitely on a floating global installation. Commit the package manifest and lockfile to your repository if you want repeatable dependency installation.

Promptfoo's official getting-started guide also documents initializing an example project with npx promptfoo@latest init --example getting-started. This article builds the minimal configuration directly so you can see each part.

Step 3: Write your first evaluation configuration

Create a file named promptfooconfig.yaml in the project directory. The configuration below uses three simple developer-oriented checks. They are intentionally easy to understand and should be treated as smoke tests, not a meaningful capability benchmark.

description: Local Ollama prompt regression suite

prompts:
  - |
    You are a concise developer assistant.
    Answer the question accurately in one or two sentences.
    Question: {{question}}

providers:
  - id: ollama:chat:qwen3:0.6b
    label: Qwen3 0.6B local
    config:
      temperature: 0
      think: false

tests:
  - vars:
      question: "What is 2 + 2? Answer with the number."
    assert:
      - type: contains
        value: "4"

  - vars:
      question: "What is the standard HTTP method commonly used to retrieve a resource?"
    assert:
      - type: icontains
        value: "GET"

  - vars:
      question: "In Python, which keyword begins a function definition?"
    assert:
      - type: icontains
        value: "def"

The prompt contains the template variable {{question}}. Each test supplies a different value, and the assertions check whether the generated output contains a required answer fragment. The Ollama provider uses its chat endpoint. The temperature and think settings are configuration options documented by Promptfoo for its Ollama provider; if your installed Ollama/model combination does not support a setting, remove it or consult the current provider guide.

Provider reference: Promptfoo's Ollama provider documentation.

Step 4: Run the tests and inspect the results

From the same folder, run:

npx promptfoo eval

Promptfoo reads promptfooconfig.yaml, sends each test prompt to the configured local provider, and reports the outputs and assertion results. A test passes only when its configured assertions pass. The actual output is generated by your installed model, so it can differ from examples you see in documentation.

To open the local results viewer, run:

npx promptfoo view

Review both the status and the output text. A green check is only as meaningful as the assertion you wrote: the phrase “GET” could appear in an incorrect explanation, for example. The official configuration guide explains how prompts, providers, variables, and assertions form an evaluation matrix.

Step 5: Make the test suite represent your real application

The toy checks prove that the test runner and local provider are connected. Replace or supplement them with cases drawn from the task your software actually performs. A useful starter suite often includes:

• Normal cases: common, well-formed inputs that users submit frequently.
• Boundary cases: empty strings, long inputs, missing fields, unexpected types, or unsupported requests.
• Required behavior: specific facts, mandatory sections, valid JSON, or a refusal when the request is outside the supported task.
• Negative cases: prompts that try to override your system instructions or make the model invent data.
• Historical failures: examples from bugs you have already fixed. These are especially valuable regression tests.

For example, if your application summarizes support tickets, test a normal ticket, a ticket with no reproduction steps, one with a stack trace, and one that contains instructions embedded in quoted customer text. Decide in advance what a correct output must contain and what it must not do.

Prefer checks that can be verified reliably. Exact string checks are good for fixed labels and required tokens. JSON/schema assertions are useful when your application consumes structured output. Semantic or model-graded assertions can help with open-ended answers, but they introduce their own variability and should be reviewed carefully. Promptfoo documents its available checks in the deterministic assertions reference.

Step 6: Check structured output when your app expects JSON

If a downstream API expects JSON, do not rely on a prompt saying “return JSON” and then assume the result is valid. Add an explicit test for JSON structure. For example, a test case can require a response containing an object with a string status field and an integer count field:

  - vars:
      question: "Return JSON with status set to ok and count set to 3."
    assert:
      - type: contains-json
        value:
          type: object
          required:
            - status
            - count
          properties:
            status:
              type: string
            count:
              type: integer

Add this case to the tests list only after adapting the prompt so that JSON is an explicit requirement. The schema assertion checks shape, not whether the values are semantically correct. For a real application, add checks for permitted enum values, required fields, ranges, and cross-field rules. Promptfoo's deterministic assertion documentation describes JSON assertions and schema validation.

Step 7: Compare a second model without changing the test set

Once the first model works, compare it with another local model using the same prompts and assertions. For example, if your machine has enough memory, download the Qwen3 4B tag:

ollama pull qwen3:4b
ollama list

Then add a second entry under providers in promptfooconfig.yaml:

  - id: ollama:chat:qwen3:4b
    label: Qwen3 4B local
    config:
      temperature: 0
      think: false

Run npx promptfoo eval again. Promptfoo evaluates the configured prompts and tests against each provider, allowing you to compare pass/fail results and inspect the actual outputs side by side. Check the official Qwen3 library page for available tags, and confirm that each exact tag is installed before the run.

Do not declare a winner from three toy cases. Expand the dataset, repeat important cases where appropriate, inspect failures, and measure latency or resource use separately if those metrics matter. A larger model may behave differently without being better for your specific workload.

Step 8: Turn evaluations into a regression habit

Run the suite whenever you make a change that could affect outputs:

1. Changing the system prompt or a prompt template.
2. Switching model tags, quantization, context settings, or inference parameters.
3. Updating Ollama, Promptfoo, or the model files.
4. Changing a retrieval pipeline, tool schema, or post-processing step.
5. Fixing a bug that previously caused a known bad answer.

Keep the evaluation configuration and a short note about the intended behavior in version control. For a CI pipeline, install the project's locked dependencies and run the same evaluation command. Decide how failures should be handled: a failed assertion can block a release, trigger a manual review, or create a report depending on the risk of the application.

For RAG applications, the suite should test retrieval and answer behavior separately where possible. A correct-looking final answer can hide a retrieval failure, and a good retrieved passage does not guarantee a grounded answer. GyanAangan's production RAG evaluation guide covers evaluation sets, retrieval quality, grounded answers, and observability in more detail.

Troubleshooting common failures

For additional provider-specific diagnostics, use Promptfoo's Ollama documentation and GyanAangan's Ollama provider troubleshooting guide.

Privacy and security: local inference is not the whole story

• Keep the model endpoint local unless remote access is intentional. The default provider uses localhost. Do not expose Ollama's port to the public internet just to make a test run work.
• Use synthetic or sanitized test data. Do not put real customer records, secrets, API keys, private source code, or production logs into an evaluation file unless your data-handling policy permits it.
• Understand telemetry. Promptfoo's official telemetry documentation says the CLI collects basic usage events by default and describes how to disable telemetry with PROMPTFOO_DISABLE_TELEMETRY=1. Check the current telemetry documentation before using it in a restricted environment.
• Avoid accidental sharing. Do not run cloud sharing commands or enable Cloud sync for sensitive results unless you have reviewed what the feature uploads. Promptfoo's sharing documentation explains that shared evaluation snapshots can include prompts, variables, outputs, traces, and configuration fields.
• Do not treat a local provider as an automatic offline guarantee. The model calls in this configuration target Ollama locally, but package installation, update checks, telemetry, account features, or other optional services may still use the network.
• Review evaluation files as executable configuration. Promptfoo supports custom assertions and other extensibility features. Only run configs and referenced scripts you trust.

Frequently asked questions

Does Promptfoo need a paid model API key for this tutorial?

No paid model API key is configured here: the test provider points to Ollama on your machine. Other Promptfoo features may have different requirements, so verify the data flow before enabling cloud-backed workflows.

Are passing tests proof that my model is accurate?

No. They prove only that the configured outputs satisfied the checks you wrote for those cases. A weak dataset can miss serious errors, and string matching can pass an answer that is wrong in context.

Can I test models other than Qwen3?

Yes. Promptfoo supports multiple providers, including other Ollama chat and completion model tags. Check the official provider docs and the model's capabilities, then change the provider ID to a tag installed on your machine.

Can I use this for RAG or agent workflows?

Yes, as a starting point. Add representative queries and assertions, but also evaluate retrieval, tool selection, permissions, and final-answer grounding. A text-only answer check cannot validate an entire multi-step agent workflow by itself.

Should I use model-graded assertions?

They can help assess open-ended qualities that are difficult to express as exact checks, but they add grader-model behavior and potential variability. Start with deterministic checks for required behavior, then add model-graded checks only where they provide clear value and can be reviewed.

Final checklist

• Ollama runs and the exact model tag is installed.
• Promptfoo is installed in the project and its version is recorded.
• The configuration uses the intended local provider.
• The test set covers normal cases, edge cases, and known past failures.
• Assertions check the actual contract instead of merely making the report look green.
• Results are reviewed before changing a model or prompt in production.
• Telemetry, network access, local result files, and sharing settings match your privacy requirements.

A small, honest evaluation suite is more useful than a long list of prompts nobody reruns. Start with a handful of important behaviors, keep the tests next to your code, and add a new regression case whenever you discover a failure worth preventing.