RAG Evaluation Deep Dive: Ragas vs DeepEval vs Phoenix with Local Ollama Judges (2026)
A RAG chatbot can sound confident while retrieving the wrong documents, ignoring the right passage, or inventing details that were never in its sources. A single “answer quality” score will not tell you which part failed. To improve a retrieval-augmented generation system, evaluate retrieval and generation separately, keep a fixed test set, and compare changes against the same questions.
This deep-dive, updated November 20, 2026, compares Ragas, DeepEval and Arize Phoenix for practical RAG evaluation. It then builds a small repeatable evaluation workflow using Python, DeepEval and a local Ollama model as the judge. The goal is not to declare one framework universally best; it is to help you choose the right tool for offline experiments, CI regression tests and production debugging.
Who this guide is for
- Developers building document Q&A, internal knowledge assistants or support bots.
- Teams tuning chunk size, top-K retrieval, rerankers, embeddings or generation prompts.
- Engineers who want evaluation without sending every prompt and document to a hosted judge model.
- Teams that need to distinguish retrieval failures from answer-generation failures.
What you will build
You will define a small, versionable evaluation dataset; record the answer and retrieved context produced by your RAG application; score retrieval relevance, answer relevance and faithfulness; run the same cases after a change; and use the results to decide whether the retriever, generator or both need attention. The example uses a local Ollama judge through DeepEval. Ragas and Phoenix are compared as alternative parts of the same workflow.
1. Understand the two stages before choosing metrics
A basic RAG request has two distinct stages:
- Retrieval: the query is embedded or otherwise searched, and the retriever returns candidate chunks.
- Generation: the application sends the query and retrieved context to a language model, which produces the answer.
If retrieval returns the wrong passage, a capable generator may still produce a plausible but unsupported answer. If retrieval is correct but the model ignores it, changing the embedding model will not fix the generation failure. Evaluate the stages separately.
| Failure | Likely stage | Useful signal |
|---|---|---|
| Relevant document never appears in top-K | Retrieval | Recall@K or contextual recall |
| Relevant chunk appears below many irrelevant chunks | Retrieval / reranking | Precision@K, MRR, NDCG or contextual precision |
| Retrieved text is mostly unrelated | Retrieval / chunking | Contextual relevance |
| Answer contains claims absent from the context | Generation | Faithfulness / groundedness |
| Answer is grounded but does not answer the question | Generation | Answer relevance |
| Answer omits a required fact | Retrieval or generation | Reference-based completeness or correctness checks |
For retrieval, classic information-retrieval metrics such as Recall@K, Precision@K, Mean Reciprocal Rank and NDCG are especially useful when you can label which documents are relevant. LLM-judge metrics can help when labels are expensive, but they are estimates, not objective ground truth.
2. Ragas vs DeepEval vs Phoenix
| Tool | Best fit | Strength | Trade-off |
|---|---|---|---|
| Ragas | Metric-driven evaluation experiments for RAG and agentic systems | Purpose-built metric catalogue including context precision, context recall, faithfulness and response relevancy; supports configurable evaluation LLMs and embeddings | You still need to design a representative dataset and understand which metric assumptions fit your task |
| DeepEval | Test-style evaluation and CI regression checks | Test cases, thresholds, explainable metrics, custom criteria and documented local Ollama / OpenAI-compatible model configuration | LLM-judge results depend on the judge model and prompt; local models may struggle with nuanced scoring or structured outputs |
| Arize Phoenix | Tracing, inspection and evaluation of live or experimental application runs | OpenTelemetry-oriented tracing, datasets, experiments and retrieval / response evaluators in one observability workflow | It is a broader observability platform, so it may be more than you need for a small offline test suite |
Practical selection: start with DeepEval if you want tests that can fail a CI job; choose Ragas if your work is centred on metric-driven RAG experiments and configurable evaluation pipelines; add Phoenix when you need traces and want to inspect which retrieved chunks and model calls caused a production failure. These tools are not mutually exclusive.
Which metrics should you actually start with?
- Contextual relevance: are the retrieved chunks relevant to the query?
- Contextual precision: are the useful chunks ranked ahead of irrelevant ones?
- Contextual recall: did retrieval capture the information needed to answer?
- Faithfulness: are the answer's claims supported by the retrieved context?
- Answer relevance: does the answer address the user's question?
Do not collect every available metric by default. Start with a small set tied to the failure you want to prevent. Add reference-based checks when you have trustworthy expected answers or relevance labels.
3. Create a reproducible evaluation dataset
Do not begin by asking a judge model to invent a score for a handful of cherry-picked prompts. Create a small dataset from real user questions, support tickets or known information needs. Include easy questions, multi-hop questions, ambiguous questions, questions with no answer in the corpus, and cases where two documents contain similar but conflicting details.
A useful test case stores at least:
input: the user's question.actual_output: the answer produced by the current RAG application.retrieval_context: the exact chunks supplied to the generator, in their original order.expected_outputor reference labels: optional but valuable for reference-based metrics.case_id, corpus version, retriever configuration and prompt version: metadata for reproducibility.
For example, save a JSON Lines file named eval_cases.jsonl:
{"input":"How long are audit logs retained?","actual_output":"Audit logs are retained for 90 days.","retrieval_context":["The audit log retention period is 90 days. Records can be exported by workspace administrators."],"expected_output":"Audit logs are retained for 90 days."}
{"input":"Can a viewer export audit logs?","actual_output":"Yes, any workspace member can export them.","retrieval_context":["Only workspace administrators can export audit logs."],"expected_output":"No. Only workspace administrators can export audit logs."}
{"input":"Does the product support biometric login?","actual_output":"The supplied documentation does not specify biometric login support.","retrieval_context":["The product supports password and SSO authentication."],"expected_output":"The supplied documentation does not establish biometric login support."}
These are illustrative examples, not measured product results. In a real dataset, keep the exact retrieved text and the exact answer from the same application run. Do not replace the retrieval context with a hand-written ideal passage: that would hide retrieval defects.
4. Prepare a local judge with Ollama
DeepEval documents a native Ollama integration. This lets you run judge-model inference against a local Ollama server rather than using a hosted provider by default. The judge is separate from the model that generated the RAG answer; you can evaluate one generator with a different judge.
Install and start Ollama using its official installation instructions, then pull a model available in your environment. The following is an example model choice, not a recommendation that this size will be accurate enough for every evaluation:
ollama pull qwen3:8b
ollama run qwen3:8b
In a separate terminal, create a Python environment and install DeepEval:
python -m venv .venv
source .venv/bin/activate
python -m pip install --upgrade pip
pip install deepeval
On Windows PowerShell, activate the environment with .venv\Scripts\Activate.ps1. Check the current DeepEval installation documentation for supported Python versions and any platform-specific requirements before pinning dependencies in a production project.
Configure DeepEval to use the local model:
deepeval set-ollama --model=qwen3:8b --base-url="http://localhost:11434"
The CLI configuration sets Ollama as the default evaluation model. You can also configure the model directly in Python for an individual metric, which makes the choice explicit in the test code.
5. Write a first DeepEval test
Create test_rag_eval.py. The example uses the standard DeepEval test-case and metric pattern. It checks faithfulness and answer relevance for one recorded RAG response:
import json
from pathlib import Path
import pytest
from deepeval import assert_test
from deepeval.metrics import AnswerRelevancyMetric, FaithfulnessMetric
from deepeval.models import OllamaModel
from deepeval.test_case import LLMTestCase
judge = OllamaModel(
model="qwen3:8b",
base_url="http://localhost:11434",
temperature=0,
)
faithfulness = FaithfulnessMetric(
threshold=0.7,
model=judge,
include_reason=True,
)
answer_relevancy = AnswerRelevancyMetric(
threshold=0.7,
model=judge,
include_reason=True,
)
def load_cases():
path = Path("eval_cases.jsonl")
return [json.loads(line) for line in path.read_text().splitlines() if line.strip()]
@pytest.mark.parametrize("case", load_cases())
def test_rag_answer(case):
test_case = LLMTestCase(
input=case["input"],
actual_output=case["actual_output"],
retrieval_context=case["retrieval_context"],
expected_output=case.get("expected_output"),
)
assert_test(test_case, [faithfulness, answer_relevancy])
Run the tests with:
deepeval test run test_rag_eval.py
Thresholds such as 0.7 are starter values for demonstrating the test structure, not universal quality standards. Calibrate thresholds against human-reviewed examples for your domain. If your installed DeepEval version changes an import or test-runner interface, follow the current official quickstart and pin the tested package version.
6. Add retrieval evaluation instead of blaming the answer model
The previous test evaluates the answer using the context that was retrieved. It does not prove the retriever found all relevant documents. Add contextual metrics when you have reference information or want to measure the quality of the retrieved context.
from deepeval.metrics import (
ContextualPrecisionMetric,
ContextualRecallMetric,
ContextualRelevancyMetric,
)
contextual_relevancy = ContextualRelevancyMetric(
threshold=0.7,
model=judge,
include_reason=True,
)
contextual_precision = ContextualPrecisionMetric(
threshold=0.7,
model=judge,
include_reason=True,
)
contextual_recall = ContextualRecallMetric(
threshold=0.7,
model=judge,
include_reason=True,
)
Attach these metrics to test cases with a meaningful expected_output where required by the metric. DeepEval's documentation distinguishes referenceless metrics from contextual precision and recall, which use reference information. If you do not have reliable references, do not fabricate them merely to make a metric run.
For a retrieval system where you can label relevant document IDs, also calculate classic Recall@K and ranking metrics directly. Those metrics are deterministic given the labels and retrieved IDs, and they do not need an LLM judge.
7. Use Ragas when you want a metric-oriented experiment
Ragas offers a broader set of metrics for RAG and agentic workflows, including context precision, context recall, faithfulness and response relevancy. Its evaluation API accepts a dataset, selected metrics and optional LLM and embedding implementations.
Ragas documents using an OpenAI-compatible Ollama endpoint as an evaluation LLM. The pattern below shows the documented client approach; the model identifier must match a model installed in your Ollama instance and the selected model must be capable enough for the chosen metric:
from openai import OpenAI
from ragas.llms import llm_factory
client = OpenAI(
api_key="ollama",
base_url="http://localhost:11434/v1",
)
judge_llm = llm_factory(
"qwen3:8b",
provider="openai",
client=client,
)
Then configure the metrics and evaluation dataset using the current Ragas quickstart and metric references. Ragas APIs have evolved across releases; keep the code aligned with the installed version rather than mixing examples from older tutorials with the latest package.
8. Add Phoenix when you need to debug real runs
Offline test results tell you that a case failed. A trace helps explain what happened in the actual application: which retriever ran, which chunks were returned, how long each stage took, and which model call generated the answer.
Arize Phoenix is an open-source observability and evaluation platform with tracing, datasets, experiments and evaluators. It uses OpenTelemetry-based instrumentation and supports workflows for inspecting retrieved documents and evaluating retrieval relevance. A practical division of responsibility is:
- Use DeepEval or Ragas to run a versioned offline evaluation suite.
- Use Phoenix traces to inspect representative application executions and diagnose failures.
- Promote reviewed failures from traces into the permanent evaluation dataset.
Phoenix is not a substitute for a carefully curated test set. Traces show what happened; they do not automatically establish what the correct answer should have been.
9. Turn the evaluation into a regression gate
Once the test set is stable, run it whenever you change a retrieval or generation component. At minimum, record:
- Git commit and application version.
- Corpus snapshot or document-index version.
- Embedding model and version.
- Chunk size, overlap, top-K and reranker configuration.
- Generator model, prompt version and decoding parameters.
- Judge model, judge prompt and metric package version.
- Per-case scores, reasons, failures and total evaluation cost or local compute time.
Do not rely on a single average. A mean faithfulness score can hide one critical failure on a high-risk question. Track per-case regressions and inspect the cases that changed. For a business-critical system, define hard rules for sensitive topics—for example, a required refusal when the corpus contains no answer—rather than hoping a broad semantic score captures the requirement.
A simple CI policy might be:
- Fail the build if a deterministic retrieval recall check falls below the agreed threshold.
- Fail or require review if any high-severity test case becomes unfaithful.
- Compare aggregate judge metrics with a baseline, but treat small score changes as signals for review rather than proof of improvement.
- Require human review when the judge model is uncertain, inconsistent or disagrees with reference labels.
10. Common mistakes and troubleshooting
- Every metric passes but users still complain: your dataset may not represent real traffic. Add real, privacy-reviewed examples and hard cases.
- Judge outputs are malformed: use a model with reliable structured output, reduce prompt complexity, or choose a stronger judge. Some metrics require structured responses.
- Ollama connection refused: confirm the Ollama service is running and that the base URL and port match the machine running DeepEval.
- Scores change between identical runs: judge models can be nondeterministic. Fix temperature where supported, record model versions, repeat borderline cases and avoid treating tiny score movements as meaningful.
- Contextual recall cannot run: check whether the metric requires an expected output or reference context and ensure your test case supplies it.
- Evaluation takes too long: begin with a small representative suite, run expensive metrics selectively and use asynchronous or batched evaluation only where the framework and provider support it.
- Local judge scores everything highly: inspect explanations against human labels. A small local model may be fast and private but too weak to grade nuanced factuality reliably.
11. Security, privacy and evaluation integrity
- Sanitize evaluation data: remove secrets, credentials, personal data and customer identifiers before storing test cases or traces.
- Check where judge calls go: a local generator does not guarantee a local evaluator. Verify the configured judge and embedding endpoints.
- Protect observability data: traces can contain full prompts, retrieved documents and generated answers. Apply access controls, retention limits and redaction.
- Defend against prompt injection in documents: retrieved text is untrusted input. Do not let a document's instructions override system policy or grant tool permissions.
- Separate judge and application credentials: do not expose production API keys to test prompts or store them in the dataset.
- Version your evaluation protocol: changing the judge model, judge prompt or metric version can change scores even if the RAG application did not change.
- Keep human review in the loop: LLM-as-a-judge is a measurement aid, not a compliance certification or a replacement for domain experts.
12. A practical adoption plan
- Day one: capture 20–30 representative questions and save exact outputs and retrieved chunks.
- Next: add a small set of deterministic retrieval checks and a faithfulness check.
- Then: compare DeepEval and Ragas on the same labelled examples if you need to choose a framework.
- After that: add Phoenix tracing if debugging real requests is difficult.
- Before release: run the frozen evaluation set against the previous and candidate configurations, inspect regressions and document the decision.
Frequently asked questions
Is Ragas better than DeepEval?
Neither is universally better. Ragas is a strong fit for metric-oriented RAG experiments. DeepEval is convenient when you want test cases, thresholds and CI-style regression checks. Compare them on your own labelled cases, judge model and cost constraints.
Can I evaluate RAG completely offline?
You can keep generation, embeddings and judge inference local with tools such as Ollama, provided your selected models and evaluation stack support the required operations. Package installation, model downloads and telemetry may still involve network access unless you prepare an offline environment.
Does a high faithfulness score mean the answer is correct?
No. Faithfulness asks whether the answer is supported by the retrieved context. The context itself can be outdated or wrong. Use source-quality checks, reference answers and human review for important domains.
Do I need expected answers for every test case?
No. Some relevance and faithfulness metrics are referenceless. Reference answers or document relevance labels enable stronger checks such as contextual recall and correctness, but they must be trustworthy.
Can I use a small local model as the judge?
Yes, but validate its judgments against human-reviewed examples. Smaller models may be less reliable on subtle contradictions, multi-hop reasoning and structured outputs. Local inference improves control over where data is processed; it does not automatically improve evaluation quality.
Should Phoenix replace my evaluation framework?
Not necessarily. Phoenix is particularly useful for tracing and diagnosing application runs. You can keep Ragas or DeepEval for a focused offline test suite and use Phoenix to inspect production behavior.
Conclusion
Reliable RAG evaluation is a loop, not a leaderboard. Freeze a representative dataset, measure retrieval separately from generation, inspect individual failures, and rerun the same cases after every meaningful change. Use DeepEval for test-oriented regression checks, Ragas for metric-driven experiments, and Phoenix when traces are needed to understand live behavior. Start small, keep the judge honest with human-reviewed examples, and never confuse a model-generated score with ground truth.
Further reading on GyanAangan
- Incremental Local RAG with Ollama and SQLite — build and update a local retrieval corpus.
- Trace Local Ollama Apps with OpenTelemetry and Phoenix — instrument local AI applications and inspect traces.
- LLM Semantic Caching with Redis and LiteLLM — understand caching trade-offs in LLM systems.
- Ollama Structured Outputs with Pydantic — build more reliable local model output pipelines.
Official documentation and references
- Ragas documentation and available metrics.
- Ragas quickstart, including its documented Ollama-compatible LLM setup.
- DeepEval RAG evaluation guide and faithfulness metric.
- DeepEval Ollama integration and local embedding configuration.
- DeepEval CLI configuration and official repository.
- Arize Phoenix repository, RAG evaluator reference and Phoenix client evaluation helpers.
- Ollama documentation and Ollama repository.
Verification note: framework capabilities and integration patterns were checked against the linked official documentation on November 20, 2026. The code is an implementation template and was not executed against a live RAG application. Pin package versions and validate the imports and CLI options in your own environment before adopting it in CI.
