RAG faithfulness: how to evaluate a RAG and where the metric stops

RAG faithfulness is the first metric most teams set up when they want to evaluate a RAG: it checks that the model’s answer is based on the retrieved passages rather than on invention. It is a good hallucination detector. It is not, however, a measure of truth.

A high faithfulness score is reassuring, sometimes wrongly. An answer can be perfectly faithful to an outdated document, or to an off-topic passage. For an SRE team that plugs an assistant into its runbooks, the difference matters: a faithful but stale procedure is still dangerous during an incident.

This article explains what faithfulness measures, how to compute it with Ragas and LlamaIndex, and above all where it stops. It complements our LlamaIndex RAG tutorial on internal documentation, which builds the pipeline; here the focus is on reading the scores.

RAG faithfulness: what the metric measures

According to the Ragas documentation, faithfulness measures how factually consistent a response is with the retrieved context. It ranges from 0 to 1, and a higher score means stronger consistency. A response is faithful when all of its claims can be supported by the context.

The calculation has three steps, per the same page:

1. identify the claims made in the response;
2. check, for each one, whether the retrieved context allows it to be inferred;
3. divide the number of supported claims by the total number of claims.

The documentation’s example splits a response into two statements. One is supported and the other is not, so the score is 1 out of 2, or 0.5.

The LlamaIndex documentation describes the same idea more briefly: faithfulness checks that an answer is supported by the retrieved contexts, which helps detect hallucination.

Evaluate a RAG: separate retrieval from generation

A RAG has two stages, and a single score does not say which one is failing. The research paper that introduces Ragas distinguishes three dimensions: retrieval quality, the model’s faithful use of the context, and generation quality. Its authors describe a reference-free evaluation, meaning no prior human annotation.

LlamaIndex draws the same line. Retrieval evaluation ranks results with metrics such as MRR, hit rate and precision. Response evaluation includes Correctness, Semantic Similarity, Faithfulness, Context Relevancy, Answer Relevancy and Guideline Adherence. Correctness and Semantic Similarity require labels, so a reference answer.

Three metrics make a useful reading grid:

• Faithfulness compares the response with the retrieved context. No reference answer is needed.
• Context Recall always requires a reference, according to Ragas: it measures whether the retrieved context covers what the correct answer contains.
• Answer Relevancy measures how well the response fits the question, without assessing factual accuracy, the documentation states.

In practice: computing faithfulness

With LlamaIndex

The FaithfulnessEvaluator takes a judge LLM and evaluates a query response. The result exposes three fields: passing (pass or fail), score and feedback. The documentation notes that, depending on the evaluator, only some fields may be filled in.

A second usage pattern is documented: call evaluate with the response and a list of contexts, one source node at a time. This gives one result per source, which helps find which passage fails to support the answer.

With Ragas

In the current Ragas API, you instantiate Faithfulness from ragas.metrics.collections with an LLM, then call ascore (or score in synchronous code) with the user’s question and the response. The older API based on SingleTurnSample is flagged as deprecated in version 0.4 and removed in 1.0. Exact parameters, including how contexts are passed, should be read on the Ragas Faithfulness page for your version.

Ragas also offers a variant, FaithfulnesswithHHEM, which hands claim verification to an open-source classifier from Vectara. It replaces only the verification step, not the extraction of claims.

How to check that it works

1. Take 20 to 30 real questions from your on-call tickets.
2. Compute the faithfulness of each answer and read the five lowest by hand.
3. For each low score, check whether the unsupported claim is a real invention or a passage that retrieval missed.
4. Also read five maximum-score answers: they must be correct, not merely faithful.

The limits of faithfulness

These limits follow from the definition of the metric itself; they are not flaws of one particular tool.

• Faithful is not correct. Faithfulness compares the answer with the retrieved context, never with reality. If the indexed runbook is wrong or out of date, an answer that copies it scores well.
• It ignores what was not retrieved. If the right passage is not returned, a model that says « I don’t know » or repeats a neighbouring passage can remain faithful. Only retrieval metrics (Context Recall, hit rate) reveal it, and Context Recall needs a reference.
• It says nothing about relevance. A faithful answer can dodge the question. Answer Relevancy exists for that case but, per Ragas, it does not judge accuracy either.
• The judge is a model. LlamaIndex’s response evaluators rely on a judging LLM; the documentation mentions a « gold » LLM such as GPT-4. A small local model gives noisier verdicts. This follows from how the method works: no figure is given here.
• Claim splitting affects the score. The score is a ratio of claims, so the number and granularity of claims extracted by the judge change the value.

A cautious reading: treat faithfulness as a drift indicator. A drop between two pipeline versions deserves investigation; an absolute value proves nothing.

Tying evaluation to observability

Offline evaluation does not replace production. Recording the retrieved contexts, similarity scores and the latency of each stage as OpenTelemetry traces lets you replay a doubtful case. The setup is described in the article Langfuse and OpenLLMetry, and broader safeguards in guardrails for AI agents in production. Nothing here assumes a native integration with Dynatrace: OpenTelemetry traces are the meeting point.

Best practices

• Evaluate retrieval and generation separately, then together.
• Keep a versioned question set and enrich it with production failures.
• Pair faithfulness with at least one metric that looks at ground truth (a human reference).
• Date and version the indexed documents: a stale source is invisible to faithfulness.
• Compare two configurations with the same judge; do not change judge midway.
• Read a sample by hand whenever the model or the chunking changes.

Frequently asked questions

What faithfulness score should I aim for?

None of the documentation read gives a universal threshold. Set it from your own question set, reading by hand the answers on either side of the line.

Does faithfulness catch every hallucination?

No. It catches claims that the retrieved context does not support. An error present in the context itself, or an answer that ignores the question, escapes it.

Ragas or LlamaIndex to measure faithfulness?

Both provide an LLM-judged faithfulness metric. LlamaIndex is convenient if the pipeline already uses it; Ragas is a dedicated evaluation framework. The choice mostly depends on your existing tooling.

Conclusion

RAG faithfulness answers a single question: does the answer fit within the retrieved context? It is useful and cheap to set up, but insufficient alone. To evaluate a RAG properly, cross it with retrieval quality, relevance and regular human review.

span.observer is an independent blog, not affiliated with Dynatrace LLC or with the projects mentioned (Ragas, LlamaIndex). Trademarks belong to their respective owners. Tools change quickly: check class names and parameters in the official documentation for your version.

---

Originally published on span.observer. French version: https://span.observer/fr/fidelite-rag-evaluer-limites/.