Evaluating a RAG Pipeline

Disentangling RAG Evaluation

When a user complains that a RAG pipeline gave a bad answer, where did the system fail?

1. Did Retrieval Fail? Vector search fetched irrelevant, noisy document chunks.
2. Did Generation Fail? Vector search fetched perfect chunks, but the LLM hallucinated or ignored the context.

To fix RAG systems, you must evaluate Retrieval and Generation separately.

The RAG Triad Framework (Ragas / TruLens) provides automated metrics for both components:

                              THE RAG TRIAD
                               User Query
                                ╱     ╲
            Context Relevance  ╱       ╲  Answer Relevance
                              ▼         ▼
                      Retrieved Context ──► Generated Answer
                              Groundedness / Faithfulness

The RAG Triad Metrics

┌──────────────────────────┬──────────────────────────┬──────────────────────────┐
│ 1. CONTEXT RELEVANCE     │ 2. GROUNDEDNESS          │ 3. ANSWER RELEVANCE      │
├──────────────────────────┼──────────────────────────┼──────────────────────────┤
│ Evaluates RETRIEVAL.     │ Evaluates GENERATION.    │ Evaluates ALIGNMENT.     │
│ "Are the retrieved       │ "Is every claim in the   │ "Does the generated      │
│ chunks relevant to the   │ answer supported by the  │ response directly answer │
│ user query?"             │ retrieved context?"      │ the user question?"      │
└──────────────────────────┴──────────────────────────┴──────────────────────────┘

1. Context Relevance (Retrieval Score)

Measures the proportion of sentences in the retrieved context chunks that are directly relevant to the user query:

$$\text{Context Relevance} = \frac{|\text{Sentences in Chunks Relevant to Query}|}{|\text{Total Sentences in Chunks}|}$$

• Low Score: Vector search is fetching noisy, irrelevant text blocks.
• Fix: Adjust chunk size, switch embedding models, or add a Cross-Encoder Reranker.

2. Groundedness / Faithfulness (Hallucination Score)

Measures whether every factual statement in the LLM answer can be inferred directly from the retrieved context:

$$\text{Groundedness} = \frac{|\text{Claims in Answer Supported by Context}|}{|\text{Total Claims in Answer}|}$$

• Low Score: The LLM is hallucinating or relying on pretraining memory instead of sticking to retrieved facts.
• Fix: Add negative constraints to system prompt ("Answer using ONLY retrieved context. If context is insufficient, state I do not know.").

3. Answer Relevance (Intent Alignment)

Measures whether the generated response directly addresses the user query:

$$\text{Answer Relevance} = \text{CosineSim}\left( \text{Embed}(\text{User Query}), ; \text{Mean}(\text{Embed}(\text{Generated Questions})) \right)$$

• Low Score: The LLM went off topic or provided incomplete answers.
• Fix: Improve prompt instructions or fine-tune response formatting.

Evaluating RAG in Production (Ragas / TruLens)

Automated RAG evaluation tools (Ragas, TruLens, DeepEval) use an LLM-as-a-Judge (GPT-4) to compute RAG Triad scores continuously on real production telemetry logs:

from ragas import evaluate
from ragas.metrics import context_relevance, faithfulness, answer_relevancy

results = evaluate(
    dataset=rag_telemetry_dataset,
    metrics=[context_relevance, faithfulness, answer_relevancy]
)

print(results)
# Output: {'context_relevance': 0.88, 'faithfulness': 0.96, 'answer_relevancy': 0.92}

Say this out loud

Evaluating RAG requires separating Retrieval from Generation using the RAG Triad framework. Context Relevance evaluates if vector search retrieved relevant chunks. Groundedness checks if LLM answer claims are supported by context to catch hallucinations. Answer Relevance checks if the output directly answers the user question.

Followups to expect

1. What is Noise Sensitivity in RAG? Evaluating how gracefully an LLM handles irrelevant background sentences inside retrieved context chunks without being distracted into wrong answers.
2. What is Negative Rejection in RAG? Testing whether the RAG pipeline correctly outputs "I cannot answer based on provided context" when vector search fails to retrieve relevant document chunks.