RAG Evaluation Frameworks

The RAG Triad Architecture

                                USER QUERY
                                    │
         ┌──────────────────────────┴──────────────────────────┐
         ▼                                                     ▼
 [ CONTEXT PRECISION ]                                [ ANSWER RELEVANCE ]
 Is retrieved context                                 Does generated answer
 relevant to query?                                   directly address query?
         │                                                     │
         ▼                                                     ▼
Retrieved Context Chunks ──────► [ FAITHFULNESS ] ──────► Generated LLM Answer
                                 Is answer 100%
                                 grounded in context?

Deep-Dive: The 3 Core RAG Metrics

1. Context Precision (Retrieval Metric)

Evaluates whether top-ranked retrieved chunks are relevant to user query $q$:

$$\text{Context Precision@K} = \frac{\sum_{k=1}^K \text{Precision}@k \cdot v_k}{\text{Total Relevant Chunks in Top } K}$$

• $v_k \in {0, 1}$: Binary relevance of chunk at rank $k$.
• High score $\implies$ Retriever places signal at the top, avoiding noisy irrelevant chunks.

2. Faithfulness / Groundedness (Generation Metric)

Prevents Hallucinations! Evaluates if claims in answer $a$ are supported by context $c$:

$$\text{Faithfulness} = \frac{\text{Number of Claims in } a \text{ Supported by } c}{\text{Total Number of Claims Extracted from } a}$$

1. LLM breaks response $a$ into discrete atomic statements $S = {s_1, s_2, \dots, s_n}$.
2. LLM verifies if each statement $s_i$ is directly supported by retrieved context $c$.

3. Answer Relevance (Generation Metric)

Evaluates if answer $a$ directly addresses query $q$, ignoring context grounding:

1. Prompt an LLM to generate $N=3$ artificial questions $q_i^*$ based on generated answer $a$.
2. Compute average embedding cosine similarity:

$$\text{Answer Relevance} = \frac{1}{N} \sum_{i=1}^N \cos(\mathbf{e}q, \mathbf{e}{q_i^*})$$

Component Breakdown Diagnostics

  Context Precision    Faithfulness    Answer Relevance    Diagnosis / Fix Required
──────────────────────────────────────────────────────────────────────────────────────────────────
       LOW                 HIGH              HIGH          Retriever is noisy! (Tune Chunking / BM25 Hybrid)
       HIGH                LOW               HIGH          LLM is Hallucinating! (Lower Temp / Add Strict System Rules)
       HIGH                HIGH              LOW           LLM is Off-Topic! (Improve Prompt Instructions)

Say this out loud

"RAGAS evaluates RAG pipelines across the RAG Triad: Context Precision measures retriever signal-to-noise ratio, Faithfulness verifies that 100% of claims are grounded in retrieved context to prevent hallucinations, and Answer Relevance measures if the response directly addresses the query. Breaking down metrics isolates whether failures stem from retrieval noise or LLM generation."

Follow-ups to expect

• What is Context Recall in RAGAS? Measures whether all ground-truth facts needed to answer the question were successfully retrieved in the context chunks: $\text{Context Recall} = \frac{|\text{Retrieved Ground-Truth Facts}|}{|\text{Total Ground-Truth Facts}|}$.
• How do you run RAGAS continuous evaluation in CI/CD pipelines? Store a 100-example Golden Dataset. On every RAG pipeline code change (e.g. updating chunk size or embedding model), trigger automated RAGAS scripts to assert Faithfulness > 0.90 and Context Precision > 0.85.