Evaluate RAG Retrieval Before Generation

If an answer is missing a required fact, first check whether that fact reached the model. Changing the generator prompt cannot reliably repair evidence that the retriever never selected. A retrieval evaluation isolates that stage before the whole pipeline becomes one ambiguous score.

Create a small query set from real tasks. For each query, record the documents or passages that supply the necessary evidence, including cases requiring more than one fact. Keep a held-out subset for comparison after tuning. Document judgment rules so another reviewer can understand why a passage counts as relevant.

Choose metrics matching the evidence need

Precision at k asks how much selected context is relevant. Recall at k asks how much known relevant material was recovered. Mean reciprocal rank rewards an early first useful result. NDCG supports graded relevance and ranking. None of them alone proves that the context contains every fact needed for a complete answer.

For a question requiring a product’s eligibility rule and its exception, a single useful passage may make hit rate look good while still leaving the answer unsupported. Add a context-coverage check for the required facts. Keep the denominator and relevance labeling policy visible.

Inspect the retrieved bundle

query_id, relevant_ids, retrieved_ids, latency_ms
q-17, [policy-main, policy-exception],
      [policy-main, policy-main-copy, unrelated], 42

This example contains a ranking success and a coverage failure. The main policy appears early, but duplicate material consumes space and the exception is absent. Inspect the actual selected text, not just document identifiers: chunk boundaries can cut the critical sentence away from an otherwise relevant document.

Compare retrieval under a fixed context budget and realistic filters. A change that improves recall by returning a much larger bundle may increase cost and distract the generator. Record latency and duplicate context as well as relevance.

Keep generation evaluation separate

Once the evidence is adequate, evaluate whether the answer uses it correctly, cites it accurately, and abstains when the query is unsupported. Test unavailable evidence and conflicting passages deliberately. A relevant retrieval result can still produce a wrong or overconfident answer.

Report slices by query type, language, document age, and multi-document complexity when those dimensions matter. Small slices need uncertainty, and incomplete relevance judgments limit what recall means. Preserve a few concrete failure examples with each comparison so a better average does not hide an important regression.

Reference tables to keep nearby

• Retrieval Evaluation Metrics for RAG — Retrieval metrics and diagnostic slices for evaluating the evidence selected before language-model generation.

Primary sources

These are the specifications and project documentation behind the examples. Check the relevant implementation and version before applying a pattern.

• Stanford introduction to information retrieval
• Stanford ranked retrieval evaluation