Retrieval Evaluation Metrics for RAG
Retrieval metrics and diagnostic slices for evaluating the evidence selected before language-model generation.
Retrieval metrics and diagnostic slices for evaluating the evidence selected before language-model generation. Each row includes a practical application and a failure to check before using it.
Use it when
- Compare retrievers using a fixed judged query set.
- Find why an answer lacks evidence before changing the generator.
Watch out for
- Retrieval quality is separate from answer correctness.
- Keep test queries out of retriever tuning and report uncertainty.
The data
10 rows
| Metric | Definition | Use | Limitation |
|---|---|---|---|
| Precision at k | Relevant retrieved items divided by k. | Measure how much top-ranked context is useful. | It depends on relevance judgments and the chosen k. |
| Recall at k | Relevant retrieved items divided by all known relevant items. | Measure how much required evidence is recovered. | Incomplete relevance labels distort the denominator. |
| Hit rate at k | Fraction of queries with at least one relevant top-k item. | Check whether the retriever finds any useful evidence. | One relevant item can still be insufficient to answer. |
| Mean reciprocal rank | Mean of the reciprocal rank of the first relevant result. | Reward finding useful evidence early. | It ignores relevant results after the first one. |
| NDCG at k | Discounted graded relevance normalized by the ideal ranking. | Compare rankings with multiple relevance grades. | Judgment policy and gain function affect scores. |
| Context coverage | Fraction of required answer facts supported by retrieved context. | Find multi-document evidence gaps. | It requires an explicit set of required facts. |
| Duplicate rate | Fraction of retrieved context redundant with other selected items. | Detect wasted context capacity. | Overlapping chunks can be useful in some tasks. |
| Retrieval latency | Elapsed time for retrieval under a declared load. | Balance evidence quality against responsiveness. | Cache state and concurrency change the distribution. |
| Empty-result rate | Fraction of queries returning no eligible evidence. | Track abstention and indexing problems. | An empty result can be the correct behavior. |
| Slice performance | A metric calculated by query or content subgroup. | Expose failures hidden by an overall average. | Small slices need uncertainty reporting. |
Fields
The shape of this dataset, read from the data itself: every column, its type and a real value. Agents can use this to plan a query without fetching the whole document.
| Section | Field | Type | Example |
|---|---|---|---|
| data | metric | string | Precision at k |
| data | definition | string | Relevant retrieved items divided by k. |
| data | use | string | Measure how much top-ranked context is useful. |
| data | limitation | string | It depends on relevance judgments and the chose… |
Fetch the same bytes
The static files contain the same reference rows as the API, with no API rate limit. Live metadata, such as view counts, may differ. Use the API for content negotiation; use the static files for a cacheable snapshot.
curl "https://yjtoon.com/api/dataset/rag-retrieval-metrics?format=toon"
const res = await fetch( "https://yjtoon.com/static-data/dataset/rag-retrieval-metrics.toon" ); const toon = await res.text();
Rate limit: 120 requests per minute per IP, no key and no signup. API reference →
Sources and editorial notes
Original editorial reference notes by YJTOON, informed by the linked primary documentation. External specifications retain their own licenses.
Editorial review: . Source documents keep their own licenses.