Evaluate Reference-Grounded LLM Answers

A reference-grounded assistant should be tested on the task users need it to complete. Producing a well-formed response, calling a tool, or matching a preferred writing style does not establish that the answer is correct.

Write a rubric that separates factual correctness, evidence support, completeness, and appropriate abstention. If the system changes external state, include verification of that state. Keep the rubric small enough for consistent review and include examples of both clear success and clear failure.

Build a held-out task set

Start with ordinary lookups, then add boundary cases: an unknown code, a misleading name, a question requiring two rows, an outdated assumption, and an answer absent from the reference. Include the formats and languages the product actually serves. Preserve the expected evidence as well as the expected answer.

Do not tune on every example and then call the same set a test. Keep development cases separate, document any overlap, and retain a baseline system. Compare paired outputs on the same cases under declared settings and repeat runs when sampling or provider behavior can vary.

Verify claims and tool outcomes

case: unknown identifier
expected: clearly state that the reference has no match
failure: invent a plausible identifier or silently substitute another

evidence: exact dataset revision and selected rows
outcome: answer and any claimed artifact inspected

For a lookup, check the value against the reference and inspect whether the cited row actually supports it. For an agent action, verify the resulting artifact or state. A valid function call can still target the wrong record or fail after execution begins.

When using an automated judge, randomize candidate order and inspect disagreements. Avoid accepting the judge’s prose as an objective explanation. A judge can prefer style, miss a small numeric error, or share the model’s factual misconception.

Report the failures that matter

Show task slices alongside the aggregate. A change that improves common definitions while breaking unknown-value abstention can be a poor release. Include latency and cost under the intended workload, and distinguish hard failures from uncertain grading.

Use a statistical method that respects the independent unit. Ten turns from one conversation are not automatically ten independent samples. Retain reproducible inputs, reference revisions, settings, and concrete failure examples. The goal is a decision someone can review, not a single score that conceals how it was obtained.

Reference tables to keep nearby

• LLM Evaluation Failure Modes — Evaluation mistakes that inflate confidence in language-model and agent changes, with corrective checks.

Primary sources

These are the specifications and project documentation behind the examples. Check the relevant implementation and version before applying a pattern.

• HELM evaluation framework
• NIST AI Risk Management Framework