api: YAML JSON TOON Database
version: 1.0.0
format: json
dataset:
  id: 1018
  slug: llm-evaluation-failure-modes
  title: LLM Evaluation Failure Modes
  description: "Evaluation mistakes that inflate confidence in language-model and agent changes, with corrective checks."
  summary: "Evaluation mistakes that inflate confidence in language-model and agent changes, with corrective checks. Each row includes a practical application and a failure to check before using it."
  use_when[2]: Review a model or prompt change before shipping.,Build a held-out test for answers grounded in reference data.
  caveats[2]: "This is an evaluation checklist, not a benchmark leaderboard.",The correct test depends on the actual user task and consequences of failure.
  related[2]: rag-retrieval-metrics,ai-ethics-interpretability
  category: AI & ML Terms
  category_slug: ai-ml-terms
  tags: "ai-ml-terms,llm evaluation failure modes,reference,engineering"
  view_count: 0
  created_at: 1791798640
  updated_at: 1791798640
  reviewed_at: 2026-10-12
  sources[2]{title,url}:
    HELM evaluation framework,"https://crfm.stanford.edu/helm/"
    NIST AI Risk Management Framework,"https://www.nist.gov/itl/ai-risk-management-framework"
  reading[1]: llm-evaluation-checklist-for-reference-grounded-answers
  provenance: "Original editorial reference notes by YJTOON, informed by the linked primary documentation. External specifications retain their own licenses."
  entry_count: 10
  fields[4]{section,field,type,example}:
    data,failure,string,Test contamination
    data,symptom,string,Strong scores on examples used during developme…
    data,better_check,string,Hold out cases and document dataset provenance.
    data,remaining_limit,string,Unknown pretraining overlap can remain.
data[10]{failure,symptom,better_check,remaining_limit}:
  Test contamination,Strong scores on examples used during development.,Hold out cases and document dataset provenance.,Unknown pretraining overlap can remain.
  Prompt overfitting,One prompt works on a small familiar set.,Use varied held-out cases and a baseline.,A larger sample can still miss rare failures.
  Judge position bias,The judge favors the first or second candidate.,Randomize candidate order and audit disagreements.,Other judge biases can remain.
  Single aggregate score,A gain hides a serious subgroup regression.,Report task slices and important failure categories.,Tiny slices have wide uncertainty.
  Uncontrolled sampling,Results move after an unrelated change.,Fix settings and compare repeated paired runs.,Provider-side nondeterminism may remain.
  Missing abstention test,The system answers even when evidence is absent.,Include unanswerable and adversarial cases.,A good abstention rate does not prove answer quality.
  Tool success proxy,A valid tool call is counted as a solved task.,Verify the intended state or external artifact.,Some external outcomes are hard to observe.
  Latency blind spot,A quality gain makes the workflow unusably slow.,Measure elapsed time and tail behavior under load.,Lab traffic may differ from production.
  Invalid confidence interval,Correlated turns are treated as independent samples.,Resample or analyze at the appropriate independent unit.,The unit must reflect the data collection process.
  No failure review,The dashboard improves while users report regressions.,Inspect concrete failures with a reproducible rubric.,Human review also needs consistency checks.