v1.0.0

Design evaluations that answer questions

An eval is a decision tool, not a number to put in a slide.

Estimated study time: 45 minutes

Progress is stored on this device.

Learning outcomes

  • Write an evaluation objective before choosing metrics
  • Create representative labeled examples and slices
  • Distinguish deterministic assertions from subjective judgments

An eval starts with a decision

Imagine changing DocDesk’s retrieval from keyword matching to embeddings. “It sounds better” is not evidence. The decision is: Does the change improve answers to real support questions without increasing unauthorized access, false claims, cost, or latency beyond acceptable limits? Your evaluation must give evidence for that decision.

Separate evaluation levels:

Level Question Typical method
Unit Is the output valid JSON? Deterministic assertions
Retrieval Was relevant evidence returned? Gold document IDs, Recall@k
Answer Does the claim follow from the evidence? Rubric + human review
End-to-end Can a real user solve the task? Scenario tests and feedback
Operations Does it meet cost and latency budgets? Measured requests and metrics

Do not treat a unit test as proof of answer quality, or an answer-quality score as proof of authorization correctness.

Build a labeled dataset

Each example should have a stable ID, tenant, question, expected relevant document IDs, whether it is answerable, a rough topic, and a reason for inclusion. Where relevant, include a reference answer and rubric, but do not pretend every good answer must match the reference wording.

json
{
  "id":"q_refund_20",
  "tenant":"acme",
  "question":"Can I cancel after 20 days?",
  "relevant_doc_ids":["acme-refunds"],
  "answerable":true,
  "slice":"refunds"
}

Our fixture set is a teaching sample, not a statistically representative benchmark. For a real application, use reviewed historical queries (with appropriate privacy controls), domain expert examples, hard negatives, and recent failures. Keep some held-out examples you do not tune against.

Slices reveal hidden regressions

A global average can hide catastrophic regressions. Consider separate slices for exact identifiers, paraphrases, exceptions, multi-hop policy questions, unanswerable questions, new documents, and tenants. If average accuracy rises while answers for the private-tenant slice fail, the release is unacceptable.

Label uncertainty explicitly

Two reviewers may disagree whether a citation is sufficient. Define a short labeling guide and adjudicate disagreements. Do not erase disagreements by averaging them without investigation. Sampling matters: a benchmark made only of easy generated questions will be misleading.

Set release gates by risk

A sensible course rule is: zero known cross-tenant leaks, no deterministic contract regression, and no decline beyond an agreed tolerance on the relevant quality slices. The numeric quality threshold should be chosen from the product’s risk appetite and evidence, not copied from another business.

Hands-on lab

Inspect `labs/fixtures/eval_cases.json`. Add two edge cases and one unanswerable case in your own copy. Document what decision each case supports. Explain why the fixture dataset cannot justify a production accuracy claim.

Download the lab kit

Knowledge check

Which dataset is most likely to be misleading?
What is a good reason to report metrics by slice?

References