# Design evaluations that answer questions

An eval is a decision tool, not a number to put in a slide.

Source: https://bitsandbias.com/course/09-eval-design/

---

## Learning outcomes

- Write an evaluation objective before choosing metrics
- Create representative labeled examples and slices
- Distinguish deterministic assertions from subjective judgments




## An eval starts with a decision

Imagine changing DocDesk's retrieval from keyword matching to embeddings. “It sounds better” is not evidence. The decision is: **Does the change improve answers to real support questions without increasing unauthorized access, false claims, cost, or latency beyond acceptable limits?** Your evaluation must give evidence for that decision.

Separate evaluation levels:

| Level | Question | Typical method |
|---|---|---|
| Unit | Is the output valid JSON? | Deterministic assertions |
| Retrieval | Was relevant evidence returned? | Gold document IDs, Recall@k |
| Answer | Does the claim follow from the evidence? | Rubric + human review |
| End-to-end | Can a real user solve the task? | Scenario tests and feedback |
| Operations | Does it meet cost and latency budgets? | Measured requests and metrics |

Do not treat a unit test as proof of answer quality, or an answer-quality score as proof of authorization correctness.

## Build a labeled dataset

Each example should have a stable ID, tenant, question, expected relevant document IDs, whether it is answerable, a rough topic, and a reason for inclusion. Where relevant, include a reference answer and rubric, but do not pretend every good answer must match the reference wording.


```json
{
  "id":"q_refund_20",
  "tenant":"acme",
  "question":"Can I cancel after 20 days?",
  "relevant_doc_ids":["acme-refunds"],
  "answerable":true,
  "slice":"refunds"
}
```

Our fixture set is a teaching sample, not a statistically representative benchmark. For a real application, use reviewed historical queries (with appropriate privacy controls), domain expert examples, hard negatives, and recent failures. Keep some **held-out** examples you do not tune against.

## Slices reveal hidden regressions

A global average can hide catastrophic regressions. Consider separate slices for exact identifiers, paraphrases, exceptions, multi-hop policy questions, unanswerable questions, new documents, and tenants. If average accuracy rises while answers for the private-tenant slice fail, the release is unacceptable.

## Label uncertainty explicitly

Two reviewers may disagree whether a citation is sufficient. Define a short labeling guide and adjudicate disagreements. Do not erase disagreements by averaging them without investigation. Sampling matters: a benchmark made only of easy generated questions will be misleading.

## Set release gates by risk

A sensible course rule is: **zero known cross-tenant leaks**, no deterministic contract regression, and no decline beyond an agreed tolerance on the relevant quality slices. The numeric quality threshold should be chosen from the product's risk appetite and evidence, not copied from another business.



## Hands-on lab

Inspect `labs/fixtures/eval_cases.json`. Add two edge cases and one unanswerable case in your own copy. Document what decision each case supports. Explain why the fixture dataset cannot justify a production accuracy claim.

[Download the lab kit](/course/downloads/docdesk-labs.zip)
