# Capstone: add evaluation and observability

Prove the system improved and make the remaining failures diagnosable.

Source: https://bitsandbias.com/course/23-measure/

---

## Learning outcomes

- Compute retrieval metrics against labeled cases
- Instrument the pipeline with meaningful spans
- Write a reproducible regression report




## Replace confidence with evidence

Run the same labeled questions through the starter and student retrieval implementations. Record top-k IDs and compute Recall@k and MRR@k. Keep tenant-mismatch and unanswerable questions in separate slices. Do not create a neat overall average by treating an empty gold set as “100% recall.”

## Your report format

A useful report includes dataset version, sample size, retrieval configuration, per-slice scores, at least two wrong cases, and a note about whether the results are meaningful beyond the toy fixture.


```text
Experiment: tokenized retrieval vs first-word baseline
Dataset: fixtures/eval_cases.json (teaching sample)
k: 3
Measures: Recall@3, MRR@3, tenant violations
Decision: compare and explain, no production claim
```

## Add observability

Instrument `answer_request`, `retrieve`, `model_complete`, and `validate_answer`. Each span needs timing, parent linkage, and status. Keep private source text and secrets out of default telemetry.

Analyze one intentionally failed request. Determine whether the failure is a provider exception, missing evidence, or contract validation error. You should be able to do this without tracing a hundred unrelated print statements.

## Quality is not just retrieval

Review at least five answer outputs manually. Label each answer as `supported`, `unsupported`, `contradicted`, or valid/unnecessary abstention. Discuss disagreements with another reviewer if possible. The fixture is too small to justify population-level quality claims.

## Prepare a release decision

Ask: Did deterministic protections improve? Did relevant retrieval improve? What broke? How much confidence do we have? What remains unmeasured? Include the actual data you observed.

## Completion criteria

Evaluation and telemetry tests pass; you can produce a trace for a failed request; `EVAL_REPORT.md` includes at least one limitation and one follow-up experiment.



## Hands-on lab

Complete `recall_at_k`, `reciprocal_rank_at_k`, and telemetry. Pass Challenges 03 and 04. Submit `EVAL_REPORT.md` containing baseline comparison, two examples, and a clear decision.

[Download the lab kit](/course/downloads/docdesk-labs.zip)
