Capstone: add evaluation and observability
Prove the system improved and make the remaining failures diagnosable.
Estimated study time: 65 minutes
Progress is stored on this device.
Learning outcomes
- Compute retrieval metrics against labeled cases
- Instrument the pipeline with meaningful spans
- Write a reproducible regression report
Replace confidence with evidence
Run the same labeled questions through the starter and student retrieval implementations. Record top-k IDs and compute Recall@k and MRR@k. Keep tenant-mismatch and unanswerable questions in separate slices. Do not create a neat overall average by treating an empty gold set as “100% recall.”
Your report format
A useful report includes dataset version, sample size, retrieval configuration, per-slice scores, at least two wrong cases, and a note about whether the results are meaningful beyond the toy fixture.
Experiment: tokenized retrieval vs first-word baseline
Dataset: fixtures/eval_cases.json (teaching sample)
k: 3
Measures: Recall@3, MRR@3, tenant violations
Decision: compare and explain, no production claimAdd observability
Instrument answer_request, retrieve, model_complete, and validate_answer. Each span needs timing, parent linkage, and status. Keep private source text and secrets out of default telemetry.
Analyze one intentionally failed request. Determine whether the failure is a provider exception, missing evidence, or contract validation error. You should be able to do this without tracing a hundred unrelated print statements.
Quality is not just retrieval
Review at least five answer outputs manually. Label each answer as supported, unsupported, contradicted, or valid/unnecessary abstention. Discuss disagreements with another reviewer if possible. The fixture is too small to justify population-level quality claims.
Prepare a release decision
Ask: Did deterministic protections improve? Did relevant retrieval improve? What broke? How much confidence do we have? What remains unmeasured? Include the actual data you observed.
Completion criteria
Evaluation and telemetry tests pass; you can produce a trace for a failed request; EVAL_REPORT.md includes at least one limitation and one follow-up experiment.
Hands-on lab
Complete `recall_at_k`, `reciprocal_rank_at_k`, and telemetry. Pass Challenges 03 and 04. Submit `EVAL_REPORT.md` containing baseline comparison, two examples, and a clear decision.