v1.0.0

Capstone: add evaluation and observability

Prove the system improved and make the remaining failures diagnosable.

Estimated study time: 65 minutes

Progress is stored on this device.

Learning outcomes

  • Compute retrieval metrics against labeled cases
  • Instrument the pipeline with meaningful spans
  • Write a reproducible regression report

Replace confidence with evidence

Run the same labeled questions through the starter and student retrieval implementations. Record top-k IDs and compute Recall@k and MRR@k. Keep tenant-mismatch and unanswerable questions in separate slices. Do not create a neat overall average by treating an empty gold set as “100% recall.”

Your report format

A useful report includes dataset version, sample size, retrieval configuration, per-slice scores, at least two wrong cases, and a note about whether the results are meaningful beyond the toy fixture.

text
Experiment: tokenized retrieval vs first-word baseline
Dataset: fixtures/eval_cases.json (teaching sample)
k: 3
Measures: Recall@3, MRR@3, tenant violations
Decision: compare and explain, no production claim

Add observability

Instrument answer_request, retrieve, model_complete, and validate_answer. Each span needs timing, parent linkage, and status. Keep private source text and secrets out of default telemetry.

Analyze one intentionally failed request. Determine whether the failure is a provider exception, missing evidence, or contract validation error. You should be able to do this without tracing a hundred unrelated print statements.

Quality is not just retrieval

Review at least five answer outputs manually. Label each answer as supported, unsupported, contradicted, or valid/unnecessary abstention. Discuss disagreements with another reviewer if possible. The fixture is too small to justify population-level quality claims.

Prepare a release decision

Ask: Did deterministic protections improve? Did relevant retrieval improve? What broke? How much confidence do we have? What remains unmeasured? Include the actual data you observed.

Completion criteria

Evaluation and telemetry tests pass; you can produce a trace for a failed request; EVAL_REPORT.md includes at least one limitation and one follow-up experiment.

Hands-on lab

Complete `recall_at_k`, `reciprocal_rank_at_k`, and telemetry. Pass Challenges 03 and 04. Submit `EVAL_REPORT.md` containing baseline comparison, two examples, and a clear decision.

Download the lab kit

Knowledge check

A retrieval score improves. Can you conclude answer accuracy improved?
Why record index and prompt versions?

References