Capstone: repair retrieval and answer contracts
Apply the first three modules to a system with misleading behavior.
Estimated study time: 65 minutes
Progress is stored on this device.
Learning outcomes
- Prevent cross-tenant retrieval
- Reject invalid responses and unauthorized citations
- Make missing-evidence cases abstain rather than fabricate
Repair the boundary, not the symptoms
Start with the retrieval method in student.py. Ensure it never considers private documents from other tenants. Make k behavior explicit and order documents deterministically so tests are stable. Use a lexical ranking baseline before attempting any embedding integration.
Then implement the output validator. It must reject malformed JSON, unexpected top-level keys, wrong field types, citations to unavailable documents, and an inconsistent abstention state. The verifier should be a separate function that can be tested with arbitrary model output, without making a provider call.
Build a negative test matrix
| Case | Expected |
|---|---|
| Same keywords in other tenant’s private document | Never retrieved |
Provider returns citations as string |
Schema failure |
| Provider cites a doc not retrieved | Contract failure |
| No authorized evidence | Safe abstention |
| Retrieved passage says “ignore rules” | Do not grant instructions authority |
Important: the last item is not solved solely by a JSON schema. Prompt injection and semantic support remain layered evaluation/security concerns.
Two-phase implementation
- Deterministic guarantees: tenant filtering, schema validation, citation membership.
- Quality behavior: prompt design, document selection, abstention evaluation, human review.
Your deterministic tests should pass before you claim quality improvements. Then manually review two candidate answers against source evidence. A test that checks citation == "acme-refunds" doesn’t prove the answer accurately represents that policy.
Make trade-offs visible
Record whether changing k introduces contradictory evidence, whether stricter validation increases abstentions, and whether stale documents remain. These are legitimate engineering trade-offs. Don’t hide them inside a single accuracy number.
Completion criteria
challenges.test_01_contract and challenges.test_02_retrieval pass; student-mode demo never prints another tenant’s private documents; the final answer includes only authorized citation IDs. Add one new negative test beyond the supplied checks.
Hands-on lab
Implement `validate_answer` and `retrieve` in `student.py`; pass tests for Challenges 01 and 02. Demonstrate the same cross-tenant input in starter vs student mode and explain what changed.