v1.0.0

Capstone: repair retrieval and answer contracts

Apply the first three modules to a system with misleading behavior.

Estimated study time: 65 minutes

Progress is stored on this device.

Learning outcomes

  • Prevent cross-tenant retrieval
  • Reject invalid responses and unauthorized citations
  • Make missing-evidence cases abstain rather than fabricate

Repair the boundary, not the symptoms

Start with the retrieval method in student.py. Ensure it never considers private documents from other tenants. Make k behavior explicit and order documents deterministically so tests are stable. Use a lexical ranking baseline before attempting any embedding integration.

Then implement the output validator. It must reject malformed JSON, unexpected top-level keys, wrong field types, citations to unavailable documents, and an inconsistent abstention state. The verifier should be a separate function that can be tested with arbitrary model output, without making a provider call.

Build a negative test matrix

Case Expected
Same keywords in other tenant’s private document Never retrieved
Provider returns citations as string Schema failure
Provider cites a doc not retrieved Contract failure
No authorized evidence Safe abstention
Retrieved passage says “ignore rules” Do not grant instructions authority

Important: the last item is not solved solely by a JSON schema. Prompt injection and semantic support remain layered evaluation/security concerns.

Two-phase implementation

  1. Deterministic guarantees: tenant filtering, schema validation, citation membership.
  2. Quality behavior: prompt design, document selection, abstention evaluation, human review.

Your deterministic tests should pass before you claim quality improvements. Then manually review two candidate answers against source evidence. A test that checks citation == "acme-refunds" doesn’t prove the answer accurately represents that policy.

Make trade-offs visible

Record whether changing k introduces contradictory evidence, whether stricter validation increases abstentions, and whether stale documents remain. These are legitimate engineering trade-offs. Don’t hide them inside a single accuracy number.

Completion criteria

challenges.test_01_contract and challenges.test_02_retrieval pass; student-mode demo never prints another tenant’s private documents; the final answer includes only authorized citation IDs. Add one new negative test beyond the supplied checks.

Hands-on lab

Implement `validate_answer` and `retrieve` in `student.py`; pass tests for Challenges 01 and 02. Demonstrate the same cross-tenant input in starter vs student mode and explain what changed.

Download the lab kit

Knowledge check

Can the prompt alone enforce tenant isolation?
Why is a separate validator useful?

References