# Capstone: repair retrieval and answer contracts

Apply the first three modules to a system with misleading behavior.

Source: https://bitsandbias.com/course/22-repair/

---

## Learning outcomes

- Prevent cross-tenant retrieval
- Reject invalid responses and unauthorized citations
- Make missing-evidence cases abstain rather than fabricate




## Repair the boundary, not the symptoms

Start with the retrieval method in `student.py`. Ensure it never considers private documents from other tenants. Make `k` behavior explicit and order documents deterministically so tests are stable. Use a lexical ranking baseline before attempting any embedding integration.

Then implement the output validator. It must reject malformed JSON, unexpected top-level keys, wrong field types, citations to unavailable documents, and an inconsistent abstention state. The verifier should be a separate function that can be tested with arbitrary model output, without making a provider call.

## Build a negative test matrix

| Case | Expected |
|---|---|
| Same keywords in other tenant's private document | Never retrieved |
| Provider returns `citations` as string | Schema failure |
| Provider cites a doc not retrieved | Contract failure |
| No authorized evidence | Safe abstention |
| Retrieved passage says “ignore rules” | Do not grant instructions authority |

**Important:** the last item is not solved solely by a JSON schema. Prompt injection and semantic support remain layered evaluation/security concerns.

## Two-phase implementation

1. **Deterministic guarantees:** tenant filtering, schema validation, citation membership.
2. **Quality behavior:** prompt design, document selection, abstention evaluation, human review.

Your deterministic tests should pass before you claim quality improvements. Then manually review two candidate answers against source evidence. A test that checks `citation == "acme-refunds"` doesn't prove the answer accurately represents that policy.

## Make trade-offs visible

Record whether changing `k` introduces contradictory evidence, whether stricter validation increases abstentions, and whether stale documents remain. These are legitimate engineering trade-offs. Don't hide them inside a single accuracy number.

## Completion criteria

`challenges.test_01_contract` and `challenges.test_02_retrieval` pass; student-mode demo never prints another tenant's private documents; the final answer includes only authorized citation IDs. Add one new negative test beyond the supplied checks.



## Hands-on lab

Implement `validate_answer` and `retrieve` in `student.py`; pass tests for Challenges 01 and 02. Demonstrate the same cross-tenant input in starter vs student mode and explain what changed.

[Download the lab kit](/course/downloads/docdesk-labs.zip)
