# Capstone: reliability, release and technical review

Finish with a defensible engineering demonstration, not a perfect-looking chatbot.

Source: https://bitsandbias.com/course/24-ship/

---

## Learning outcomes

- Bound provider retries and estimate operating costs
- Present release/rollback decisions
- Defend trade-offs and remaining risks with evidence




## Final implementation tasks

Implement the cost estimator with correct per-million-token units and input validation. Implement bounded retry: retry only transient failures, use a total attempt budget, respect cancel/deadline where your runtime supports it, and avoid nested SDK retries. Keep side-effecting operations outside blind automatic retries.

Add a safe no-evidence path. The best fallback for a support-policy assistant without policy evidence is an honest abstention, not the provider's general knowledge.

## Your final demonstration (10 minutes)

**Minute 0–2:** state the product contract and the threats it prevents.

**Minute 2–4:** reproduce the original bug and show the new failing-then-passing test.

**Minute 4–6:** show retrieval eval and one correctly graded, grounded answer.

**Minute 6–8:** show the trace for one successful and one failed request. Explain the cost estimate and deadline policy.

**Minute 8–10:** describe the release gate, rollback plan, and the biggest remaining unknown.

## Scoring rubric (100 points)

| Criterion | Points | Evidence |
|---|---:|---|
| Contract + tenant safety | 25 | Unit/integration tests, failure demo |
| Retrieval + evidence handling | 20 | Ranked results, review examples |
| Evaluation rigor | 20 | Labeled cases, metrics, slices, limitations |
| Observability + debugging | 15 | Correlated trace, safe logs |
| Reliability + cost | 10 | Deadlines, retries, cost calculation |
| Communication + reproducibility | 10 | Clear README, commands, release decision |

**Non-negotiable:** a known cross-tenant exposure or fabricated benchmark claim invalidates a claim of production readiness, even if other points are high.

## Submission package

- Working `student.py` and passing challenge tests.
- `BASELINE.md`, `EVAL_REPORT.md`, and `RELEASE_DECISION.md`.
- At least one new negative test.
- Trace sample with secrets and sensitive text excluded.
- A short walkthrough recording or written technical explanation.

## Reflect like an engineer

What did you initially believe that the tests disproved? What remains unmeasured? Which trade-off would change if latency or privacy requirements became stricter? A strong course does not end with “all good.” It ends with a trustworthy explanation of what you built and where it can fail.

## After the course

Possible extensions: replace lexical retrieval with embeddings; add a real provider behind your adapter; implement HTTP endpoints; export traces to OpenTelemetry; scale the eval set with expert labeling; build a live dashboard. Do these as **independent measured experiments**, not as a pile of fashionable dependencies.



## Hands-on lab

Pass all challenge tests; run `python -m docdesk demo --mode student` and `python -m docdesk eval --mode student`; write your release decision and deliver the ten-minute review using the 100-point rubric.

[Download the lab kit](/course/downloads/docdesk-labs.zip)
