Capstone: reliability, release and technical review
Finish with a defensible engineering demonstration, not a perfect-looking chatbot.
Estimated study time: 75 minutes
Progress is stored on this device.
Learning outcomes
- Bound provider retries and estimate operating costs
- Present release/rollback decisions
- Defend trade-offs and remaining risks with evidence
Final implementation tasks
Implement the cost estimator with correct per-million-token units and input validation. Implement bounded retry: retry only transient failures, use a total attempt budget, respect cancel/deadline where your runtime supports it, and avoid nested SDK retries. Keep side-effecting operations outside blind automatic retries.
Add a safe no-evidence path. The best fallback for a support-policy assistant without policy evidence is an honest abstention, not the provider’s general knowledge.
Your final demonstration (10 minutes)
Minute 0–2: state the product contract and the threats it prevents.
Minute 2–4: reproduce the original bug and show the new failing-then-passing test.
Minute 4–6: show retrieval eval and one correctly graded, grounded answer.
Minute 6–8: show the trace for one successful and one failed request. Explain the cost estimate and deadline policy.
Minute 8–10: describe the release gate, rollback plan, and the biggest remaining unknown.
Scoring rubric (100 points)
| Criterion | Points | Evidence |
|---|---|---|
| Contract + tenant safety | 25 | Unit/integration tests, failure demo |
| Retrieval + evidence handling | 20 | Ranked results, review examples |
| Evaluation rigor | 20 | Labeled cases, metrics, slices, limitations |
| Observability + debugging | 15 | Correlated trace, safe logs |
| Reliability + cost | 10 | Deadlines, retries, cost calculation |
| Communication + reproducibility | 10 | Clear README, commands, release decision |
Non-negotiable: a known cross-tenant exposure or fabricated benchmark claim invalidates a claim of production readiness, even if other points are high.
Submission package
- Working
student.pyand passing challenge tests. BASELINE.md,EVAL_REPORT.md, andRELEASE_DECISION.md.- At least one new negative test.
- Trace sample with secrets and sensitive text excluded.
- A short walkthrough recording or written technical explanation.
Reflect like an engineer
What did you initially believe that the tests disproved? What remains unmeasured? Which trade-off would change if latency or privacy requirements became stricter? A strong course does not end with “all good.” It ends with a trustworthy explanation of what you built and where it can fail.
After the course
Possible extensions: replace lexical retrieval with embeddings; add a real provider behind your adapter; implement HTTP endpoints; export traces to OpenTelemetry; scale the eval set with expert labeling; build a live dashboard. Do these as independent measured experiments, not as a pile of fashionable dependencies.
Hands-on lab
Pass all challenge tests; run `python -m docdesk demo --mode student` and `python -m docdesk eval --mode student`; write your release decision and deliver the ten-minute review using the 100-point rubric.