v1.0.0

Capstone: reliability, release and technical review

Finish with a defensible engineering demonstration, not a perfect-looking chatbot.

Estimated study time: 75 minutes

Progress is stored on this device.

Learning outcomes

  • Bound provider retries and estimate operating costs
  • Present release/rollback decisions
  • Defend trade-offs and remaining risks with evidence

Final implementation tasks

Implement the cost estimator with correct per-million-token units and input validation. Implement bounded retry: retry only transient failures, use a total attempt budget, respect cancel/deadline where your runtime supports it, and avoid nested SDK retries. Keep side-effecting operations outside blind automatic retries.

Add a safe no-evidence path. The best fallback for a support-policy assistant without policy evidence is an honest abstention, not the provider’s general knowledge.

Your final demonstration (10 minutes)

Minute 0–2: state the product contract and the threats it prevents.

Minute 2–4: reproduce the original bug and show the new failing-then-passing test.

Minute 4–6: show retrieval eval and one correctly graded, grounded answer.

Minute 6–8: show the trace for one successful and one failed request. Explain the cost estimate and deadline policy.

Minute 8–10: describe the release gate, rollback plan, and the biggest remaining unknown.

Scoring rubric (100 points)

Criterion Points Evidence
Contract + tenant safety 25 Unit/integration tests, failure demo
Retrieval + evidence handling 20 Ranked results, review examples
Evaluation rigor 20 Labeled cases, metrics, slices, limitations
Observability + debugging 15 Correlated trace, safe logs
Reliability + cost 10 Deadlines, retries, cost calculation
Communication + reproducibility 10 Clear README, commands, release decision

Non-negotiable: a known cross-tenant exposure or fabricated benchmark claim invalidates a claim of production readiness, even if other points are high.

Submission package

  • Working student.py and passing challenge tests.
  • BASELINE.md, EVAL_REPORT.md, and RELEASE_DECISION.md.
  • At least one new negative test.
  • Trace sample with secrets and sensitive text excluded.
  • A short walkthrough recording or written technical explanation.

Reflect like an engineer

What did you initially believe that the tests disproved? What remains unmeasured? Which trade-off would change if latency or privacy requirements became stricter? A strong course does not end with “all good.” It ends with a trustworthy explanation of what you built and where it can fail.

After the course

Possible extensions: replace lexical retrieval with embeddings; add a real provider behind your adapter; implement HTTP endpoints; export traces to OpenTelemetry; scale the eval set with expert labeling; build a live dashboard. Do these as independent measured experiments, not as a pile of fashionable dependencies.

Hands-on lab

Pass all challenge tests; run `python -m docdesk demo --mode student` and `python -m docdesk eval --mode student`; write your release decision and deliver the ten-minute review using the 100-point rubric.

Download the lab kit

Knowledge check

What belongs in the final demo?
If quality scores look good but a tenant leak remains, is the system ready?

References