Capstone: establish the broken baseline
You inherit a buggy DocDesk and must produce evidence before changing anything.
Estimated study time: 50 minutes
Progress is stored on this device.
Learning outcomes
- Reproduce three deliberate defects
- Record a measurable baseline and hypotheses
- Create a repair plan ranked by impact and risk
The capstone brief
You’re the engineer responsible for DocDesk, a support-policy assistant. A previous team built a prototype. It sometimes returns private documents from the wrong tenant, trusts malformed provider output, reports misleading quality scores, and behaves badly under provider errors. Your task is to improve it with tests and operational evidence.
The project intentionally includes docdesk/student.py, an incomplete implementation, plus docdesk/reference.py as a working reference. The challenges/ tests are your student-facing gates. Do not open the reference implementation until you have tried each challenge; use it as a review artifact, not a source to copy blindly.
Phase 1: inventory
From labs/:
python -m docdesk demo --mode starter
python -m docdesk eval --mode starter
python -m unittest discover -s challenges -vThe final command is expected to fail at first. Failed challenge tests are your work queue, not a broken download. Reference smoke tests should pass before you change anything:
python -m unittest discover -s tests -vWrite a defect report
For each defect include:
- Reproduction: exact command or input.
- Expected result: grounded in the product contract.
- Actual result: show output, not a guess.
- Security impact: potential leak, misinformation, cost, or availability.
- Hypothesis: one plausible cause, clearly labeled as a hypothesis.
- Test: what should become red and then green.
Prioritize risk
Fix tenant isolation before you optimize the ranking score. A retrieval strategy that gets 95% relevance but leaks another customer’s documents is unacceptable. Then fix response validation, refusal/abstention behavior, reliability, quality measurement, and performance.
Deliverable
Commit a BASELINE.md with measured results and three defects. Do not write fictional benchmark values. If the toy fixture is too small for meaningful statistical claims, state that explicitly. Accuracy about limitations is part of the assessment.
Hands-on lab
Run the three commands above, copy observed results into `BASELINE.md`, and rank your fixes. Your baseline must have at least one reproducible security bug, one contract bug, and one quality/operations bug.