v1.0.0

Capstone: establish the broken baseline

You inherit a buggy DocDesk and must produce evidence before changing anything.

Estimated study time: 50 minutes

Progress is stored on this device.

Learning outcomes

  • Reproduce three deliberate defects
  • Record a measurable baseline and hypotheses
  • Create a repair plan ranked by impact and risk

The capstone brief

You’re the engineer responsible for DocDesk, a support-policy assistant. A previous team built a prototype. It sometimes returns private documents from the wrong tenant, trusts malformed provider output, reports misleading quality scores, and behaves badly under provider errors. Your task is to improve it with tests and operational evidence.

The project intentionally includes docdesk/student.py, an incomplete implementation, plus docdesk/reference.py as a working reference. The challenges/ tests are your student-facing gates. Do not open the reference implementation until you have tried each challenge; use it as a review artifact, not a source to copy blindly.

Phase 1: inventory

From labs/:

bash
python -m docdesk demo --mode starter
python -m docdesk eval --mode starter
python -m unittest discover -s challenges -v

The final command is expected to fail at first. Failed challenge tests are your work queue, not a broken download. Reference smoke tests should pass before you change anything:

bash
python -m unittest discover -s tests -v

Write a defect report

For each defect include:

  • Reproduction: exact command or input.
  • Expected result: grounded in the product contract.
  • Actual result: show output, not a guess.
  • Security impact: potential leak, misinformation, cost, or availability.
  • Hypothesis: one plausible cause, clearly labeled as a hypothesis.
  • Test: what should become red and then green.

Prioritize risk

Fix tenant isolation before you optimize the ranking score. A retrieval strategy that gets 95% relevance but leaks another customer’s documents is unacceptable. Then fix response validation, refusal/abstention behavior, reliability, quality measurement, and performance.

Deliverable

Commit a BASELINE.md with measured results and three defects. Do not write fictional benchmark values. If the toy fixture is too small for meaningful statistical claims, state that explicitly. Accuracy about limitations is part of the assessment.

Hands-on lab

Run the three commands above, copy observed results into `BASELINE.md`, and rank your fixes. Your baseline must have at least one reproducible security bug, one contract bug, and one quality/operations bug.

Download the lab kit

Knowledge check

What should be fixed before ranking quality?
What is the first step in an inherited broken system?

References