Turn evaluations into release gates
An eval only becomes engineering when it changes release decisions.
Estimated study time: 45 minutes
Progress is stored on this device.
Learning outcomes
- Define a CI-style regression workflow
- Create a failure report with baseline comparisons
- Design a release gate that includes safety, quality and operating costs
The evaluation lifecycle
A reproducible experiment records a dataset version, prompt version, retrieval configuration, model identifier, code revision, and scoring rubric. Otherwise a result like “quality improved 5%” is impossible to reproduce.
Store an eval report containing:
{
"dataset_version": "2026-11-v1",
"system_revision": "commit-sha",
"metric": "recall_at_3",
"baseline": 0.76,
"candidate": 0.81,
"slice": "refund_questions"
}The numbers above are illustrative. Never write them as if they came from a real evaluation run.
A useful release gate
Think in layers:
Gate 1: Hard invariants. No cross-tenant exposure, no invalid contracts, no unauthorized citations. A known violation blocks release.
Gate 2: Quality tolerance. Compare per-slice metrics to an agreed baseline with enough sample size to make the comparison meaningful. Treat tiny changes on tiny datasets as inconclusive.
Gate 3: Operating constraints. Confirm deadlines, error rates, and cost budgets. A higher quality score is not a free pass for doubling latency and exceeding budget.
Gate 4: Manual review. Review regressions, new types of failures, and high-risk scenarios.
CI is not magic
A deterministic eval fixture can run on every commit. Live-model evaluations are variable, sometimes costly, and may need a scheduled job or a guarded environment. Avoid making tests flaky by asserting an exact wording from a stochastic model. Instead, keep deterministic contract tests in CI and run sampled, repeated, reviewed quality tests on a controlled cadence.
Experiment hygiene
Change one factor at a time. Use the same labeled query set for baseline and candidate, and reserve a separate holdout to detect overfitting. Keep a changelog of failures; adding a regression case is often more valuable than endlessly tweaking prompts.
An action-oriented report
Don’t write: “Recall got worse.” Write: “Recall@3 on exact identifiers fell from 0.92 to 0.67 across 30 reviewed queries after switching to semantic-only retrieval. Six failures involve error codes. Restore lexical matching for exact identifiers and rerun held-out cases.” This is the difference between reporting and engineering.
Course release rehearsal
Deliberately change your retriever so it returns only one result. Compare against the earlier baseline and note which slice degrades. Write a short decision: ship, revert, or investigate. Your answer must cite actual measured cases, not instincts.
Hands-on lab
Run `python -m docdesk eval` to generate the reference report. Change `k` using the CLI flag, rerun, and write a `docs/release-decision.md` with baseline, candidate, failure examples, and your decision. Include at least one result you are uncertain about.