# Turn evaluations into release gates

An eval only becomes engineering when it changes release decisions.

Source: https://bitsandbias.com/course/12-regressions/

---

## Learning outcomes

- Define a CI-style regression workflow
- Create a failure report with baseline comparisons
- Design a release gate that includes safety, quality and operating costs




## The evaluation lifecycle

A reproducible experiment records a dataset version, prompt version, retrieval configuration, model identifier, code revision, and scoring rubric. Otherwise a result like “quality improved 5%” is impossible to reproduce.

Store an eval report containing:


```json
{
  "dataset_version": "2026-11-v1",
  "system_revision": "commit-sha",
  "metric": "recall_at_3",
  "baseline": 0.76,
  "candidate": 0.81,
  "slice": "refund_questions"
}
```

The numbers above are illustrative. Never write them as if they came from a real evaluation run.

## A useful release gate

Think in layers:

**Gate 1: Hard invariants.** No cross-tenant exposure, no invalid contracts, no unauthorized citations. A known violation blocks release.

**Gate 2: Quality tolerance.** Compare per-slice metrics to an agreed baseline with enough sample size to make the comparison meaningful. Treat tiny changes on tiny datasets as inconclusive.

**Gate 3: Operating constraints.** Confirm deadlines, error rates, and cost budgets. A higher quality score is not a free pass for doubling latency and exceeding budget.

**Gate 4: Manual review.** Review regressions, new types of failures, and high-risk scenarios.

## CI is not magic

A deterministic eval fixture can run on every commit. Live-model evaluations are variable, sometimes costly, and may need a scheduled job or a guarded environment. Avoid making tests flaky by asserting an exact wording from a stochastic model. Instead, keep deterministic contract tests in CI and run sampled, repeated, reviewed quality tests on a controlled cadence.

## Experiment hygiene

Change one factor at a time. Use the same labeled query set for baseline and candidate, and reserve a separate holdout to detect overfitting. Keep a changelog of failures; adding a regression case is often more valuable than endlessly tweaking prompts.

## An action-oriented report

Don't write: “Recall got worse.” Write: “Recall@3 on exact identifiers fell from 0.92 to 0.67 across 30 reviewed queries after switching to semantic-only retrieval. Six failures involve error codes. Restore lexical matching for exact identifiers and rerun held-out cases.” This is the difference between reporting and engineering.

## Course release rehearsal

Deliberately change your retriever so it returns only one result. Compare against the earlier baseline and note which slice degrades. Write a short decision: ship, revert, or investigate. Your answer must cite actual measured cases, not instincts.



## Hands-on lab

Run `python -m docdesk eval` to generate the reference report. Change `k` using the CLI flag, rerun, and write a `docs/release-decision.md` with baseline, candidate, failure examples, and your decision. Include at least one result you are uncertain about.

[Download the lab kit](/course/downloads/docdesk-labs.zip)
