# SLOs, rollout, and production readiness

An AI feature is not ready because its demo succeeded. It is ready when you can operate and recover it.

Source: https://bitsandbias.com/course/20-release/

---

## Learning outcomes

- Define observable service-level indicators and objectives
- Design a safe rollout with rollback and verification
- Connect quality evals, security checks, and operational monitoring




## Choose service indicators carefully

An **SLI** is a measured signal, such as the proportion of eligible requests that return a valid response within 4 seconds. An **SLO** is the desired target over a defined time window. Avoid putting a target in a slide without specifying denominator, measurement window, and exclusions.

Example course SLI:


```text
Good request = authorized, valid-schema, non-error response,
               completed within 4 seconds.
SLI = good_requests / eligible_requests over rolling 7 days.
```

This still does not capture whether the answer was factually correct. Treat sampled answer quality as a **separate quality SLI**, with explicit review methodology, rather than pretending a 200 response means a correct answer.

## Release readiness checklist

- **Correctness:** deterministic contract tests, authorization, citation membership, abstention.
- **Quality:** reviewed held-out evals and relevant slice comparisons.
- **Security:** key management, input limits, prompt-injection threat model, least-privileged tools.
- **Reliability:** deadlines, bounded retries, provider failure handling, rate limits.
- **Operations:** trace correlation, metrics, safe logs, alerts, usage/spend caps.
- **Rollback:** ability to restore previous code/prompt/index versions without guessing which changed.

## Progressive rollout

Start with a canary or small internal audience where appropriate. Compare against the current behavior, inspect error and quality reports, then expand. Keep version labels on prompts, models, and indexes. Roll back if hard security guarantees fail, not only if a KPI looks slightly worse.

## Error budgets and prioritization

An error budget represents the portion of service failures tolerated by your SLO within the measurement window. It helps decide whether to prioritize features or reliability work, but cannot make an unauthorized data leak acceptable. Some conditions are hard invariants, not budgeted degradations.

## What to put in a runbook

Document the top five failure modes, commands to diagnose them, safe fallback behavior, rollback instructions, on-call ownership, provider status links, and how to disable features. Keep runbooks short enough to follow while someone is tired.

## The engineering finish line

You can explain: what the system does, what it must never do, how you evaluate its answers, where it spends time and money, and what happens when its dependencies fail. That is far closer to production readiness than “connected a model API successfully.”



## Hands-on lab

Fill out `docs/release-checklist.md` for DocDesk. Give two SLIs with numerators, denominators, and seven-day windows; name three rollback triggers and identify which are hard security blockers.

[Download the lab kit](/course/downloads/docdesk-labs.zip)
