# Investigate a slow and wrong answer

Practice a production-style diagnosis with evidence, hypotheses and a postmortem.

Source: https://bitsandbias.com/course/15-incident/

---

## Learning outcomes

- Read a staged incident without guessing
- Prioritize hypotheses using traces and evaluation cases
- Write a corrective action with a regression test




## Incident 17: “The wrong cancellation policy”

You receive a ticket: users sometimes get a 30-day answer, sometimes a 14-day answer. p95 latency increased after yesterday's release. The first reaction is often “change the prompt.” Resist that. Create an incident timeline.

### Available evidence

- 10:00: `v2` indexing job completed.
- 10:10: first wrong-answer report.
- 10:18: p95 rises from 1.8 seconds to 5.9 seconds (illustrative data).
- 10:25: trace shows 4.5 seconds in `model_complete`, 0.2 seconds in retrieval.
- 10:40: top-k result list includes both old and new refund policy versions.

Two issues appear: **stale indexing** is plausibly connected to contradictory answers; **model latency** appears connected to the p95 increase. One root cause may not explain both symptoms. Don't force them into a single story.

## The investigation workflow

1. Confirm impact and severity: number of requests, affected tenants, exposure risk.
2. Stabilize: roll back index version or disable stale documents if safe; reduce traffic if provider is overloaded.
3. Gather bounded evidence: trace IDs, version, retrieved document IDs, request timings, sampled answer labels.
4. Test competing hypotheses: stale index, prompt regression, provider delay, client retry amplification.
5. Make the smallest reversible repair.
6. Add a regression case and verify metrics after the fix.
7. Record what was observed versus merely suspected.

## A useful hypothesis table

| Hypothesis | Evidence for | Evidence against | Next check |
|---|---|---|---|
| New policy not replacing old index | Both versions returned | None yet | Inspect index manifest |
| Retrieval latency caused p95 | None | Retrieval only 200 ms | Deprioritize |
| Nested retries amplified provider calls | New release updated SDK | No attempt counts yet | Compare attempts per request |

## Blameless does not mean vague

A good postmortem identifies a system weakness and a verifiable action. “Engineers should be more careful” is not a useful preventive measure. “Enforce one active policy version per tenant in the index, and test it in CI” is actionable.

## Write the postmortem

Include: user impact, detection time, mitigation, timeline, immediate root cause hypothesis, contributing factors, data still missing, owner, and verification criteria. If the root cause is unproven, say so. A confident but invented postmortem is worse than an honest investigation.



## Hands-on lab

Use `docs/incident-template.md` and produce a postmortem for Incident 17. Include at least two independent hypotheses, one immediate mitigation, and three verification checks. The trace explorer on lesson 13 contains a short simulation you can use as supporting evidence.

[Download the lab kit](/course/downloads/docdesk-labs.zip)
