Investigate a slow and wrong answer
Practice a production-style diagnosis with evidence, hypotheses and a postmortem.
Estimated study time: 55 minutes
Progress is stored on this device.
Learning outcomes
- Read a staged incident without guessing
- Prioritize hypotheses using traces and evaluation cases
- Write a corrective action with a regression test
Incident 17: “The wrong cancellation policy”
You receive a ticket: users sometimes get a 30-day answer, sometimes a 14-day answer. p95 latency increased after yesterday’s release. The first reaction is often “change the prompt.” Resist that. Create an incident timeline.
Available evidence
- 10:00:
v2indexing job completed. - 10:10: first wrong-answer report.
- 10:18: p95 rises from 1.8 seconds to 5.9 seconds (illustrative data).
- 10:25: trace shows 4.5 seconds in
model_complete, 0.2 seconds in retrieval. - 10:40: top-k result list includes both old and new refund policy versions.
Two issues appear: stale indexing is plausibly connected to contradictory answers; model latency appears connected to the p95 increase. One root cause may not explain both symptoms. Don’t force them into a single story.
The investigation workflow
- Confirm impact and severity: number of requests, affected tenants, exposure risk.
- Stabilize: roll back index version or disable stale documents if safe; reduce traffic if provider is overloaded.
- Gather bounded evidence: trace IDs, version, retrieved document IDs, request timings, sampled answer labels.
- Test competing hypotheses: stale index, prompt regression, provider delay, client retry amplification.
- Make the smallest reversible repair.
- Add a regression case and verify metrics after the fix.
- Record what was observed versus merely suspected.
A useful hypothesis table
| Hypothesis | Evidence for | Evidence against | Next check |
|---|---|---|---|
| New policy not replacing old index | Both versions returned | None yet | Inspect index manifest |
| Retrieval latency caused p95 | None | Retrieval only 200 ms | Deprioritize |
| Nested retries amplified provider calls | New release updated SDK | No attempt counts yet | Compare attempts per request |
Blameless does not mean vague
A good postmortem identifies a system weakness and a verifiable action. “Engineers should be more careful” is not a useful preventive measure. “Enforce one active policy version per tenant in the index, and test it in CI” is actionable.
Write the postmortem
Include: user impact, detection time, mitigation, timeline, immediate root cause hypothesis, contributing factors, data still missing, owner, and verification criteria. If the root cause is unproven, say so. A confident but invented postmortem is worse than an honest investigation.
Hands-on lab
Use `docs/incident-template.md` and produce a postmortem for Incident 17. Include at least two independent hypotheses, one immediate mitigation, and three verification checks. The trace explorer on lesson 13 contains a short simulation you can use as supporting evidence.