v1.0.0

Investigate a slow and wrong answer

Practice a production-style diagnosis with evidence, hypotheses and a postmortem.

Estimated study time: 55 minutes

Progress is stored on this device.

Learning outcomes

  • Read a staged incident without guessing
  • Prioritize hypotheses using traces and evaluation cases
  • Write a corrective action with a regression test

Incident 17: “The wrong cancellation policy”

You receive a ticket: users sometimes get a 30-day answer, sometimes a 14-day answer. p95 latency increased after yesterday’s release. The first reaction is often “change the prompt.” Resist that. Create an incident timeline.

Available evidence

  • 10:00: v2 indexing job completed.
  • 10:10: first wrong-answer report.
  • 10:18: p95 rises from 1.8 seconds to 5.9 seconds (illustrative data).
  • 10:25: trace shows 4.5 seconds in model_complete, 0.2 seconds in retrieval.
  • 10:40: top-k result list includes both old and new refund policy versions.

Two issues appear: stale indexing is plausibly connected to contradictory answers; model latency appears connected to the p95 increase. One root cause may not explain both symptoms. Don’t force them into a single story.

The investigation workflow

  1. Confirm impact and severity: number of requests, affected tenants, exposure risk.
  2. Stabilize: roll back index version or disable stale documents if safe; reduce traffic if provider is overloaded.
  3. Gather bounded evidence: trace IDs, version, retrieved document IDs, request timings, sampled answer labels.
  4. Test competing hypotheses: stale index, prompt regression, provider delay, client retry amplification.
  5. Make the smallest reversible repair.
  6. Add a regression case and verify metrics after the fix.
  7. Record what was observed versus merely suspected.

A useful hypothesis table

Hypothesis Evidence for Evidence against Next check
New policy not replacing old index Both versions returned None yet Inspect index manifest
Retrieval latency caused p95 None Retrieval only 200 ms Deprioritize
Nested retries amplified provider calls New release updated SDK No attempt counts yet Compare attempts per request

Blameless does not mean vague

A good postmortem identifies a system weakness and a verifiable action. “Engineers should be more careful” is not a useful preventive measure. “Enforce one active policy version per tenant in the index, and test it in CI” is actionable.

Write the postmortem

Include: user impact, detection time, mitigation, timeline, immediate root cause hypothesis, contributing factors, data still missing, owner, and verification criteria. If the root cause is unproven, say so. A confident but invented postmortem is worse than an honest investigation.

Hands-on lab

Use `docs/incident-template.md` and produce a postmortem for Incident 17. Include at least two independent hypotheses, one immediate mitigation, and three verification checks. The trace explorer on lesson 13 contains a short simulation you can use as supporting evidence.

Download the lab kit

Knowledge check

What did the trace mainly implicate in the latency rise?
Which corrective action is measurable?

References