Instrument and review your system
Telemetry should be designed around questions someone will actually ask during an incident.
Estimated study time: 50 minutes
Progress is stored on this device.
Learning outcomes
- Define a minimal observability dashboard
- Correlate one request with its stage timing and error category
- Identify when traces must be sampled or redacted
Build the dashboard backwards
Start with decisions, not graphs. An on-call engineer needs to know if user impact is increasing, where latency is coming from, whether the issue started with a release, and if results are trustworthy.
A useful first dashboard includes:
- Traffic and success rate: requests per minute, error categories, abstention rate.
- Latency: p50, p95, and p99, both total and per stage, with sample counts.
- Model usage and spend: tokens, error rates, estimated cost.
- Retrieval: zero-hit rate, candidate count, query categories.
- Quality: recent reviewed eval trend, stale document detection, incident-linked failures.
Sampling policy
Storing full traces for every request can be costly. A possible policy: sample a small percentage of normal requests, retain a higher fraction of errors, and always record lightweight aggregate metrics. This is an example approach; retention, sensitive fields, and sampling implementation require application-specific decisions. If you sample only successes and drop failures, you’ll have a dashboard that looks excellent during outages.
Correlate without oversharing
Use a trace ID to connect the API response, safe logs, and internal spans. If support staff need access to raw user content for a particular incident, use a separate audited workflow, not routine full-content tracing.
The tracer exercise
In the Python lab, a minimal TraceRecorder collects named stages and durations. It has the same conceptual role as tracing instrumentation but does not implement the complete OpenTelemetry specification or exporters. Its purpose is to make the developer prove the parent-child structure and distinguish stage timings.
with tracer.span("answer_request"):
with tracer.span("retrieve", document_count=len(docs)):
matches = retrieve(question, tenant, docs, k=3)In a multi-service production application, propagate context using standardized tracing instrumentation instead of inventing your own protocol.
Review checklist
Before shipping, ask: Can you tie a customer-visible failure to a trace? Are error codes stable? Can you distinguish permission failures from missing evidence? Are dashboards showing current data? Do they expose unnecessary personal or confidential text? Would someone unfamiliar with your code know where to start?
If the answer to the last question is no, improve the diagnostic path, not the color palette of the dashboard.
Hands-on lab
Instrument the example pipeline to output per-stage timing and result counts. Create a one-page dashboard specification with five panels, their data sources, thresholds, and a question each panel answers.