Distributed traces and span design
An end-to-end request should tell a coherent story, not produce five unrelated log lines.
Estimated study time: 45 minutes
Progress is stored on this device.
Learning outcomes
- Model parent-child spans and trace IDs
- Choose useful stage boundaries and attributes
- Distinguish duration observations from causal proof
Why traces matter
A response takes 4.8 seconds. Was retrieval slow? Was the model slow? Did your app spend time waiting on a queue? A single total duration cannot tell you. A trace captures connected operations (spans) under one trace ID, which makes it possible to inspect where the time went.
DocDesk’s parent span is answer_request. Its children are retrieve, build_context, model_complete, and validate_answer. Each span can have timing, status, attributes, and potentially events.
answer_request 0–4800 ms
├── authorize 0–12 ms
├── retrieve 12–220 ms
├── build_context 220–233 ms
├── model_complete 233–4730 ms
└── validate_answer 4730–4800 msIn this example, model completion dominates latency. Optimizing the SQL/index of retrieval is unlikely to move the overall total much. This is the first lesson of performance analysis: target measured bottlenecks.
Trace versus span
A trace represents the request’s connected execution. A span represents one operation. Each child span has a parent, so you can reconstruct nesting. In real distributed systems, context must propagate across process and network boundaries. OpenTelemetry provides standards and SDKs for this, though the local lab uses a minimal teaching tracer to avoid setup requirements.
Choose meaningful attributes
Good: stage name, hashed request ID, tenant code (if permitted), document count, retrieved ID count, model name, status category, token counts, provider request ID, and code revision. Bad: unrestricted access tokens, raw private documents, or full conversations copied into logs by default.
Trace IDs must be unique enough to correlate calls; attributes must be useful enough to investigate a failure. An unbounded attribute like the entire prompt is expensive, risky, and often impossible to search effectively.
What to measure
- End-to-end duration.
- Duration by stage.
- Error category by stage.
- Retrieval result count.
- Model usage metadata when available.
- Validation failure type.
A trace does not magically prove why a model hallucinated. It gives observations about execution; quality evaluation still requires checking content against evidence.
Production nuance
Production tracing needs sampling, propagation, collection, retention controls, and access control. Local JSON span records are useful to learn the shape but are not a complete observability platform. Review OpenTelemetry’s current GenAI semantic conventions before naming production instrumentation attributes.
Hands-on lab
Implement `answer_query` in `labs/docdesk/student.py` so it emits `answer_request`, `retrieve`, `model_complete`, and `validate_answer` spans using the supplied `TraceRecorder`. Run `python -m unittest challenges.test_04_telemetry -v` and inspect one trace tree.