# Distributed traces and span design

An end-to-end request should tell a coherent story, not produce five unrelated log lines.

Source: https://bitsandbias.com/course/13-traces/

---

## Learning outcomes

- Model parent-child spans and trace IDs
- Choose useful stage boundaries and attributes
- Distinguish duration observations from causal proof




## Why traces matter

A response takes 4.8 seconds. Was retrieval slow? Was the model slow? Did your app spend time waiting on a queue? A single total duration cannot tell you. A trace captures connected operations (spans) under one trace ID, which makes it possible to inspect where the time went.

DocDesk's parent span is `answer_request`. Its children are `retrieve`, `build_context`, `model_complete`, and `validate_answer`. Each span can have timing, status, attributes, and potentially events.


```text
answer_request                          0–4800 ms
├── authorize                            0–12 ms
├── retrieve                            12–220 ms
├── build_context                      220–233 ms
├── model_complete                     233–4730 ms
└── validate_answer                   4730–4800 ms
```

In this example, model completion dominates latency. Optimizing the SQL/index of retrieval is unlikely to move the overall total much. This is the first lesson of performance analysis: target measured bottlenecks.

## Trace versus span

A **trace** represents the request's connected execution. A **span** represents one operation. Each child span has a parent, so you can reconstruct nesting. In real distributed systems, context must propagate across process and network boundaries. OpenTelemetry provides standards and SDKs for this, though the local lab uses a minimal teaching tracer to avoid setup requirements.

## Choose meaningful attributes

Good: stage name, hashed request ID, tenant code (if permitted), document count, retrieved ID count, model name, status category, token counts, provider request ID, and code revision. Bad: unrestricted access tokens, raw private documents, or full conversations copied into logs by default.

Trace IDs must be unique enough to correlate calls; attributes must be useful enough to investigate a failure. An unbounded attribute like the entire prompt is expensive, risky, and often impossible to search effectively.

## What to measure

- End-to-end duration.
- Duration by stage.
- Error category by stage.
- Retrieval result count.
- Model usage metadata when available.
- Validation failure type.

A trace does **not** magically prove why a model hallucinated. It gives observations about execution; quality evaluation still requires checking content against evidence.

## Production nuance

Production tracing needs sampling, propagation, collection, retention controls, and access control. Local JSON span records are useful to learn the shape but are **not** a complete observability platform. Review OpenTelemetry's current GenAI semantic conventions before naming production instrumentation attributes.



## Hands-on lab

Implement `answer_query` in `labs/docdesk/student.py` so it emits `answer_request`, `retrieve`, `model_complete`, and `validate_answer` spans using the supplied `TraceRecorder`. Run `python -m unittest challenges.test_04_telemetry -v` and inspect one trace tree.

[Download the lab kit](/course/downloads/docdesk-labs.zip)
