Logs, metrics, and data minimization
Logs explain individual events; metrics tell you how frequently they happen.
Estimated study time: 45 minutes
Progress is stored on this device.
Learning outcomes
- Differentiate logs, metrics, and traces
- Define actionable counters and latency summaries
- Minimize telemetry sensitivity and cardinality
Three observability tools
Logs record discrete events, such as validation_failed with a reason code. Metrics aggregate events across time, such as the share of requests that failed validation. Traces tie the stages of one request together. Use all three, but do not log everything everywhere.
For DocDesk, start with these metrics:
| Signal | Type | Why |
|---|---|---|
| Request count by outcome | Counter | Traffic and failure rate |
| End-to-end duration | Histogram | p50 / p95 / p99 and distributions |
| Provider error category | Counter | Transient vs permanent failures |
| Retrieval zero-hit rate | Counter / ratio | Evidence coverage problems |
| Output contract failure | Counter | Structured-output quality |
| Input/output tokens | Counter / distribution | Cost and traffic profile |
Beware cardinality. A metric label with every unique user question or request ID creates an enormous set of time series. Put unique identifiers in trace metadata or securely controlled logs, not in high-cardinality metric labels.
The privacy design
Never log API keys. Avoid raw prompts, user names, emails, and retrieved content by default. Prefer IDs, counts, timings, and redacted error categories. If raw content is essential for a controlled investigation, make it opt-in with appropriate consent, retention, access control, and deletion processes.
safe_event = {
"event": "model_failure",
"trace_id": trace_id,
"stage": "model_complete",
"error_type": "timeout",
"elapsed_ms": 1200,
"input_tokens": 700,
}Not this:
# Do not do this in a production request logger:
print({"api_key": secret, "prompt": private_context})Analyze distributions rather than averages alone
If 95 fast calls and 5 slow calls are mixed, a mean may look acceptable while users experience occasional terrible delays. Histograms or raw sample distributions let you see tail behavior. Always state the window, population, request type, and sample count next to a percentile.
Instrumentation overhead
Telemetry isn’t free. Excessively verbose traces, synchronous network exporters, and large payloads can add latency and cost. Batch and sample as needed, but keep errors and high-risk paths visible. Retention policies matter as much as visualization.
A small practical standard
For each production error, you should be able to answer: What failed? Where? How often? Is it new? Which release introduced it? Can we reproduce it safely? That is a more useful target than a dashboard containing fifty unlabeled lines.
Hands-on lab
Write a telemetry schema with six fields for `model_complete` and six for `retrieval_complete`. Identify which fields are not allowed in metrics labels. Make a test that rejects a telemetry record containing `api_key` or `raw_prompt`.