v1.0.0

Logs, metrics, and data minimization

Logs explain individual events; metrics tell you how frequently they happen.

Estimated study time: 45 minutes

Progress is stored on this device.

Learning outcomes

  • Differentiate logs, metrics, and traces
  • Define actionable counters and latency summaries
  • Minimize telemetry sensitivity and cardinality

Three observability tools

Logs record discrete events, such as validation_failed with a reason code. Metrics aggregate events across time, such as the share of requests that failed validation. Traces tie the stages of one request together. Use all three, but do not log everything everywhere.

For DocDesk, start with these metrics:

Signal Type Why
Request count by outcome Counter Traffic and failure rate
End-to-end duration Histogram p50 / p95 / p99 and distributions
Provider error category Counter Transient vs permanent failures
Retrieval zero-hit rate Counter / ratio Evidence coverage problems
Output contract failure Counter Structured-output quality
Input/output tokens Counter / distribution Cost and traffic profile

Beware cardinality. A metric label with every unique user question or request ID creates an enormous set of time series. Put unique identifiers in trace metadata or securely controlled logs, not in high-cardinality metric labels.

The privacy design

Never log API keys. Avoid raw prompts, user names, emails, and retrieved content by default. Prefer IDs, counts, timings, and redacted error categories. If raw content is essential for a controlled investigation, make it opt-in with appropriate consent, retention, access control, and deletion processes.

python
safe_event = {
    "event": "model_failure",
    "trace_id": trace_id,
    "stage": "model_complete",
    "error_type": "timeout",
    "elapsed_ms": 1200,
    "input_tokens": 700,
}

Not this:

python
# Do not do this in a production request logger:
print({"api_key": secret, "prompt": private_context})

Analyze distributions rather than averages alone

If 95 fast calls and 5 slow calls are mixed, a mean may look acceptable while users experience occasional terrible delays. Histograms or raw sample distributions let you see tail behavior. Always state the window, population, request type, and sample count next to a percentile.

Instrumentation overhead

Telemetry isn’t free. Excessively verbose traces, synchronous network exporters, and large payloads can add latency and cost. Batch and sample as needed, but keep errors and high-risk paths visible. Retention policies matter as much as visualization.

A small practical standard

For each production error, you should be able to answer: What failed? Where? How often? Is it new? Which release introduced it? Can we reproduce it safely? That is a more useful target than a dashboard containing fifty unlabeled lines.

Hands-on lab

Write a telemetry schema with six fields for `model_complete` and six for `retrieval_complete`. Identify which fields are not allowed in metrics labels. Make a test that rejects a telemetry record containing `api_key` or `raw_prompt`.

Download the lab kit

Knowledge check

Why should request IDs not be metric labels?
What captures the distribution of request latency?

References