# Logs, metrics, and data minimization

Logs explain individual events; metrics tell you how frequently they happen.

Source: https://bitsandbias.com/course/14-logs-metrics/

---

## Learning outcomes

- Differentiate logs, metrics, and traces
- Define actionable counters and latency summaries
- Minimize telemetry sensitivity and cardinality




## Three observability tools

**Logs** record discrete events, such as `validation_failed` with a reason code. **Metrics** aggregate events across time, such as the share of requests that failed validation. **Traces** tie the stages of one request together. Use all three, but do not log everything everywhere.

For DocDesk, start with these metrics:

| Signal | Type | Why |
|---|---|---|
| Request count by outcome | Counter | Traffic and failure rate |
| End-to-end duration | Histogram | p50 / p95 / p99 and distributions |
| Provider error category | Counter | Transient vs permanent failures |
| Retrieval zero-hit rate | Counter / ratio | Evidence coverage problems |
| Output contract failure | Counter | Structured-output quality |
| Input/output tokens | Counter / distribution | Cost and traffic profile |

Beware cardinality. A metric label with every unique user question or request ID creates an enormous set of time series. Put unique identifiers in trace metadata or securely controlled logs, not in high-cardinality metric labels.

## The privacy design

Never log API keys. Avoid raw prompts, user names, emails, and retrieved content by default. Prefer IDs, counts, timings, and redacted error categories. If raw content is essential for a controlled investigation, make it opt-in with appropriate consent, retention, access control, and deletion processes.


```python
safe_event = {
    "event": "model_failure",
    "trace_id": trace_id,
    "stage": "model_complete",
    "error_type": "timeout",
    "elapsed_ms": 1200,
    "input_tokens": 700,
}
```

**Not** this:


```python
# Do not do this in a production request logger:
print({"api_key": secret, "prompt": private_context})
```

## Analyze distributions rather than averages alone

If 95 fast calls and 5 slow calls are mixed, a mean may look acceptable while users experience occasional terrible delays. Histograms or raw sample distributions let you see tail behavior. Always state the window, population, request type, and sample count next to a percentile.

## Instrumentation overhead

Telemetry isn't free. Excessively verbose traces, synchronous network exporters, and large payloads can add latency and cost. Batch and sample as needed, but keep errors and high-risk paths visible. Retention policies matter as much as visualization.

## A small practical standard

For each production error, you should be able to answer: What failed? Where? How often? Is it new? Which release introduced it? Can we reproduce it safely? That is a more useful target than a dashboard containing fifty unlabeled lines.



## Hands-on lab

Write a telemetry schema with six fields for `model_complete` and six for `retrieval_complete`. Identify which fields are not allowed in metrics labels. Make a test that rejects a telemetry record containing `api_key` or `raw_prompt`.

[Download the lab kit](/course/downloads/docdesk-labs.zip)
