# Latency budgets, tails, and streaming

Measure what users experience, not only the mean of your fastest test.

Source: https://bitsandbias.com/course/18-latency/

---

## Learning outcomes

- Distinguish time to first token from full completion time
- Allocate a per-stage deadline budget
- Explain p95 tail latency and when streaming helps




## The anatomy of response time

Total request latency includes network, authentication, retrieval, context construction, provider queueing/inference, output validation, and response transmission. If stages run serially, their durations add up; if independent stages overlap, the critical path determines total time.

Write a budget before calling the model:

| Stage | Example budget |
|---|---:|
| Authorization/input validation | 50 ms |
| Retrieval | 300 ms |
| Context construction | 100 ms |
| Model completion | 2,500 ms |
| Validation/response | 50 ms |
| **Total** | **3,000 ms** |

These are illustrative targets. They don't guarantee the real provider can meet them. Validate with representative traffic and record deadline exceeded as a distinct outcome.

## p50 versus p95

p50 is the median. p95 means roughly 95% of observed durations are at or below that threshold for the measured sample/window. Do not call p95 a “worst case”; the slowest 5% remain above it.

A model may average 1.5 seconds yet have a p95 of 6 seconds. For an interactive product those slow requests matter. Test across varied prompt sizes, providers, and realistic network conditions.

## Streaming improves perceived responsiveness

Streaming can deliver the first visible token sooner, which often feels faster than waiting for the completed answer. But streaming **does not necessarily reduce full response time**, and it makes output validation more complicated. If the UI displays an unverified answer mid-stream and your validator rejects it at the end, the user may already have seen misinformation.

For policy-related assistants, a safer approach is to stream a non-committal progress indicator and render the validated answer once complete, or clearly mark provisional text. Choose based on risk, not fashion.

## Parallelism is not free

Running retrieval from multiple indexes concurrently can reduce critical-path latency, but may increase infrastructure pressure and cost. Adding retries increases tail latency. Measure effects before adding concurrency.

## Diagnose, then optimize

If `retrieve` takes 50 ms and `model_complete` takes 3 seconds, reducing retrieval to 10 ms barely matters. Better options may include shorter output, an appropriate faster model, avoiding unnecessary calls, and a validated caching strategy. Use trace data from Module 4.



## Hands-on lab

Choose a 4-second total deadline for DocDesk and assign budgets to stages. Write a test that simulates a provider exceeding its budget. Explain what the UI should display while the request runs and how errors are surfaced.

[Download the lab kit](/course/downloads/docdesk-labs.zip)
