v1.0.0

Latency budgets, tails, and streaming

Measure what users experience, not only the mean of your fastest test.

Estimated study time: 50 minutes

Progress is stored on this device.

Learning outcomes

  • Distinguish time to first token from full completion time
  • Allocate a per-stage deadline budget
  • Explain p95 tail latency and when streaming helps

The anatomy of response time

Total request latency includes network, authentication, retrieval, context construction, provider queueing/inference, output validation, and response transmission. If stages run serially, their durations add up; if independent stages overlap, the critical path determines total time.

Write a budget before calling the model:

Stage Example budget
Authorization/input validation 50 ms
Retrieval 300 ms
Context construction 100 ms
Model completion 2,500 ms
Validation/response 50 ms
Total 3,000 ms

These are illustrative targets. They don’t guarantee the real provider can meet them. Validate with representative traffic and record deadline exceeded as a distinct outcome.

p50 versus p95

p50 is the median. p95 means roughly 95% of observed durations are at or below that threshold for the measured sample/window. Do not call p95 a “worst case”; the slowest 5% remain above it.

A model may average 1.5 seconds yet have a p95 of 6 seconds. For an interactive product those slow requests matter. Test across varied prompt sizes, providers, and realistic network conditions.

Streaming improves perceived responsiveness

Streaming can deliver the first visible token sooner, which often feels faster than waiting for the completed answer. But streaming does not necessarily reduce full response time, and it makes output validation more complicated. If the UI displays an unverified answer mid-stream and your validator rejects it at the end, the user may already have seen misinformation.

For policy-related assistants, a safer approach is to stream a non-committal progress indicator and render the validated answer once complete, or clearly mark provisional text. Choose based on risk, not fashion.

Parallelism is not free

Running retrieval from multiple indexes concurrently can reduce critical-path latency, but may increase infrastructure pressure and cost. Adding retries increases tail latency. Measure effects before adding concurrency.

Diagnose, then optimize

If retrieve takes 50 ms and model_complete takes 3 seconds, reducing retrieval to 10 ms barely matters. Better options may include shorter output, an appropriate faster model, avoiding unnecessary calls, and a validated caching strategy. Use trace data from Module 4.

Hands-on lab

Choose a 4-second total deadline for DocDesk and assign budgets to stages. Write a test that simulates a provider exceeding its budget. Explain what the UI should display while the request runs and how errors are surfaced.

Download the lab kit

Knowledge check

Does streaming necessarily lower total generation time?
What does p95=6 seconds mean?

References