Latency budgets, tails, and streaming
Measure what users experience, not only the mean of your fastest test.
Estimated study time: 50 minutes
Progress is stored on this device.
Learning outcomes
- Distinguish time to first token from full completion time
- Allocate a per-stage deadline budget
- Explain p95 tail latency and when streaming helps
The anatomy of response time
Total request latency includes network, authentication, retrieval, context construction, provider queueing/inference, output validation, and response transmission. If stages run serially, their durations add up; if independent stages overlap, the critical path determines total time.
Write a budget before calling the model:
| Stage | Example budget |
|---|---|
| Authorization/input validation | 50 ms |
| Retrieval | 300 ms |
| Context construction | 100 ms |
| Model completion | 2,500 ms |
| Validation/response | 50 ms |
| Total | 3,000 ms |
These are illustrative targets. They don’t guarantee the real provider can meet them. Validate with representative traffic and record deadline exceeded as a distinct outcome.
p50 versus p95
p50 is the median. p95 means roughly 95% of observed durations are at or below that threshold for the measured sample/window. Do not call p95 a “worst case”; the slowest 5% remain above it.
A model may average 1.5 seconds yet have a p95 of 6 seconds. For an interactive product those slow requests matter. Test across varied prompt sizes, providers, and realistic network conditions.
Streaming improves perceived responsiveness
Streaming can deliver the first visible token sooner, which often feels faster than waiting for the completed answer. But streaming does not necessarily reduce full response time, and it makes output validation more complicated. If the UI displays an unverified answer mid-stream and your validator rejects it at the end, the user may already have seen misinformation.
For policy-related assistants, a safer approach is to stream a non-committal progress indicator and render the validated answer once complete, or clearly mark provisional text. Choose based on risk, not fashion.
Parallelism is not free
Running retrieval from multiple indexes concurrently can reduce critical-path latency, but may increase infrastructure pressure and cost. Adding retries increases tail latency. Measure effects before adding concurrency.
Diagnose, then optimize
If retrieve takes 50 ms and model_complete takes 3 seconds, reducing retrieval to 10 ms barely matters. Better options may include shorter output, an appropriate faster model, avoiding unnecessary calls, and a validated caching strategy. Use trace data from Module 4.
Hands-on lab
Choose a 4-second total deadline for DocDesk and assign budgets to stages. Write a test that simulates a provider exceeding its budget. Explain what the UI should display while the request runs and how errors are surfaced.