Timeouts, retries, and idempotency
A retry is not a fix. It is another request that consumes time and capacity.
Estimated study time: 50 minutes
Progress is stored on this device.
Learning outcomes
- Choose retryable vs non-retryable failures
- Create bounded backoff with jitter and a total deadline
- Explain why nested retries and side effects are dangerous
Classify first, retry second
Typical potentially retryable failures are short network interruptions, transient 503 responses, and some rate-limit responses. Typical non-retryable failures include malformed requests, invalid credentials, exhausted billing quota, and schema errors in your own application. Every provider has nuances; its error classification is the source of truth.
When available, respect Retry-After. Add jitter so 1,000 clients do not retry simultaneously. Never exceed your user’s total deadline, and cap attempts. The provider SDK might already retry; nested retry systems can multiply traffic.
# Example schedule, ignoring server-provided Retry-After:
# base=0.4 s, max_delay=3 s, attempts<=4
# wait = min(3.0, 0.4 * 2 ** retry_index)
# jitter is sampled separately for each attempt.A request budget has multiple parts. If the user can wait 5 seconds, three 4-second timeouts are not a strategy; they are a hidden 12-second promise.
Idempotency is about effects, not text
A repeated read-only model completion usually has no business side effect, but may be charged twice and may produce a different answer. A repeated tool call that creates a refund, posts a message, or writes to a database can perform the action twice. Side-effecting operations need idempotency keys, deduplication, and transaction-aware design. Never give a model unilateral authority to retry a payment.
A concrete decision table
| Error | Retry automatically? | Why |
|---|---|---|
| Provider 503, deadline remaining | Usually, bounded | Temporary unavailability |
429 with valid Retry-After |
Only within the budget | Wait at least specified delay |
| Invalid API key | No | Configuration must be fixed |
| Bad JSON in completed model output | Not as transport retry | Output quality issue |
| User cancels | No | Respect cancellation |
| Unknown result of payment tool | Not blindly | Duplicate side effects possible |
Worked calculation
You allow 4 total attempts, not 4 retries. Suppose attempt times are 0.5, 0.7, 0.8, and 0.6 seconds; waits are 0.4, 0.8, and 1.6 seconds. Worst observed total from these values is 5.4 seconds. Under a 5-second deadline, the fourth attempt should never start if its remaining budget is insufficient. This is why elapsed time must be measured, not assumed.
Design for failed final attempts
Tell the caller a temporary dependency failed; record the failure category and request ID; do not invent an answer. For a consumer app, a safe alternative may be an explicit “try again” action. For a batch job, put the item into a bounded retry queue with deduplication and visibility.
Hands-on lab
Implement `bounded_retry` in `labs/docdesk/student.py` and run `python -m unittest challenges.test_05_reliability -v`. Test 503-then-success, non-retryable errors, and exhausting a deadline. Use the interactive retry planner on this page to choose a policy before coding.