v1.0.0

Timeouts, retries, and idempotency

A retry is not a fix. It is another request that consumes time and capacity.

Estimated study time: 50 minutes

Progress is stored on this device.

Learning outcomes

  • Choose retryable vs non-retryable failures
  • Create bounded backoff with jitter and a total deadline
  • Explain why nested retries and side effects are dangerous

Classify first, retry second

Typical potentially retryable failures are short network interruptions, transient 503 responses, and some rate-limit responses. Typical non-retryable failures include malformed requests, invalid credentials, exhausted billing quota, and schema errors in your own application. Every provider has nuances; its error classification is the source of truth.

When available, respect Retry-After. Add jitter so 1,000 clients do not retry simultaneously. Never exceed your user’s total deadline, and cap attempts. The provider SDK might already retry; nested retry systems can multiply traffic.

python
# Example schedule, ignoring server-provided Retry-After:
# base=0.4 s, max_delay=3 s, attempts<=4
# wait = min(3.0, 0.4 * 2 ** retry_index)
# jitter is sampled separately for each attempt.

A request budget has multiple parts. If the user can wait 5 seconds, three 4-second timeouts are not a strategy; they are a hidden 12-second promise.

Idempotency is about effects, not text

A repeated read-only model completion usually has no business side effect, but may be charged twice and may produce a different answer. A repeated tool call that creates a refund, posts a message, or writes to a database can perform the action twice. Side-effecting operations need idempotency keys, deduplication, and transaction-aware design. Never give a model unilateral authority to retry a payment.

A concrete decision table

Error Retry automatically? Why
Provider 503, deadline remaining Usually, bounded Temporary unavailability
429 with valid Retry-After Only within the budget Wait at least specified delay
Invalid API key No Configuration must be fixed
Bad JSON in completed model output Not as transport retry Output quality issue
User cancels No Respect cancellation
Unknown result of payment tool Not blindly Duplicate side effects possible

Worked calculation

You allow 4 total attempts, not 4 retries. Suppose attempt times are 0.5, 0.7, 0.8, and 0.6 seconds; waits are 0.4, 0.8, and 1.6 seconds. Worst observed total from these values is 5.4 seconds. Under a 5-second deadline, the fourth attempt should never start if its remaining budget is insufficient. This is why elapsed time must be measured, not assumed.

Design for failed final attempts

Tell the caller a temporary dependency failed; record the failure category and request ID; do not invent an answer. For a consumer app, a safe alternative may be an explicit “try again” action. For a batch job, put the item into a bounded retry queue with deduplication and visibility.

Hands-on lab

Implement `bounded_retry` in `labs/docdesk/student.py` and run `python -m unittest challenges.test_05_reliability -v`. Test 503-then-success, non-retryable errors, and exhausting a deadline. Use the interactive retry planner on this page to choose a policy before coding.

Download the lab kit

Knowledge check

What is the main danger of both SDK and application retries?
A user cancels while waiting. What should happen?

References