# Timeouts, retries, and idempotency

A retry is not a fix. It is another request that consumes time and capacity.

Source: https://bitsandbias.com/course/04-retries/

---

## Learning outcomes

- Choose retryable vs non-retryable failures
- Create bounded backoff with jitter and a total deadline
- Explain why nested retries and side effects are dangerous




## Classify first, retry second

Typical **potentially retryable** failures are short network interruptions, transient 503 responses, and some rate-limit responses. Typical **non-retryable** failures include malformed requests, invalid credentials, exhausted billing quota, and schema errors in your own application. Every provider has nuances; its error classification is the source of truth.

When available, respect `Retry-After`. Add jitter so 1,000 clients do not retry simultaneously. Never exceed your user's total deadline, and cap attempts. The provider SDK might already retry; nested retry systems can multiply traffic.


```python
# Example schedule, ignoring server-provided Retry-After:
# base=0.4 s, max_delay=3 s, attempts<=4
# wait = min(3.0, 0.4 * 2 ** retry_index)
# jitter is sampled separately for each attempt.
```

A request budget has multiple parts. If the user can wait 5 seconds, three 4-second timeouts are not a strategy; they are a hidden 12-second promise.

## Idempotency is about effects, not text

A repeated *read-only* model completion usually has no business side effect, but may be charged twice and may produce a different answer. A repeated tool call that creates a refund, posts a message, or writes to a database can perform the action twice. Side-effecting operations need idempotency keys, deduplication, and transaction-aware design. Never give a model unilateral authority to retry a payment.

## A concrete decision table

| Error | Retry automatically? | Why |
|---|---|---|
| Provider 503, deadline remaining | Usually, bounded | Temporary unavailability |
| 429 with valid `Retry-After` | Only within the budget | Wait at least specified delay |
| Invalid API key | No | Configuration must be fixed |
| Bad JSON in completed model output | Not as transport retry | Output quality issue |
| User cancels | No | Respect cancellation |
| Unknown result of payment tool | Not blindly | Duplicate side effects possible |

## Worked calculation

You allow 4 total attempts, not 4 retries. Suppose attempt times are 0.5, 0.7, 0.8, and 0.6 seconds; waits are 0.4, 0.8, and 1.6 seconds. Worst observed total from these values is **5.4 seconds**. Under a 5-second deadline, the fourth attempt should never start if its remaining budget is insufficient. This is why elapsed time must be measured, not assumed.

## Design for failed final attempts

Tell the caller a temporary dependency failed; record the failure category and request ID; do not invent an answer. For a consumer app, a safe alternative may be an explicit “try again” action. For a batch job, put the item into a bounded retry queue with deduplication and visibility.



## Hands-on lab

Implement `bounded_retry` in `labs/docdesk/student.py` and run `python -m unittest challenges.test_05_reliability -v`. Test 503-then-success, non-retryable errors, and exhausting a deadline. Use the interactive retry planner on this page to choose a policy before coding.

[Download the lab kit](/course/downloads/docdesk-labs.zip)
