Build a provider boundary
HTTP and model SDKs change. Your business logic should not depend on their quirks.
Estimated study time: 40 minutes
Progress is stored on this device.
Learning outcomes
- Separate transport errors from model outputs
- Specify a model adapter with consistent input/output types
- Preserve request-level metadata without leaking secrets
Keep the model at the edge
A common prototype has a model call buried inside the route handler, plus prompt construction, parsing, logging, and exception handling. This is quick to demonstrate and painful to test. Introduce an adapter so the rest of your application depends on your contract, not on one vendor’s response type.
from dataclasses import dataclass
@dataclass(frozen=True)
class ModelResult:
text: str
input_tokens: int
output_tokens: int
provider_request_id: str | None = None
class ModelGateway:
def complete(self, prompt: str, *, timeout_s: float) -> ModelResult:
raise NotImplementedErrorThe gateway should translate vendor-specific exceptions into your categories, such as TransientProviderError, PermanentProviderError, and DeadlineExceeded. A user-facing API can then choose a status code and message without pattern-matching vendor exception names.
Understand the real request lifecycle
A provider request crosses DNS/TLS, network transfer, queueing, inference, and response transfer. A timeout might happen after the provider started work. Therefore timed out does not necessarily mean not billed or never executed. This matters especially for actions with side effects, where blindly retrying can duplicate them.
For the optional real-model adapter, use credentials in environment variables, set an explicit timeout, and avoid printing prompts or tokens. For an OpenAI Responses API-style integration, a simplified call looks like:
from openai import OpenAI
client = OpenAI() # reads OPENAI_API_KEY
response = client.responses.create(
model="YOUR_SUPPORTED_MODEL",
input="Reply with one factual sentence.",
)
print(response.output_text)This is an illustration, not a production gateway. Production code must bound time, handle refusals and incomplete output, record usage where available, and consider the SDK’s own retries. Model IDs and API capabilities change, so check current provider documentation.
Test against a fake provider first
DocDesk includes FakeProvider, a predictable provider that costs nothing and can simulate malformed output or temporary unavailability. Inject it into the pipeline. The point is not pretending a fake is an LLM; it is testing the system around the LLM deterministically.
Your test matrix should cover:
- success with token usage metadata;
- temporary 429/503-like error;
- permanent unauthorized/quota error;
- malformed completed output;
- timeout and cancellation;
- a valid answer that cites an unauthorized ID.
Important distinction: an API error is different from a model-generated refusal. A refusal may arrive as a successful HTTP response and should be represented intentionally rather than confused with a transport exception.
The design rule
Keep your adapter thin. Put domain rules (tenant authorization, citation checks, answer policy) in the application layer, not inside the provider class. If you later switch models, the business behavior should remain testable.
Hands-on lab
Inspect `labs/docdesk/provider.py`. Implement a `ModelResult` type in your own notes, list the four exception cases you expect, and add one deliberately failing provider scenario to your test plan. The optional real-provider example is in `labs/OPTIONAL_REAL_API.md`.