# Build a provider boundary

HTTP and model SDKs change. Your business logic should not depend on their quirks.

Source: https://bitsandbias.com/course/02-provider/

---

## Learning outcomes

- Separate transport errors from model outputs
- Specify a model adapter with consistent input/output types
- Preserve request-level metadata without leaking secrets




## Keep the model at the edge

A common prototype has a model call buried inside the route handler, plus prompt construction, parsing, logging, and exception handling. This is quick to demonstrate and painful to test. Introduce an adapter so the rest of your application depends on **your** contract, not on one vendor's response type.


```python
from dataclasses import dataclass

@dataclass(frozen=True)
class ModelResult:
    text: str
    input_tokens: int
    output_tokens: int
    provider_request_id: str | None = None

class ModelGateway:
    def complete(self, prompt: str, *, timeout_s: float) -> ModelResult:
        raise NotImplementedError
```

The gateway should translate vendor-specific exceptions into **your categories**, such as `TransientProviderError`, `PermanentProviderError`, and `DeadlineExceeded`. A user-facing API can then choose a status code and message without pattern-matching vendor exception names.

## Understand the real request lifecycle

A provider request crosses DNS/TLS, network transfer, queueing, inference, and response transfer. A timeout might happen after the provider started work. Therefore **timed out** does not necessarily mean **not billed** or **never executed**. This matters especially for actions with side effects, where blindly retrying can duplicate them.

For the optional real-model adapter, use credentials in environment variables, set an explicit timeout, and avoid printing prompts or tokens. For an OpenAI Responses API-style integration, a simplified call looks like:


```python
from openai import OpenAI
client = OpenAI()  # reads OPENAI_API_KEY
response = client.responses.create(
    model="YOUR_SUPPORTED_MODEL",
    input="Reply with one factual sentence.",
)
print(response.output_text)
```

This is an **illustration**, not a production gateway. Production code must bound time, handle refusals and incomplete output, record usage where available, and consider the SDK's own retries. Model IDs and API capabilities change, so check current provider documentation.

## Test against a fake provider first

DocDesk includes `FakeProvider`, a predictable provider that costs nothing and can simulate malformed output or temporary unavailability. Inject it into the pipeline. The point is not pretending a fake is an LLM; it is testing the system around the LLM deterministically.

Your test matrix should cover:

- success with token usage metadata;
- temporary 429/503-like error;
- permanent unauthorized/quota error;
- malformed completed output;
- timeout and cancellation;
- a valid answer that cites an unauthorized ID.

**Important distinction:** an API error is different from a model-generated refusal. A refusal may arrive as a successful HTTP response and should be represented intentionally rather than confused with a transport exception.

## The design rule

Keep your adapter thin. Put domain rules (tenant authorization, citation checks, answer policy) in the application layer, not inside the provider class. If you later switch models, the business behavior should remain testable.



## Hands-on lab

Inspect `labs/docdesk/provider.py`. Implement a `ModelResult` type in your own notes, list the four exception cases you expect, and add one deliberately failing provider scenario to your test plan. The optional real-provider example is in `labs/OPTIONAL_REAL_API.md`.

[Download the lab kit](/course/downloads/docdesk-labs.zip)
