# Structured output, validation, and abstention

JSON is a transport format; correctness is a contract enforced at multiple layers.

Source: https://bitsandbias.com/course/03-structured/

---

## Learning outcomes

- Validate response shape and business rules separately
- Understand schema-constrained output and its limits
- Handle refusals, incompleteness, and ungrounded citations




## Three different questions

Suppose a model returns:


```json
{"answer":"All refunds are unlimited","citations":["competitor-secret"],"abstained":false}
```

Ask three distinct questions. **Is it parseable JSON?** Yes. **Does it match the schema?** Perhaps. **Is it authorized and factually grounded?** Almost certainly not. Production applications should never collapse these into one `json.loads()` check.

## An explicit validator

The following code illustrates two validation layers:


```python
import json

def parse_answer(text: str, allowed_ids: set[str]) -> dict:
    obj = json.loads(text)
    if not isinstance(obj, dict) or set(obj) != {
        "answer", "citations", "abstained"
    }:
        raise ValueError("unexpected response shape")
    if not isinstance(obj["answer"], str):
        raise ValueError("answer must be a string")
    if not isinstance(obj["citations"], list):
        raise ValueError("citations must be a list")
    if not all(isinstance(x, str) for x in obj["citations"]):
        raise ValueError("citation IDs must be strings")
    if type(obj["abstained"]) is not bool:
        raise ValueError("abstained must be boolean")
    if not set(obj["citations"]).issubset(allowed_ids):
        raise ValueError("citation outside authorized context")
    if obj["abstained"] and obj["citations"]:
        raise ValueError("abstention cannot cite evidence")
    return obj
```

This still cannot establish semantic support. It only verifies a necessary condition: citations refer to documents the system actually supplied. To judge whether a document supports the claim, you need a separate evaluation method or a human reviewer.

## Use constrained generation, not wishful prompting

When the provider supports **schema-constrained structured outputs**, request a supported JSON schema and strict adherence. It reduces malformed responses, but cannot replace validation of domain-specific rules. A schema can require field types, not guarantee an accurate policy answer. Provider-specific supported JSON Schema subsets and refusal behavior differ.

Do not assume every non-error completion contains a successful answer. Explicitly represent incomplete generation, policy refusal, and provider failure. The pipeline should expose an abstention or a typed failure when appropriate, not silently transform an error into a confident answer.

## A failure you should recognize

A tempting shortcut is: “If parsing fails, send the text back to the model with instructions to repair JSON.” That creates extra costs, more variable latency, and can preserve a factually incorrect answer. A repair step may be appropriate in low-risk contexts, but first measure whether a constrained schema solves the real cause.

**Rule of thumb:** treat invalid output as a product event that you can count, debug, and evaluate.



## Hands-on lab

Implement `validate_answer` in `labs/docdesk/student.py`. Start with `python -m unittest challenges.test_01_contract -v` (from the `labs/` folder). Make malformed JSON, wrong field types, and unauthorized citations fail explicitly.

[Download the lab kit](/course/downloads/docdesk-labs.zip)
