v1.0.0

Documents, chunks, and metadata

Chunking trades retrieval precision for context completeness, and metadata determines who may see what.

Estimated study time: 45 minutes

Progress is stored on this device.

Learning outcomes

  • Define a document schema with provenance and version metadata
  • Explain chunk size/overlap trade-offs
  • Preserve document and tenant identity through indexing

Documents are data products

A useful document is more than a blob of text. Give it a stable ID, tenant or access scope, source URI, timestamp/version, title, and body. Keep the original source so a retrieved chunk can point back to something a user can verify.

python
from dataclasses import dataclass

@dataclass(frozen=True)
class Chunk:
    chunk_id: str
    document_id: str
    tenant_id: str
    text: str
    position: int
    version: str

A chunk is a derived record. Updating a source document means updating or invalidating its corresponding indexed chunks. Otherwise the assistant can confidently quote policy that no longer exists.

Why fixed-size chunks can fail

Imagine the source reads: “Annual plans can be refunded within 30 days. Exception: enterprise contracts are nonrefundable.” If the first sentence appears in chunk A and the exception in chunk B, top-1 retrieval may return a misleading policy. Overlap can help retain context but increases duplication and index size. A structure-aware splitter that keeps headings and adjacent exceptions together may work better.

Try three strategies:

  • Whole document: simplest when documents are short; risk of irrelevant context.
  • Fixed token/character window: easy and fast; can separate semantically linked rules.
  • Structure-aware: split at headings/paragraphs and keep policy qualifiers with the section.

There is no universal optimal chunk size. Choose based on document shape, query distribution, and measured retrieval outcomes, not a magic number from a blog post.

Metadata filtering is security-sensitive

A query like tenant_id = acme belongs in retrieval’s authorization filter, preferably enforced by the data layer as well. It must not be a polite request hidden in the prompt. Public/shared documents need an explicit policy; never assume an unknown tenant means public.

Test the update path

Create version 2 of a policy that changes “30 days” to “14 days.” After reindexing, search results must point to version 2. Record not only relevance, but freshness and duplicate rate. If an old chunk remains, your ranking may be excellent while the answer is wrong.

What goes in the artifact

Your lab’s dataset is deliberately tiny. That is intentional: inspect each result manually. Once your algorithm is transparent, evaluate on a broader realistic sample. Toy datasets demonstrate mechanics; they do not establish real-world accuracy.

Hands-on lab

Open `labs/fixtures/documents.json`. Define two additional fields you would need for an auditable production knowledge base. Write a chunking function in a scratch file that preserves the exception sentence alongside its refund rule. Record one failed fixed-window example.

Download the lab kit

Knowledge check

What can happen when policy exceptions split into a different chunk?
Is overlap always better?

References