v1.0.0

RAG as an evidence pipeline

Find the exact point where evidence becomes unreliable, instead of calling every error a hallucination.

Estimated study time: 40 minutes

Progress is stored on this device.

Learning outcomes

  • Trace the five stages of retrieval-augmented generation
  • Separate retrieval, authorization, and generation failures
  • Construct a useful failure taxonomy

Stop treating RAG like a magic box

Retrieval-augmented generation (RAG) is a workflow: ingest documents, represent and index them, retrieve relevant passages, assemble context, and generate a response. An answer may fail even when the language model behaves perfectly. If the retrieval step did not include the necessary policy, the generator cannot cite it.

DocDesk answers support questions about different fictional tenants. Each document has an ID, title, tenant, and body. Tenant filtering must happen before content can enter a model prompt. Filtering citations after the answer is produced is too late because private text has already crossed a trust boundary.

Five places the pipeline can break

  1. Ingestion: content missing, stale, incorrectly parsed, or corrupted.
  2. Chunking/index: an important qualification is separated from the policy it modifies.
  3. Retrieval: a semantically related but wrong document outranks the exact answer.
  4. Context building: relevant evidence gets cut off, duplicates fill the window, or private chunks leak in.
  5. Generation: an answer ignores evidence, combines incompatible policies, or invents a citation.

Each requires a different fix. If the correct document never appears in the top results, rewriting the generator prompt won’t repair retrieval recall. If the right passage is present but the model contradicts it, improving chunking won’t directly fix that failure.

Run a diagnostic conversation

Question: “What happens if I cancel an annual plan on day 20?”

  1. Verify the user has tenant acme.
  2. Look for an authoritative acme refund document. Check its date/version.
  3. Inspect the top-k retrieved IDs and their ranks.
  4. Inspect the exact context supplied, within logging privacy limits.
  5. Compare the generated claim against the cited passage.
  6. Classify the failure using the earliest wrong stage.

Try a deceptively similar query about a different tenant. Matching the words “annual plan” is not permission to view its policy.

Baselines before embeddings

Start with deterministic lexical matching. It is transparent, free, and can be surprisingly strong for exact product names and codes. Dense embeddings are useful for semantic similarity, but they can obscure debugging and do not solve access control. In this course the baseline uses simple token-based lexical ranking; you will later compare improvements.

The operating principle

Make retrieval inspectable. Save the authorized candidate count, ranked IDs, retrieval scores, document versions, and stage timings. Avoid storing raw document text unless your privacy policy and access controls expressly allow it.

Hands-on lab

Run `python -m docdesk demo` and inspect the retrieved document IDs. Identify whether a wrong answer is caused by missing evidence, wrong ranking, or unsupported generation. Write three failure hypotheses before editing any code.

Download the lab kit

Knowledge check

The correct document never reaches the model. What should be debugged first?
When is tenant access filtering required?

References