Curriculum
A guided path from application contracts to a defensible release decision.
This self-paced course takes approximately 22–32 hours. It is designed for developers comfortable with Python, HTTP, JSON, and a terminal. No paid API key or GPU is required.
Module 1: Application Engineering
Contracts, provider adapters, output validation, bounded retries.
- Define the product contract — 35 minutes. Before making a model call, decide exactly what a correct answer means.
- Build a provider boundary — 40 minutes. HTTP and model SDKs change. Your business logic should not depend on their quirks.
- Structured output, validation, and abstention — 45 minutes. JSON is a transport format; correctness is a contract enforced at multiple layers.
- Timeouts, retries, and idempotency — 50 minutes. A retry is not a fix. It is another request that consumes time and capacity.
Module 2: Retrieval & Grounding
RAG failures, indexing, ranking, authorization, grounded answers.
- RAG as an evidence pipeline — 40 minutes. Find the exact point where evidence becomes unreliable, instead of calling every error a hallucination.
- Documents, chunks, and metadata — 45 minutes. Chunking trades retrieval precision for context completeness, and metadata determines who may see what.
- Build and compare retrieval strategies — 55 minutes. Put ranking strategies on the same queries, then measure rather than guess.
- Grounded answers and trust boundaries — 50 minutes. A retrieved page is evidence, not a new instruction source.
Module 3: Evaluation Engineering
Test datasets, Recall@k, answer rubrics, regression gates.
- Design evaluations that answer questions — 45 minutes. An eval is a decision tool, not a number to put in a slide.
- Compute retrieval metrics correctly — 50 minutes. Recall@k, Hit@k, and reciprocal rank answer different questions.
- Grade answer quality without fooling yourself — 50 minutes. Answer quality is multidimensional. Start with clear criteria and examples.
- Turn evaluations into release gates — 45 minutes. An eval only becomes engineering when it changes release decisions.
Module 4: Observability & Debugging
Traces, privacy-safe metrics, incident investigation, dashboards.
- Distributed traces and span design — 45 minutes. An end-to-end request should tell a coherent story, not produce five unrelated log lines.
- Logs, metrics, and data minimization — 45 minutes. Logs explain individual events; metrics tell you how frequently they happen.
- Investigate a slow and wrong answer — 55 minutes. Practice a production-style diagnosis with evidence, hypotheses and a postmortem.
- Instrument and review your system — 50 minutes. Telemetry should be designed around questions someone will actually ask during an incident.
Module 5: Cost, Latency & Reliability
Request economics, tail latency, caching, SLOs and rollout.
- Price a request, not a model name — 50 minutes. Understand total cost with measured token usage and a realistic request mix.
- Latency budgets, tails, and streaming — 50 minutes. Measure what users experience, not only the mean of your fastest test.
- Caching, fallback, and failure containment — 50 minutes. Every optimization has a cache-invalidation and correctness story.
- SLOs, rollout, and production readiness — 55 minutes. An AI feature is not ready because its demo succeeded. It is ready when you can operate and recover it.
Module 6: The DocDesk Capstone
Four graded stages, runnable tests, and a technical review.
- Capstone: establish the broken baseline — 50 minutes. You inherit a buggy DocDesk and must produce evidence before changing anything.
- Capstone: repair retrieval and answer contracts — 65 minutes. Apply the first three modules to a system with misleading behavior.
- Capstone: add evaluation and observability — 65 minutes. Prove the system improved and make the remaining failures diagnosable.
- Capstone: reliability, release and technical review — 75 minutes. Finish with a defensible engineering demonstration, not a perfect-looking chatbot.
Practice
Download the offline lab kit and follow the lab setup guide. The checkpoint quizzes test recall; completing the implementation and recording evidence develops the practical skills.