v1.0.0

Build and compare retrieval strategies

Put ranking strategies on the same queries, then measure rather than guess.

Estimated study time: 55 minutes

Progress is stored on this device.

Learning outcomes

  • Explain lexical, semantic, and hybrid retrieval
  • Implement a safe tenant-aware ranked search
  • Use reciprocal rank fusion and inspect failure cases

Lexical ranking finds shared terms; it is useful for precise identifiers such as ERR-429, plan names, and legal clauses. Semantic retrieval maps meanings into a vector space; it can match “get my money back” with “refund eligibility,” but it may also retrieve similar-sounding yet incorrect policies. Hybrid retrieval combines evidence from both.

This lab intentionally uses a dependency-free lexical baseline. It is not a production-grade embedding engine. The goal is to expose ranking decisions, test authorization, and build a benchmark before introducing extra infrastructure.

The top-k decision

k=1 supplies little context and can miss necessary exceptions. Larger k improves the chance of retrieving evidence but can add distracting or conflicting documents, prompt cost, and latency. Optimize k against retrieval recall, answer quality, cost, and privacy constraints together.

For two rankers, reciprocal rank fusion (RRF) is a simple rank-based combination:

text
RRF(d) = sum(1 / (c + rank_r(d)) for each ranker r returning d)

Here c is a positive smoothing constant, often chosen around 60 in examples; it is a tuning parameter, not a law. RRF uses positions, so scores from incompatible rankers need not be normalized together. If a document is missing from one ranking, that ranker contributes nothing.

A tiny ranking experiment

Search query: “How many days can Acme customers request refunds?”

Rank Lexical Semantic (illustrative)
1 acme-refunds acme-billing
2 acme-billing acme-refunds
3 acme-security acme-security

With two rankings both placing acme-refunds in the top two, hybrid fusion keeps it competitive. But if semantic search returns a private document and you fuse before authorization, you may have already created a security issue. Filter authorization first, rank second.

Add failure-focused queries

Build a small query set that includes paraphrases, exact identifiers, ambiguous policy questions, exceptions, unanswerable questions, and tenant mismatches. Inspect the returned IDs rather than celebrating a vector similarity score.

Don’t overinterpret retrieval scores

A cosine similarity of 0.82 is not automatically “82% confidence.” Scores depend on representation, normalization, model, and distribution. Use thresholds only after calibration and validation for your corpus.

Hands-on lab

Implement `retrieve(question, tenant_id, docs, k)` in `labs/docdesk/student.py`. It must enforce tenant filtering, return stable ranked document IDs, handle `k<=0`, and perform better than the naive baseline on the provided questions. Run `python -m unittest challenges.test_02_retrieval -v`.

Download the lab kit

Knowledge check

What does a cosine similarity of 0.85 mean?
What is one reason to use lexical search?

References