Build and compare retrieval strategies
Put ranking strategies on the same queries, then measure rather than guess.
Estimated study time: 55 minutes
Progress is stored on this device.
Learning outcomes
- Explain lexical, semantic, and hybrid retrieval
- Implement a safe tenant-aware ranked search
- Use reciprocal rank fusion and inspect failure cases
Lexical vs semantic search
Lexical ranking finds shared terms; it is useful for precise identifiers such as ERR-429, plan names, and legal clauses. Semantic retrieval maps meanings into a vector space; it can match “get my money back” with “refund eligibility,” but it may also retrieve similar-sounding yet incorrect policies. Hybrid retrieval combines evidence from both.
This lab intentionally uses a dependency-free lexical baseline. It is not a production-grade embedding engine. The goal is to expose ranking decisions, test authorization, and build a benchmark before introducing extra infrastructure.
The top-k decision
k=1 supplies little context and can miss necessary exceptions. Larger k improves the chance of retrieving evidence but can add distracting or conflicting documents, prompt cost, and latency. Optimize k against retrieval recall, answer quality, cost, and privacy constraints together.
For two rankers, reciprocal rank fusion (RRF) is a simple rank-based combination:
RRF(d) = sum(1 / (c + rank_r(d)) for each ranker r returning d)Here c is a positive smoothing constant, often chosen around 60 in examples; it is a tuning parameter, not a law. RRF uses positions, so scores from incompatible rankers need not be normalized together. If a document is missing from one ranking, that ranker contributes nothing.
A tiny ranking experiment
Search query: “How many days can Acme customers request refunds?”
| Rank | Lexical | Semantic (illustrative) |
|---|---|---|
| 1 | acme-refunds |
acme-billing |
| 2 | acme-billing |
acme-refunds |
| 3 | acme-security |
acme-security |
With two rankings both placing acme-refunds in the top two, hybrid fusion keeps it competitive. But if semantic search returns a private document and you fuse before authorization, you may have already created a security issue. Filter authorization first, rank second.
Add failure-focused queries
Build a small query set that includes paraphrases, exact identifiers, ambiguous policy questions, exceptions, unanswerable questions, and tenant mismatches. Inspect the returned IDs rather than celebrating a vector similarity score.
Don’t overinterpret retrieval scores
A cosine similarity of 0.82 is not automatically “82% confidence.” Scores depend on representation, normalization, model, and distribution. Use thresholds only after calibration and validation for your corpus.
Hands-on lab
Implement `retrieve(question, tenant_id, docs, k)` in `labs/docdesk/student.py`. It must enforce tenant filtering, return stable ranked document IDs, handle `k<=0`, and perform better than the naive baseline on the provided questions. Run `python -m unittest challenges.test_02_retrieval -v`.