# Build and compare retrieval strategies

Put ranking strategies on the same queries, then measure rather than guess.

Source: https://bitsandbias.com/course/07-retrieval/

---

## Learning outcomes

- Explain lexical, semantic, and hybrid retrieval
- Implement a safe tenant-aware ranked search
- Use reciprocal rank fusion and inspect failure cases




## Lexical vs semantic search

Lexical ranking finds shared terms; it is useful for precise identifiers such as `ERR-429`, plan names, and legal clauses. Semantic retrieval maps meanings into a vector space; it can match “get my money back” with “refund eligibility,” but it may also retrieve similar-sounding yet incorrect policies. Hybrid retrieval combines evidence from both.

This lab intentionally uses a **dependency-free lexical baseline**. It is not a production-grade embedding engine. The goal is to expose ranking decisions, test authorization, and build a benchmark before introducing extra infrastructure.

## The top-k decision

`k=1` supplies little context and can miss necessary exceptions. Larger `k` improves the chance of retrieving evidence but can add distracting or conflicting documents, prompt cost, and latency. Optimize k against **retrieval recall, answer quality, cost, and privacy constraints together**.

For two rankers, reciprocal rank fusion (RRF) is a simple rank-based combination:


```text
RRF(d) = sum(1 / (c + rank_r(d)) for each ranker r returning d)
```

Here `c` is a positive smoothing constant, often chosen around 60 in examples; it is a tuning parameter, not a law. RRF uses **positions**, so scores from incompatible rankers need not be normalized together. If a document is missing from one ranking, that ranker contributes nothing.

## A tiny ranking experiment

Search query: “How many days can Acme customers request refunds?”

| Rank | Lexical | Semantic (illustrative) |
|---|---|---|
| 1 | `acme-refunds` | `acme-billing` |
| 2 | `acme-billing` | `acme-refunds` |
| 3 | `acme-security` | `acme-security` |

With two rankings both placing `acme-refunds` in the top two, hybrid fusion keeps it competitive. But if semantic search returns a private document and you fuse before authorization, you may have already created a security issue. **Filter authorization first, rank second.**

## Add failure-focused queries

Build a small query set that includes paraphrases, exact identifiers, ambiguous policy questions, exceptions, unanswerable questions, and tenant mismatches. Inspect the returned IDs rather than celebrating a vector similarity score.

## Don't overinterpret retrieval scores

A cosine similarity of 0.82 is not automatically “82% confidence.” Scores depend on representation, normalization, model, and distribution. Use thresholds only after calibration and validation for your corpus.



## Hands-on lab

Implement `retrieve(question, tenant_id, docs, k)` in `labs/docdesk/student.py`. It must enforce tenant filtering, return stable ranked document IDs, handle `k<=0`, and perform better than the naive baseline on the provided questions. Run `python -m unittest challenges.test_02_retrieval -v`.

[Download the lab kit](/course/downloads/docdesk-labs.zip)
