v1.0.0

Compute retrieval metrics correctly

Recall@k, Hit@k, and reciprocal rank answer different questions.

Estimated study time: 50 minutes

Progress is stored on this device.

Learning outcomes

  • Calculate Hit@k, Recall@k, MRR@k manually
  • Handle multi-relevant-document queries
  • Avoid misleading metric aggregation

One ranking, multiple interpretations

Take a query with relevant documents {A, C} and a ranked retrieval [B, A, D, C].

  • At k=1, Hit@1 = 0, Recall@1 = 0, RR@1 = 0.
  • At k=2, Hit@2 = 1 because at least one relevant document appears; Recall@2 = 1/2 because only A was returned; RR@2 = 1/2 because the first relevant document was rank 2.
  • At k=4, Recall@4 = 1 since both A and C appear. RR@4 remains 1/2.

The formulas are:

Hit@k = 1 if any relevant ID is in top k, otherwise 0

Recall@k = count(unique relevant IDs in top k) / count(gold relevant IDs)

RR@k = 1 / first relevant rank (if within k), otherwise 0

MRR@k is the arithmetic mean of RR@k over queries. Hit@k is usually averaged across queries as a hit rate. Define the policy for queries with an empty gold set: for this course, exclude them from relevance recall averages and evaluate their abstention behavior separately. That is a design choice, and must be reported.

Why a single metric is insufficient

Hit@k tells you whether any relevant evidence appeared, but not if the query needs all parts. MRR cares mostly about the first relevant item. Recall@k rewards retrieving more of the gold set, but ignores irrelevant documents occupying other ranks. nDCG can represent graded relevance, though it introduces labeling and interpretation complexity.

If the correct evidence arrives but the model fabricates an answer, retrieval metrics can still be perfect. Never report retrieval quality as end-to-end answer accuracy.

A realistic measurement loop

  1. Freeze dataset and retrieval configuration.
  2. Record the ranked document IDs for each case.
  3. Compute per-query metrics and slice means.
  4. Inspect errors and label ambiguity.
  5. Change one factor (chunk size, k, ranker).
  6. Repeat with the same dataset, then validate on a held-out set.

Changing retrieval and prompt and model all at once makes it difficult to know what caused an improvement.

Beware easy math mistakes

Use unique document IDs rather than repeated chunks of the same document for document-level recall. Deduplicate top-k IDs before calculating recall so duplicates do not inflate scores. Verify k is nonnegative and define behavior for zero.

Hands-on lab

Implement `recall_at_k` and `reciprocal_rank_at_k` in `labs/docdesk/student.py`. Run `python -m unittest challenges.test_03_evals -v`. Use the interactive metric workbench to verify your calculation before writing code.

Download the lab kit

Knowledge check

Gold docs are {A,C}; top 2 are [B,A]. Recall@2?
Does a high MRR guarantee all required documents were found?

References