# Grade answer quality without fooling yourself

Answer quality is multidimensional. Start with clear criteria and examples.

Source: https://bitsandbias.com/course/11-answer-evals/

---

## Learning outcomes

- Define groundedness, correctness, abstention, and helpfulness separately
- Write a reviewer rubric with anchored labels
- Explain limits of LLM-as-judge and human agreement




## What does “good answer” mean?

For a documentation assistant, at least four dimensions matter:

1. **Factual correctness:** the answer's claims match the current applicable policy.
2. **Groundedness:** claims are supported by the evidence the model was supplied.
3. **Abstention appropriateness:** the model refuses to guess when the evidence is insufficient.
4. **Usefulness:** the response is clear and actually answers the user's question.

An answer can be correct by coincidence and ungrounded. It can also be grounded in an outdated document and therefore factually wrong. Keep these axes distinct.

## A concrete reviewer rubric

For each answer, compare to the question, authorized retrieved passages, and current policy:

| Label | Meaning | Example |
|---|---|---|
| `supported` | All material claims have direct evidence | “30 days” with a matching policy sentence |
| `unsupported` | Material claim lacks evidence | “Every plan is refundable” with no plan exception |
| `contradicted` | Evidence explicitly disagrees | “365 days” when source says “30 days” |
| `abstained-correctly` | Evidence truly unavailable | “I can't confirm from the provided policy” |
| `abstained-unnecessarily` | Evidence available, but model refuses | Policy clearly states 30 days |

You can derive a pass/fail result for a release gate, but retain labels for diagnosis. A single aggregate score loses the reason why answers fail.

## Use model judges carefully

A model judge can apply a rubric at scale, but may prefer long answers, be influenced by order, or make its own mistakes. Calibrate against a reviewed human-labeled sample; report agreement and where disagreements occur. Blind pairwise comparisons can be useful, but are not a substitute for a grounded reference review.

If you use a model judge, give it the **question, evidence, candidate answer, and scoring rubric**. Require short reasons and structured labels. Keep a human escalation path for high-impact disagreements.

## Evaluate abstention separately

Create a confusion matrix for answerable vs actually answered:

- Answerable + answered correctly: good.
- Answerable + abstained: missed opportunity.
- Unanswerable + abstained: safe.
- Unanswerable + fabricated answer: dangerous.

Optimize for the costs of these errors, not for minimizing abstention at all costs. A support policy assistant should usually prefer an honest gap to a fabricated refund entitlement.

## Manual review exercise

Candidate answer: “All Acme annual plans are refundable for 30 days, including enterprise customers.” The base refund policy mentions 30 days; a separate enterprise exception may override it. A citation to the base policy alone is insufficient. Label at least `unsupported` until you find applicable enterprise evidence; if the exception explicitly forbids refunds, label `contradicted`.



## Hands-on lab

Write `docs/evaluation-rubric.md` with your five label definitions and three concrete examples. Review three generated DocDesk answers manually and record one label per answer with supporting evidence. Explain how you would measure agreement with another reviewer.

[Download the lab kit](/course/downloads/docdesk-labs.zip)
