RAG Formation · Lesson · Foundations & measurement
Build a trustworthy evaluation set
Turn expectations about quality into explicit, testable ground truth.
$ lesson --status
▸ course RAG Formation
▸ lesson 03 / 16
▸ phase Foundations & measurement
▸ status Complete
● build → measure → learnStandalone lesson
Learn the full lesson and test your understanding here.
The 16-lesson RAG Formation curriculum is complete. This page contains the complete lesson content, local reference material, and instant browser exercises.
Today’s tangible win
By the end, you can pick apart a single evaluation question and say exactly why its ground truth holds up—or what’s wrong with it if it doesn’t.
Watch the walkthrough
1 · An evaluation is a contract
An evaluation question isn’t just something a user might type. It’s a contract between the product team and whatever’s doing the scoring:
Question: what the user asks
Expected sources: which documents must support the answer
Required facts / must_include: the high-signal facts a correct answer must contain
Expected answer: the reference meaning used by the judge
Expected refusal / expect_refusal: whether the system should decline because the corpus has no answer
Skip that contract, and your score can look perfectly rigorous while quietly measuring the wrong thing entirely.
Primary reading: OpenAI’s Evaluation Best Practices—pay attention to why nondeterministic AI systems need structured evals in the first place.
2 · Read the demo set
Open up:
demo/eval_set.yamlThis demo set packs 65 questions across 25 synthetic documents, split like this:
- 35 factual questions
- 7 reference-lookup questions
- 9 multi-hop questions
- 5 cross-lingual questions
- 3 acronym questions
- 6 unanswerable questions
That mix isn’t arbitrary. A chatbot can breeze through ordinary semantic questions and still choke on exact SOP identifiers, multi-document reasoning, or knowing when to say no.
3 · Why retrieval matching uses metadata only
Take q30:
Expected sources: `PR-QA-MRD-001`, `PR-QA-MRD-009`
Required fact: `90 jours`
Here’s the trap: documents in this corpus cross-reference each other. A chunk from `PR-QA-MRD-001` might mention `PR-QA-MRD-009` right in its body text. Search that whole chunk blindly, and the scorer could wrongly credit `PR-QA-MRD-009` as retrieved when it never actually showed up.
That’s exactly why `eval.py` only trusts source metadata—fields like `source`, `document_name`, and `filename`—instead of handing out credit just because an identifier shows up somewhere in the body text.
4 · Answerable versus unanswerable
Those six unanswerable questions aren’t sloppy test design—they’re safety checks. Each one asks a simple thing: does the system know where its own knowledge ends?
| Question type | Expected source | Expected behavior |
|---|---|---|
| Answerable | One or more documents | Answer the requested fact and cite the sources |
| Unanswerable | None | Decline or state that the indexed data does not contain the answer |
5 · Practice: trust the question or reject it?
Case A: `expect_sources` names `PR-QA-MRD-009`, but no such file exists anywhere in the corpus. Does this question still count as valid?
Case B: The expected document ID only shows up inside another document’s body text, nowhere else. Does retrieval still deserve credit?
Case C: A question has no expected source at all and is flagged `expect_refusal: true`. What’s the right move for a good chatbot?
6 · Your evaluation-set review
Pick one question from each category in `eval_set.yaml` and write down:
- Is the question answerable at all?
- Which source document(s) actually support it?
- What fact has to show up in a correct answer?
- Could a cross-reference trick the scorer into a false credit?
- What does a safe failure look like here?
Mission connection
In SOP, QA, regulatory, supplier, and audit use cases, the evaluation set isn’t just a testing tool—it’s part of the product’s safety boundary. It’s what defines “supported,” “correct,” and “unknown” in the first place.
Next action
Run the offline consistency check one more time:
cd demo
python3 eval.py verifyThen pick one question from each of the six categories and write up a short review. Send me your answers—especially one case where metadata-only matching saves you from a false retrieval credit.
Got questions about ground truth, scoring, or any of these categories? Ask me. Once your review is in, I’ll mark Lesson 3 complete.