RAG Formation · Reference
Trustworthy RAG evaluation sets
A standalone cheat sheet for learning and testing RAG quality on this site.
$ reference --status
▸ course RAG Formation
▸ mode standalone reference
▸ status Complete
● learn → test → verifyReference sheet
Trustworthy RAG evaluation sets
The guided RAG Formation course content is complete; this reference is available for self-directed practice.
The evaluation tuple
Question + category + expected sources + required facts + expected behavior
For an answerable question, the system should give a correct answer grounded in the evidence. For an unanswerable question, it should refuse.
What a good set tests
| Dimension | Why it matters | Meridian demo |
|---|---|---|
| Factual lookup | Find a precise number, name, date, or requirement. | 35 questions |
| Reference lookup | Find the exact SOP or document identifier. | 7 questions |
| Multi-hop | Pull together evidence from more than one document. | 9 questions |
| Cross-lingual | Match a question with a source written in another language. | 5 questions |
| Acronym | Resolve domain terms and abbreviations. | 3 questions |
| Unanswerable | Measure whether the system refuses instead of rewarding a hallucination. | 6 questions |
Ground truth rules
- Every expected source must be present in the corpus.
- Every required fact must appear literally in an expected source.
- An unanswerable question must have neither an expected source nor a required fact.
- Match retrieved sources by metadata, not by arbitrary text in a chunk.
- Keep the evaluation set unchanged while comparing system changes.
Bad measurement
Count a vector result as a hit just because its chunk text mentions the expected document ID.
Better measurement
Count it as a hit only when the source metadata identifies the expected document.
Trust boundary
An evaluation harness can be wrong and still print a confident-looking percentage. Test the scorer itself with accents, multi-source questions, body-text cross-references, missing citations, and refusals.
Continue learning: Open the matching standalone lesson · Course roadmap