RAG Formation · Reference

RAG QA quality cheat sheet

A standalone cheat sheet for learning and testing RAG quality on this site.

$ reference --status
▸ course RAG Formation
▸ mode standalone reference
▸ status Complete
● learn → test → verify

Reference sheet

RAG QA quality cheat sheet

The guided RAG Formation course content is complete; this reference is available for self-directed practice.

The pipeline

Question → retrieve evidence → assemble context → generate answer → cite sources → evaluate.

When an answer goes wrong, ask one question first: Was the evidence missing, or was it there but used incorrectly?

Two quality layers

Layer	Question	Demo signal
Retrieval	Did the system return every expected source?	Retrieval union / per-index hit rates
Generation	Did the answer state the requested fact without contradiction?	Strict and lenient judge accuracy
Grounding	Did it cite the documents supporting the answer?	Source hit rate
Safety	Did it refuse questions absent from the corpus?	Refusal accuracy and hallucination rate
Operations	Can users get answers quickly and consistently?	Latency p50 / p95 and API errors

How to read the current demo

Strict accuracy: count only a judge verdict of correct. Use this as the headline number.
Lenient accuracy: count correct + partial. It helps with diagnosis, but it does not replace strict accuracy.
Source hit rate: check that the final answer cites every expected source document.
Refusal accuracy: check that the system correctly declines questions it cannot answer.
Hallucination rate: count how often the system answers an unanswerable question anyway.

Retrieval failure

The right document never makes it into the answer prompt. Look at chunking, metadata, query handling, hybrid retrieval, filters, and ranking.

Generation failure

The right evidence is there, but the answer leaves it out, mixes it up, or contradicts it. Look at context assembly, instructions, answer structure, and model choice.

Course rule

Do not call a prompt improvement a success because of one impressive answer. Run the evaluation harness, inspect the failures, and compare the same metrics with the previous run.

Continue learning: Open the matching standalone lesson · Course roadmap