RAG Formation · Lesson · Foundations & measurement
From employee question to cited answer
Build the first mental model for a RAG chatbot people can actually trust.
$ lesson --status
▸ course RAG Formation
▸ lesson 01 / 16
▸ phase Foundations & measurement
▸ status Complete
● build → measure → learnStandalone lesson
Learn the full lesson and test your understanding here.
The 16-lesson RAG Formation curriculum is complete. This page contains the complete lesson content, local reference material, and instant browser exercises.
Today’s tangible win
By the end of this lesson, you’ll be able to look at a chatbot’s answer and pinpoint whether it broke at retrieval or at generation — and that one distinction decides what you fix next.
Watch the walkthrough
1 · RAG in one sentence
Retrieval-augmented generation (RAG) works like this: the moment someone asks a question, the system pulls relevant company evidence, drops it into the model’s context, and asks the model to answer based on what’s actually there.
That’s very different from permanently teaching the model your SOPs — it only ever sees the evidence handed to it for this one request. Which means the chatbot can already have failed before the model writes a single word: it might grab the wrong document, the wrong chunk, or nothing useful at all.
In a regulated, audited environment, “sounds plausible” doesn’t count as quality. An answer only counts if it’s backed by the right evidence — and cites it.
Primary reading: OpenAI’s Optimizing LLM Accuracy guide — for now, just read the RAG section.
2 · Your actual system
Your demo harness at demo is our safe sandbox: 25 fictional Meridian Labs documents and 65 questions, six of which the system should flat-out refuse to answer.
Question
“Quel est le délai de clôture d’un plan d’action pour une non-conformité majeure ?”
1. Retrieve → Qdrant semantic chunks + Qdrant summaries + Meilisearch keyword hits + LightRAG context
2. Assemble → `qa_service.py` combines and truncates the evidence
3. Generate → a completion model writes the answer from the assembled context
4. Cite → sources are extracted from the context items actually shown to the model
5. Evaluate → `demo/eval.py` separately scores retrieval, answer correctness, citations, refusals, hallucinations, and latency
That separation is the whole foundation of this course — the demo report isn’t just a percentage score, it tells you exactly where the system is weak.
3 · Diagnose before changing code
| Observation | Likely layer | First place to inspect |
|---|---|---|
| Expected document absent from `/api/v1/search` | Retrieval | Index, chunking, query form, thresholds, ranking |
| Expected document retrieved, but not cited | Context/citation assembly | Context truncation and source extraction |
| Document cited, answer omits a required fact | Generation | Prompt, context organization, model behavior |
| Unanswerable question receives a confident answer | Grounding/safety | Refusal instruction, evidence threshold, eval set |
| Correct answer but slow response | Operations | Backend latency, graph calls, context size, concurrency |
4 · Practice: retrieve or generate?
Scenario A: The answer says “24 hours” — but `/api/v1/search` never returned `PR-QA-MRD-009` in the results. What failed first?
Scenario B: `PR-QA-MRD-009` is right there in the search results, but the answer says “60 days” while the evidence clearly says “45 days.” What broke?
Scenario C: A question with no real answer gets a detailed, confident reply — and no source in sight. Which metric should catch this?
5 · Your prototype success contract
Before you run another experiment, write these five targets down in your own words:
- Which question categories absolutely have to work first?
- What accuracy bar does the first MVP need to clear?
- What’s the minimum acceptable source hit rate?
- What hallucination rate is a dealbreaker?
- What response time will a real employee actually tolerate?
Don’t just copy the historical `demo-fixed` numbers and call them your targets. They’re a baseline for discussion, not proof the current system is production-ready.
Mission connection
Your users are QA teams, regulators, suppliers, auditors — people who don’t care how eloquent an answer sounds. They care whether it checks out against the right SOP, policy, or questionnaire. That’s why our first engineering habit is this:
When an answer is wrong, check the retrieved evidence before you touch the prompt.
Next action
Stick to the sanitized demo and run these offline checks:
cd demo
python eval.py verify
python test_scoring.pyThen open the existing report, demo-fixed.md, and find two failures: one where retrieval worked but the answer still came out partial or wrong, and one where a source never got retrieved at all.
Ask me follow-up questions about anything that’s still fuzzy — a term, a metric, a failure case. Next lesson, we trace one question all the way through the real code and evaluation harness.