RAG Formation · Lesson · Foundations & measurement

Establish the MVP baseline

Run a live evaluation and read its report like an engineer, not a marketer.

$ lesson --status
▸ course RAG Formation
▸ lesson 04 / 16
▸ phase Foundations & measurement
▸ status Complete
● build → measure → learn

Standalone lesson

Learn the full lesson and test your understanding here.

The 16-lesson RAG Formation curriculum is complete. This page contains the complete lesson content, local reference material, and instant browser exercises.

Today’s tangible win

By the end of today you'll have a fresh, live baseline for the sanitized Meridian Labs system — and you'll know exactly what it proves, and just as importantly, what it doesn't.

Watch the walkthrough

1 · Safety first: use the isolated demo

Run every command from the demo directory — not from `api`:

cd demo

The demo stack is fully isolated: its own service names, ports, collections, workspaces, and data directories. It only reads the demo configuration and the one demo API key file. Never print or open the secret environment files.

Before you start, double-check:

Docker is running.
The demo API key file exists and actually has a key in it.
Port `8000` isn't already taken by the production project.

2 · Run the baseline recipe

Start with the offline checks:

python3 eval.py verify
python3 test_scoring.py

Bring up the isolated stack:

python3 setup_demo.py up

Check the status any time with:

python3 setup_demo.py status

Now index the synthetic corpus:

python3 setup_demo.py ingest

Ingestion isn't instant — the graph and summary indexes both call the LLM, so budget real time for it. If the corpus is already indexed and you want to keep it that way, use the documented options carefully. Don't casually blow away production data.

3 · Measure retrieval before answers

Run the retrieval-only evaluation first — it skips the LLM judge entirely:

python3 eval.py run \
  --mode retrieval \
  --workspace meridian_demo \
  --tag lesson-4-retrieval

This shows you exactly what each index is pulling its weight on:

Vector only
Vector plus full-text
Vector plus full-text plus summary
All indexes, including approximate graph matching

4 · Measure generated answers

Next, run the answer evaluation:

python3 eval.py run \
  --mode answer \
  --workspace meridian_demo \
  --tag lesson-4-answer

This run hits `/api/v1/qa`, checks the citations and required keywords, and lets a judge model sort each answer into `correct`, `partial`, or `incorrect`. It also tracks refusals and latency along the way.

Doing this for the first time? Limit the sample size:

python3 eval.py run \
  --mode both \
  --workspace meridian_demo \
  --limit 10 \
  --tag lesson-4-smoke

5 · Read the report like an engineer

Reports land here:

demo/reports/

Never quote a number without its definition attached. Write down:

Metric	Your value	Meaning
Retrieval union	_____	All expected sources found across the combined indexes
Strict accuracy	_____	Judge verdict is `correct`
Source hit rate	_____	Every expected source cited in the final answer
Refusal accuracy	_____	Unanswerable questions correctly declined
Hallucination rate	_____	Unanswerable questions answered anyway
Latency p50 / p95	_____	Typical and slow-tail response time

6 · Practice: baseline or overclaim?

Statement A: “On the 65-question Meridian Labs eval set, using `gpt-4o-mini`, topk 5, in this workspace, strict answer accuracy came out to 70%.” Does that count as a solid baseline statement?

Statement B: “The chatbot is 70% accurate.” Would that fly in a technical report?

Statement C: Retrieval union is high, but strict answer accuracy is low. Where do you look next?

Mission connection

A QA or regulatory team can't do anything with a number they can't interpret or reproduce. A baseline is what keeps a prototype honest — it lays bare what's working, what isn't, and whether your next change actually moved the needle.

Next action

At minimum, run both the retrieval evaluation and the answer evaluation. Then send me:

Both report tags.
The retrieval union percentage.
Strict answer accuracy.
Source hit rate.
Refusal accuracy and hallucination rate.
One failure you want to dig into first, along with its judge reason.

Ask follow-up questions anytime Docker, ingestion, the judge, or a metric doesn't make sense to you. I'll help you make sense of the live numbers before we call Lesson 4 done.

← Previous lesson Reference sheet Next lesson →