RAG Formation · Lesson · Retrieval engineering
Query handling and reranking
Improve difficult queries without hiding retrieval failures behind a larger top-k.
$ lesson --status
▸ course RAG Formation
▸ lesson 08 / 16
▸ phase Retrieval engineering
▸ status Complete
● build → measure → learnStandalone lesson
Learn the full lesson and test your understanding here.
The 16-lesson RAG Formation curriculum is complete. This page contains the complete lesson content, local reference material, and instant browser exercises.
Today’s tangible win
One falsifiable retrieval experiment card: hypothesis, metric, baseline, regression check, stop condition — all spelled out.
Watch the walkthrough
1 · The skill
Pick one of q24, q46, q52, or q63 and write a single hypothesis about query normalization that could fix it — that's the technique you can actually test live against this demo stack in Section 2. Reranking and deduplication are presented separately below, and more honestly than usual: they're real code in this codebase, but neither one runs the way you'd assume from the name, and that gap is itself worth understanding before you'd trust either in production.
- Query normalization: Turn one user question into several searchable forms — the raw question itself, any document IDs buried in it, acronyms spelled out, or terms translated.
- Reranking: A real rerank service exists in this codebase (
aperag/llm/rerank/rerank_service.py,aperag/flow/runners/rerank.py) — it even encodes a documented fallback policy that graph results are "better quality" and should be prioritized first. But nothing in the live retrieval path this demo's/api/v1/searchactually hits (qa_service.py,search.py) ever calls it. Retrieval today gives you a rough first pass, full stop — there's no reordering step after it. Before trusting reranking in production, you'd want to verify: does that fallback policy match your own priorities, what latency does the rerank call add, and what happens to the answer if the rerank call itself fails? - Deduplication: Also more limited than the name implies. Today, deduplication only happens on the citations list shown to the user, matched by filename. The actual chunk text placed in the LLM's prompt context is never deduplicated — two overlapping chunks from the same document can both eat into your evidence budget even though only one citation ends up on screen.
2 · Run the evidence loop
cd demo
python3 eval.py run --mode retrieval --workspace meridian_demo --category reference_lookup --tag lesson-08-baselineWrite down what source you expected to see, whether the baseline actually surfaced it, the one query variant you tried, and how that same metric moved afterward.
Cranking up top-k just buries a retrieval failure under more noise — it doesn’t tell you what actually went wrong.
3 · Decision worksheet
| Baseline evidence | One hypothesis/change | Metric | Regression check | Bounded conclusion |
|---|---|---|---|---|
| _____ | _____ | _____ | _____ | _____ |
4 · Retrieval practice
Question: q24 has an exact procedure ID sitting right in it. What’s the first thing worth testing?
Mission connection
The baseline already tells the story: retrieval union is high, but multi-hop answers are still weak. A tightly controlled query experiment can target that exact failure mode — no need to touch the rest of the pipeline.
5 · Deliverable and next action
- Keep a copy of the command output or the report.
- Fill out the decision worksheet.
- Write down one failure case and what you’ll verify next.
One falsifiable retrieval experiment card: hypothesis, metric, baseline, regression check, stop condition — all spelled out.
Next: Lesson 9 — Context assembly and evidence budgets.
Ask me follow-up questions about the evidence, the metric, the failure case, or whatever you want to try next. I’m your teacher for this course — ask away.