RAG Formation · Lesson · Reliability & product readiness
Latency, cost, and observability
Balance answer quality with response time, token cost, graph calls, and operational visibility.
$ lesson --status
▸ course RAG Formation
▸ lesson 14 / 16
▸ phase Reliability & product readiness
▸ status Complete
● build → measure → learnStandalone lesson
Learn the full lesson and test your understanding here.
The 16-lesson RAG Formation curriculum is complete. This page contains the complete lesson content, local reference material, and instant browser exercises.
Today’s tangible win
By the end of this lesson: an MVP performance budget plus a telemetry checklist, with every line tagged measured or assumed.
Watch the walkthrough
1 · The skill
Pull up a report and pin down which budgets — latency, backend calls, context size, model calls — are actually measured, and which ones you’re just assuming.
- p50 and p95: your median response time and your slow-tail response time. Track both — a snappy average can hide a handful of requests that crawl. One gotcha specific to this harness: p95 is only computed for answer-mode runs — a retrieval-only run reports p50 alone (see Section 2).
- Cost drivers: embedding calls, graph LLM calls, answer generation, judge calls, and how much context you’re stuffing in.
- The one real lever you can pull:
--graph-topk(default 20). LightRAG does LLM-based keyword extraction plus multiple graph traversals per query, so lowering it is a genuine, measurable trade of graph-signal depth against latency. There is no way to disable LightRAG entirely on a live request in this codebase — don't go looking for a cost toggle that isn't there. - Observability: structured traces, metrics, and logs detailed enough to explain what happened — without leaking secrets or document contents.
2 · Run the evidence loop
cd demo
python3 eval.py run --mode both --workspace meridian_demo --tag lesson-14-latencyThis is the same eval_set.yaml baseline you first ran in Lesson 4 — compare against that result, not a fresh unfamiliar number. Use --mode both, not --mode retrieval alone: the aggregate retrieval summary only ever reports latency_p50_s; p95 exists only in the answer summary, which needs answer mode to run at all. Capture, per question and in aggregate: results[].answer.stats.total_context_chars, results[].answer.stats.graph_context_chars (retrieval mode's own field for the same idea is named differently — results[].retrieval.counts.graph_chars — the two modes compute and label it independently, that's not a harness bug), summary.answer.latency_p50_s / latency_p95_s, and summary.retrieval.latency_p50_s. Then try lowering --graph-topk (default 20) on a second run and compare — never log secrets or raw sensitive content.
Remember: a good-looking average latency can hide a slow tail bad enough to sink a pilot.
3 · Decision worksheet
| Baseline evidence | One hypothesis/change | Metric | Regression check | Bounded conclusion |
|---|---|---|---|---|
| _____ | _____ | _____ | _____ | _____ |
4 · Retrieval practice
Question: Your median looks great, but p95 is dragging. What do you actually report?
Mission connection
A pilot team doesn’t just need correct answers — they need to know the system won’t bankrupt them, won’t make them wait, and won’t turn into a black box the moment quality slips.
5 · Deliverable and next action
- Save the command output or the report itself.
- Fill out the decision worksheet.
- Write down one failure case and one next thing to verify.
The deliverable: an MVP performance budget and telemetry checklist, each item tagged measured or assumed.
Next: Lesson 15 — Security, permissions, and deployment boundaries.
Ask me follow-up questions about the evidence, the metric, the failure case, or what you’d try next. I’m your instructor for this course.