Chunking and document structure

Watch the walkthrough

1 · The smallest useful model

• Chunk/window: a bounded slice of source text. Handled well, it carries a fact together with its heading and document identity. Handled badly, it splits them apart.
• Overlap: the text repeated across the boundary into the next window. It can rescue a fact that straddles a boundary — but push it too far and you just end up with near-duplicate evidence.
• Chunk size is a real design choice, not one universal setting: Qdrant and Meilisearch both index at ~500 tokens per chunk, while LightRAG deliberately indexes at ~1200 tokens — a bigger window gives its entity-extraction LLM more context per call, meaning fewer, cheaper calls at the cost of coarser entity boundaries. There's no single "right" window size; the right size depends on what the index is for.
• Evidence-and-identity co-location: the metric this lesson runs on — does a source window hold both the expected document reference and every required fact term?
• Bounded conclusion: a source-window result gives you a chunking hypothesis, nothing more. It doesn't prove the live index retrieves or answers any better.

Primary reading: LlamaIndex — Sentence Window Retrieval. Watch for how a retrieved window can restore local context around a fact without losing track of which source it came from.

2 · Run one controlled source-window experiment

The offline chunk_lab.py utility only touches two things: the synthetic Markdown corpus and eval_set.yaml. It never contacts Docker, reads a secret, calls the API, or reindexes anything.

cd demo
python3 chunk_lab.py --question q24 --profile chunk --output-dir reports/lesson-06-chunk-profile
python3 chunk_lab.py --question q24 --profile graph --output-dir reports/lesson-06-graph-profile

--profile chunk and --profile graph are presets, not arbitrary numbers — they map to the real ~500-token (Qdrant/Meilisearch) and ~1200-token (LightRAG) window sizes from Section 1, translated to characters. Overlap is held steady at 200 characters across both; the only thing you're changing is window size. Then compare these report files:

reports/lesson-06-chunk-profile/lesson-06-q24-chunks.md
reports/lesson-06-graph-profile/lesson-06-q24-chunks.md

The metric here is simple: how many windows — and what rate — contain both the expected reference and every must_include term. A higher number can back up a chunking hypothesis. It is not retrieval accuracy. Expect the graph profile's rate to look better almost by construction — a big enough window trivially fits the whole document, which is exactly the tradeoff to name in your conclusion, not treat as a win.

3 · Inspect three question shapes

After q24, run the same experiment for q30 and q52. Together the three give you distinct failure shapes: a revision lookup, an answer that spans two documents, and a factual-recovery target.

4 · Record the before/after evidence

Fill in this sentence frame: “With the ___ profile (___ characters, ___ overlap), ___ of ___ windows put the required fact and the source identity in the same place. That supports the hypothesis that ___. It doesn't prove live retrieval quality, and it doesn't mean the bigger window is simply better — next I'd weigh that against LightRAG's real extraction cost and reindex-plus-retrieval check.”

5 · Retrieval practice

6 · Deliverable and next action

1. Save the two reports from your controlled q24 comparison.
2. Fill in the evidence table for q24 plus one of q30 or q52.
3. Write down one bounded conclusion, one regression risk, and the reindex-plus-retrieval check you'd run next.

Next: Lesson 7 — Metadata, filters, and access boundaries. Hang on to this record — it tells you whether a retrieval problem actually starts upstream of the search backends.

Got questions? Ask me. About the reports, the proxy metric, a boundary effect, or what to run next live — I'm your teacher for this course.