01The demo problem
A first index built from a clean export always looks impressive. Then pricing changes, a policy is rewritten, three PDFs are superseded, and the assistant keeps answering from last quarter with total confidence.
Accuracy is not a launch metric. It is a maintenance property, and it decays unless something keeps the index synchronised with the source of truth.
02Chunking and metadata do the heavy lifting
Splitting documents on arbitrary character counts destroys meaning. Split on structure — sections, clauses, product records — and attach metadata: source, effective date, language, audience.
That metadata is what lets you filter before you retrieve, so an Arabic customer asking about a UAE policy never gets an English draft from 2024.
03Refusal is a feature
A system that answers everything is a system that invents things. Set a relevance floor: below it, the assistant says it does not know and offers a human.
Log every refusal. That log is the highest-value backlog you will get — it tells you exactly which content is missing.
04Evaluate on every change
Keep a fixed set of real questions with expected answers and citations. Run it whenever the model, prompt, chunking or corpus changes, and treat a regression as a build failure.