A team had built a RAG system to answer internal policy and product questions from a growing document set. Early demos looked great — the model answered fluently, in the right tone, citing what looked like the right sections. Then support staff started flagging answers that were confidently wrong: correct-sounding policy language that didn't match what the actual documents said. The root cause wasn't the language model. It was retrieval — nobody had measured whether the right passages were being pulled in the first place, so a confident wrong answer and a confident right one looked identical from the outside.
The system sounded right.
That was the problem.
The system sounded right, which was the problem.
What it was built on.
A fairly typical generative AI development setup for this scale of problem — representative of what we see across similar engagements, not necessarily the exact tools in any one project:
The source documents were internal policy pages and product documentation — the kind of content that changes incrementally over time, which matters for how chunking and re-indexing need to be handled.
Where the eval layer sits.
Treat retrieval as something you measure.
Before touching the generation side, we built a golden set — a set of real questions paired with the document sections that should have been retrieved for each. That gave us an actual number for retrieval precision and recall, instead of a vague sense that "it mostly works." From there:
- Re-tuned chunking to match how the source documents were actually structured, instead of a fixed-size splitter that was cutting policy clauses in half.
- Added hybrid search (keyword plus embedding) to catch exact-match terms the embedding model alone was missing.
- Chunk-level confidence scoring, so low-confidence retrievals surfaced a flag instead of generating an equally confident-sounding answer regardless.
- Visible citations on every answer, so a wrong retrieval was checkable instead of invisible.
Retrieval precision and recall, re-checked on every change.
The golden set became a standing regression check: every change to chunking, the embedding model, or the retrieval ranking had to be re-scored against it before shipping. Acceptance threshold was simple — retrieval precision and recall could not regress against the previous baseline, full stop. This is the same discipline we build into AI agent testing & evaluation work more broadly.
What changed once retrieval was measured.
This is the pattern of improvement we typically see once a retrieval eval layer goes in — illustrative, not a specific client's measured figures.
Permission-aware retrieval.
For internal knowledge systems specifically, retrieval has to respect the same access boundaries as the source documents — a retrieval layer that surfaces a restricted policy document to someone without clearance is a security regression even if the citation is accurate. Permission checks sit in the retrieval step itself, not bolted on after generation.
What a client walks away with.
For this class of engagement, the deliverables are the golden evaluation set, the retrieval and chunking configuration, and a standing regression check wired into the team's own deployment process — so retrieval quality stays measured after we're gone, not just at launch.
Tell us what
your RAG system is retrieving.
Book a 30-minute call — we'll tell you honestly whether retrieval is actually being measured.