Case Study — RAG Development

The system sounded right.
That was the problem.

Illustrative scenario. This is a pattern-level write-up of a recurring failure mode and its fix — not a write-up of one named client project. Stack, numbers, and timeline are representative of this class of engagement, not a specific client's exact figures.
Pattern: Internal knowledge tooling Illustrative timeline: ~5 weeks Related: RAG Development

The system sounded right, which was the problem.

A team had built a RAG system to answer internal policy and product questions from a growing document set. Early demos looked great — the model answered fluently, in the right tone, citing what looked like the right sections. Then support staff started flagging answers that were confidently wrong: correct-sounding policy language that didn't match what the actual documents said. The root cause wasn't the language model. It was retrieval — nobody had measured whether the right passages were being pulled in the first place, so a confident wrong answer and a confident right one looked identical from the outside.

What it was built on.

A fairly typical generative AI development setup for this scale of problem — representative of what we see across similar engagements, not necessarily the exact tools in any one project:

Vector database (embeddings-based retrieval) Production LLM API Chunked document ingestion pipeline No retrieval eval layer No groundedness check

The source documents were internal policy pages and product documentation — the kind of content that changes incrementally over time, which matters for how chunking and re-indexing need to be handled.

Where the eval layer sits.

Document ingestion
→
Chunking
→
Retrieval (scored)
→
Generation + citations
→
Confidence flag

Treat retrieval as something you measure.

Before touching the generation side, we built a golden set — a set of real questions paired with the document sections that should have been retrieved for each. That gave us an actual number for retrieval precision and recall, instead of a vague sense that "it mostly works." From there:

  • Re-tuned chunking to match how the source documents were actually structured, instead of a fixed-size splitter that was cutting policy clauses in half.
  • Added hybrid search (keyword plus embedding) to catch exact-match terms the embedding model alone was missing.
  • Chunk-level confidence scoring, so low-confidence retrievals surfaced a flag instead of generating an equally confident-sounding answer regardless.
  • Visible citations on every answer, so a wrong retrieval was checkable instead of invisible.

Retrieval precision and recall, re-checked on every change.

The golden set became a standing regression check: every change to chunking, the embedding model, or the retrieval ranking had to be re-scored against it before shipping. Acceptance threshold was simple — retrieval precision and recall could not regress against the previous baseline, full stop. This is the same discipline we build into AI agent testing & evaluation work more broadly.

What changed once retrieval was measured.

This is the pattern of improvement we typically see once a retrieval eval layer goes in — illustrative, not a specific client's measured figures.

Before
Confident wrong answers, no way to measure how often
After
Retrieval precision tracked as a number, re-checked on every change
Ongoing
Low-confidence answers flagged instead of surfaced silently

Permission-aware retrieval.

For internal knowledge systems specifically, retrieval has to respect the same access boundaries as the source documents — a retrieval layer that surfaces a restricted policy document to someone without clearance is a security regression even if the citation is accurate. Permission checks sit in the retrieval step itself, not bolted on after generation.

What a client walks away with.

For this class of engagement, the deliverables are the golden evaluation set, the retrieval and chunking configuration, and a standing regression check wired into the team's own deployment process — so retrieval quality stays measured after we're gone, not just at launch.

If this looks familiar.

This is a pattern-level breakdown of what our RAG development work is built to prevent. If your system is giving fluent answers you can't fully trust, it's worth checking whether retrieval quality is actually being measured — not just eyeballed. See also document AI for the extraction side of unstructured content, and our AI development services for how a fix like this is scoped and delivered.

Get started

Tell us what
your RAG system is retrieving.

Book a 30-minute call — we'll tell you honestly whether retrieval is actually being measured.