Fixing a RAG pipeline
that hallucinated confidently.

A team had built a RAG system to answer internal policy and product questions from a growing document set. Early demos looked great — the model answered fluently, in the right tone, citing what looked like the right sections. Then support staff started flagging answers that were confidently wrong: correct-sounding policy language that didn't match what the actual documents said.

The root cause wasn't the language model. It was retrieval. The vector search was returning chunks that were semantically similar to the question but not actually the right source — close enough that the model treated them as ground truth and extrapolated a plausible answer on top. There was no layer checking whether retrieval had actually found the right thing before generation ran.

Before touching the generation side, we built a golden set — a set of real questions paired with the document sections that should have been retrieved for each. That gave us an actual number for retrieval precision and recall, instead of a vague sense that "it mostly works."

From there:

• Re-tuned chunking to match how the source documents were actually structured, instead of a fixed-size splitter that was cutting policy clauses in half.
• Added chunk-level confidence scoring, so low-confidence retrievals either triggered a fallback or were flagged rather than silently feeding the model.
• Added a groundedness check after generation — comparing the answer against what was actually retrieved, to catch the model asserting something the source never said.
• Made the eval set permanent, re-run on every change to chunking, embeddings, or prompts — so retrieval quality became a tracked number, not a one-time check.

This is a pattern-level breakdown of what our RAG development work is built to prevent. If your system is giving fluent answers you can't fully trust, it's worth checking whether retrieval quality is actually being measured — not just eyeballed.