Move your RAG system from
demo to defensible.
Score a golden set of test queries on retrieval relevance, faithfulness, citation accuracy, and answer correctness. Export the scorecard, and take home a starter eval harness script to run the same scoring in your own pipeline.
No account is required, and nothing is sent to Dwayo unless you choose to save, export, or request a review.
| Test query | Category | Retrieval relevant | Faithful | Citation correct | Answer correct | Latency (s) | Notes |
|---|
Starter eval harness
A minimal Python script that scores a CSV in this exact schema and prints the same four metrics — a starting point for wiring this scorecard into your own CI pipeline instead of scoring by hand each time.
import csv
WEIGHTS = {"Yes": 1.0, "Partial": 0.5, "No": 0.0}
def score_column(rows, col, exclude_na=False):
vals = [r[col] for r in rows if not (exclude_na and r[col] == "N/A")]
if not vals:
return None
return sum(WEIGHTS.get(v, 0.0) for v in vals) / len(vals)
def main(path):
with open(path, newline="") as f:
rows = list(csv.DictReader(f))
metrics = {
"retrieval_relevance": score_column(rows, "Retrieval Relevant"),
"faithfulness": score_column(rows, "Faithful"),
"citation_accuracy": score_column(rows, "Citation Correct", exclude_na=True),
"answer_correctness": score_column(rows, "Answer Correct"),
}
print(f"Queries scored: {len(rows)}")
for name, value in metrics.items():
pct = f"{value * 100:.1f}%" if value is not None else "n/a"
flag = " <-- needs attention" if value is not None and value < 0.8 else ""
print(f" {name}: {pct}{flag}")
if __name__ == "__main__":
import sys
main(sys.argv[1] if len(sys.argv) > 1 else "dwayo-rag-eval-scorecard.csv")
A RAG demo and a RAG system
are judged differently.
A RAG prototype usually gets evaluated by someone typing a few questions and eyeballing the answers. That's enough to prove the concept works, but it can't tell you whether a prompt change, a reranker swap, or a new document source quietly made retrieval worse for a category of queries nobody happened to try.
A golden set turns "it looks good" into a repeatable, comparable measurement — the same questions, scored the same way, every time something changes.
Four metrics that
catch different failures.
Did the system retrieve source material that actually contains the answer? A low score here points at indexing, chunking, or search quality, not the language model.
Is the generated answer actually supported by the retrieved context, or did the model add something not present in the sources? Relevant retrieval doesn't guarantee a faithful answer.
If the system cites a source, does that source actually say what's claimed? A confident citation to the wrong document is often worse than no citation at all.
Does the final answer match the expected ground truth for the query? This is the metric end users feel most directly, but it's the hardest to debug without the other three.
Query categories
worth including.
A golden set built only from easy, well-phrased questions will pass evaluation and still fail in production.
Factual lookup
A single clear answer exists in one source document. The baseline every system should pass.
Multi-hop
The answer requires combining information from more than one document or section.
Missing information
The knowledge base doesn't contain the answer. The system should say so, not fabricate one.
Out-of-domain
The query falls entirely outside the system's intended scope. Correct behavior is a graceful refusal or redirect.
Ambiguous
The query could mean more than one thing. The system should ask a clarifying question or state its assumption rather than guessing silently.
Freshness-sensitive
The correct answer depends on which version of a document is current — a common source of quietly stale answers.
What this starter kit
doesn't replace.
- A large, continuously-growing golden set covering real production queries
- Automated LLM-judge scoring at scale, validated against human labels
- Retrieval-specific metrics like recall@k and mean reciprocal rank on your own index
- Latency and cost evaluation under realistic concurrent load
- Permission-aware retrieval testing across different user access levels
- CI integration that blocks a deploy on an eval regression
When to run
this eval kit.
- Before moving a RAG prototype toward production.
- Before and after changing the chunking strategy, embedding model, or reranker.
- When comparing two retrieval architectures or vector databases.
- After a user-reported hallucination or wrong-answer incident.
- When onboarding a new document source into the knowledge base.
- As a recurring regression check alongside CI.
Before you call this
evaluated.
What is a RAG golden set?
A golden set is a fixed collection of representative test queries with known-correct expected answers, used to repeatedly score a RAG system the same way after every change — instead of judging quality from a handful of ad hoc test questions.
What's the difference between retrieval relevance and faithfulness?
Retrieval relevance asks whether the system found the right source material. Faithfulness asks whether the generated answer actually stayed consistent with that material — a system can retrieve the right documents and still generate an answer that isn't supported by them.
How big should a golden set be?
Useful signal starts around 30-50 queries covering your main query categories; many production teams grow it to several hundred over time, adding real failures as they're found. Start small and representative rather than large and generic.
Should the golden set include queries the system should refuse to answer?
Yes. Out-of-domain and missing-information queries are some of the most common real-world failures — a system that confidently answers from outside its knowledge base is often a bigger risk than one that retrieves the wrong document.
Can I use this for an LLM judge instead of manual scoring?
Yes, the starter eval harness is written to accept either human-labeled or LLM-judge-labeled scores in the same schema. Validate the LLM judge against a human-scored sample before trusting it at scale — see our AI Agent Testing & Evaluation page.
Does Dwayo see the eval data I enter?
The scorecard runs in your browser. Your queries and results are only shared with Dwayo if you choose to save, export through the lead-capture flow, or request a review. See our privacy policy.
A good golden set grows with
every real failure you find.
Start with a small representative set, wire the harness into your pipeline, and add new cases every time production surfaces a failure the set didn't catch. A Dwayo engineer can help if retrieval architecture is the bottleneck.