DEV Community

Cover image for Evals Deep: Beyond RAGAS, Build Your Own Eval Harness
Muhammad Saud
Muhammad Saud

Posted on Originally published at logicstack.org

Evals Deep: Beyond RAGAS, Build Your Own Eval Harness

Evals Deep: Beyond RAGAS, Build Your Own Eval Harness

Deep Part 6 settled chunking and Deep Part 7 measured retrieval, but every number in both parts came from a test set we scored by hand. That works for one article. It does not work for a system that changes every week. This part turns the hand scoring into a repeatable gate: a question set with known answers, a script that runs every retriever change against it, and a rule that blocks a deploy when a score drops. We built the harness as a single Python file using only the standard library, with no API calls, no model, and no bill. Then we ran it on 20 questions and three small systems, and the numbers taught us something RAGAS alone would never show: a completely wrong answer can score token F1 of 0.857 and perfect faithfulness at the same time.

Our position is simple. RAGAS is a good starting point and a bad finish line. It grades your system with the same kind of model that built it, it charges you per question, and it hands you one number that hides the exact failure you care about. A hand built harness costs nothing, runs in milliseconds, and shows you the failing question by name. Run the cheap harness on every change. Run the LLM judge on a sample, once a week, calibrated against human labels. Never ship on a single aggregate number.

What RAGAS actually measures

RAGAS, from Es and colleagues in 2023, proposed three reference free metrics for RAG pipelines, each scored by prompting a language model. Faithfulness splits the answer into statements and checks each one against the retrieved context, then scores supported statements over total statements. Answer relevance asks the model to generate questions the answer would fit, then measures embedding similarity between those questions and the original. Context relevance asks the model to extract the sentences from the context that are needed to answer, then scores extracted sentences over total sentences in the context. The paper tested this on WikiEval, a dataset the authors built from 50 Wikipedia pages with human judgments, using gpt-3.5-turbo-16k as the judge. If you want the hands on version of the same tool, our earlier guide walks through it step by step: Evaluate Your RAG System with RAGAS.

The agreement numbers from the paper are worth reading closely, because they are better than most people remember and worse than most people assume. On pairwise comparisons against human annotators, RAGAS agreed with humans 0.95 of the time on faithfulness, 0.78 on answer relevance, and 0.70 on context relevance. A baseline that simply asked ChatGPT to assign a score from 0 to 10 managed 0.72, 0.52, and 0.63. A baseline that asked it to rank two answers managed 0.54, 0.40, and 0.52. So the structured metrics beat naive prompting by a wide margin, and faithfulness at 0.95 is genuinely strong. But 0.70 on context relevance means the judge disagrees with humans on nearly one comparison in three, and the authors say themselves that context relevance was the hardest dimension, because the judge struggles to pick the crucial sentences out of longer contexts.

Two more facts matter before you adopt it. First, the paper is from 2023 and the library moved on. The current release, ragas 0.4.3, ships six text metrics, and the names changed: answer relevance is now response relevancy, and context precision now has a reference free path and a reference based path. Three of the six metrics need a reference answer in every documented mode, which quietly breaks the original promise of reference free evaluation. Second, every RAGAS metric needs at least one LLM call per question. Faithfulness needs two, one to extract statements and one to verify them. Answer relevancy needs one generation plus an embedding call. Context relevancy needs one extraction. That is 4 LLM calls and 1 embedding call per question, or 80 LLM calls and 20 embedding calls for a 20 question test set. That is the price of every run, and it is also the reason most teams run evals rarely, which is exactly when regressions slip through.

The problem with judge only evaluation

The first problem is cost and speed. Eighty judge calls for 20 questions means a 200 question test set needs 800 calls. At that price teams shrink the test set until it stops catching anything, or they run evals once a month instead of on every change. The second problem is correlation. Zheng and colleagues showed that GPT-4 as a judge agreed with human preferences more than 80 percent of the time, about the same rate at which human judges agreed with each other. That sounds good until you flip it: the judge disagrees with people about one time in five, and when the judge and the generator are the same family of model, their blind spots overlap. A judge that shares your generator's training will forgive the same phrasing habits and the same confident mistakes. The third problem is aggregation. A single faithfulness score of 0.92 tells you nothing about which question failed, and the one question that failed is usually the one your users will ask first.

ARES, from Saad-Falcon and colleagues at Stanford in 2023, is the honest answer to the trust question. It fine tunes lightweight judge models on synthetic data for context relevance, faithfulness, and answer relevance, then corrects those judges with prediction powered inference, a statistical technique that uses a small set of human labelled examples to put confidence intervals on the scores. Across eight knowledge intensive tasks in KILT, SuperGLUE, and AIS, it scored RAG systems accurately with only a few hundred human annotations, and the authors ask for about 150 labelled examples as the validation set. That is the standard we hold our own harness to: a number without a confidence interval and without a named failing example is a press release, not an evaluation.

The setup: one corpus, three systems, twenty questions

We built a small fictional corpus about Kiln Coffee Roasters, a made up company with 12 short passages: founding year and city, a second roastery, bag prices, cafe hours, decaf process, shipping rules, subscription price, roast levels, sourcing countries, wholesale terms, and composting. Fictional matters. A real corpus lets the model answer from memory and inflates every score. A fictional corpus forces the system to use only the retrieved text, which is the entire point of RAG evaluation. We wrote 20 questions against it, each with a short gold answer and the passage that contains it.

Then we built three answerers from the same keyword overlap retriever, so the only difference between them is the policy under test. System A retrieves the top 1 passage and answers with the best matching sentence in it. System B retrieves the top 3 passages and answers with the best matching sentence across all three. System C is System A with one invented detail appended to every answer, the words with free worldwide shipping, which appear nowhere in the corpus. System C is the trap detector: any metric that cannot tell C from A is not measuring groundedness at all. All three systems answer by copying a sentence from the corpus, which is deliberate. Extractive answers isolate the retrieval and selection logic from generation quality, so a failure points at the retriever, not at the wording.

The metrics, implemented from scratch

We implemented six metrics in plain Python, no libraries beyond the standard library. Recall at k is the share of questions where a gold passage appears in the top k retrieved. Context precision at k is the share of the retrieved passages that are gold. Exact match is 1 only when the predicted tokens equal the gold tokens exactly. Token F1 is the harmonic mean of token precision and token recall between prediction and gold. Substring accuracy is 1 when the gold answer appears inside the predicted answer. Faithfulness, our proxy for the RAGAS idea, is the share of the answer content words that appear anywhere in the retrieved context. Two of these functions carry the whole idea, and they fit in a few lines.

def token_f1(pred, gold):
p, g = set(tokens(pred)), set(tokens(gold))
common = len(p & g)
if common == 0:
return 0.0
prec, rec = common / len(p), common / len(g)
return 2 * prec * rec / (prec + rec)

def faithfulness(answer, passage_ids):
ctx_words = set(tokens(" ".join(PASSAGES[i] for i in passage_ids)))
words = content_tokens(answer)
if not words:
return 0.0
return sum(1 for w in words if w in ctx_words) / len(words)

Note what faithfulness can and cannot see. It checks whether the answer words exist in the context, not whether the sentence is true, not whether it answers the question, and not whether the right sentence was picked. That limitation is the point of the experiment. We wanted to find out how far word level checks get you, and where they stop.

The measured table

We ran all three systems over the same 20 questions. The full run took 0.010 seconds of wall time for 60 system question pairs, at zero cost and zero API calls. For comparison, scoring the same 20 questions the RAGAS way needs 80 LLM calls and 20 embedding calls. Here is what the harness reported.

SystemRecallContext precisionExact matchToken F1Substring accuracyFaithfulness
A: top 1 passage0.850.850.000.3120.801.00
B: top 3 passages1.000.330.000.3320.851.00
C: top 1 plus invented detail0.850.850.000.2250.800.70

Read the recall column with the retrieval size in mind. System A retrieves one passage, so its 0.85 is recall at 1. System B retrieves three, so its 1.00 is recall at 3. The extra context fixed every retrieval miss, and the price is visible in the next column: only one of the three passages B retrieves is gold, so context precision falls to 0.33. That is the trade Deep Part 7 already measured from the other side: more context helps recall and adds noise for the model to wade through. If you want the retrieval numbers behind this, read RAG Deep: Hybrid Search and Reranking, Measured and RAG Deep: Chunking Strategies Tested at Scale.

What the numbers actually say

Exact match is 0.00 for all three systems, and that is a finding, not a bug. The gold answers are short, like 16 dollars, and the systems answer with full sentences, like the single origin bag costs 16 dollars. Exact match punishes complete sentences, so a system that answers correctly in a full sentence scores zero. If exact match is your gate, you will reject every system that speaks in sentences. Token F1 has the same disease in milder form: 0.312 for A and 0.332 for B look like failing grades, but substring accuracy says 0.80 and 0.85 of the answers contain the gold answer verbatim. The low F1 mostly measures verbosity, not wrongness. This is why our series part on evals warned against single metric thinking in the first place: Evals: The Skill 56 Percent of Jobs Want.

Faithfulness is 1.00 for both A and B, even though both systems answer three questions wrong. Extractive answers are faithful by construction: every word comes from the context, so a word level faithfulness check passes by design. It catches invented content and nothing else. System C proves it: the same answers with one invented phrase appended drop faithfulness to 0.70, while token F1 only falls to 0.225 and substring accuracy stays at 0.80. Faithfulness is the only metric in our set that separates C from A, and that separation is exactly its job. But a faithful answer can still be the wrong sentence, and our harness shows that happening three times in 20 questions.

The three misses are worth naming, because they are three different failure modes. On how much is shipping outside the United States, the system retrieved the right passage and picked the wrong sentence from it: orders over 40 dollars ship free in the United States, when the answer was in the next sentence, international shipping costs 12 dollars per order. On how often does the single origin change, it picked the single origin bag costs 16 dollars, because keyword overlap cannot tell costs from changes. On what is the minimum wholesale order, it retrieved a passage about wholesale orders that never mentions the minimum, because the words wholesale orders matched and the number did not matter to the scorer. Retrieval found the topic. Selection picked the wrong fact. No aggregate score names these. Only the per question table does.

The adversarial probe: a wrong answer that scores 0.857

The most important number in this whole part did not come from the three systems. It came from a probe we ran after the main experiment, to test what token F1 actually rewards. We fed the harness a hand written wrong answer, Kiln Coffee was founded in 2019 in Seattle, against the gold Kiln Coffee was founded in Portland in 2019, with both the Portland and Seattle passages in the retrieved context. The result: exact match 0.00, token F1 0.857, faithfulness 1.00. Every word of the wrong answer appears in the context, and seven of the eight gold tokens appear in the answer, so F1 calls it nearly perfect. The answer is wrong on the one fact the question asked about, the city, and two of the three metrics cannot see it. The only checks that catch it are the substring check for Portland, which fails, and a human reading the sentence.

This is the core argument of this part. Token overlap metrics measure word sharing, not truth. A wrong city, a wrong year, or a wrong price can hide inside a high F1 score whenever the rest of the sentence matches. RAGAS faithfulness has the same blind spot by design: it asks whether claims can be inferred from the context, and Seattle is in the context, just attached to the wrong event. The fix is not a better aggregate metric. The fix is a test set where the wrong entity is present as a distractor, plus a human reading the five worst examples on every run. Our corpus has the distractor built in: the Seattle roastery passage exists precisely so a careless retriever can pick the wrong city, and it did, in the probe and in spirit in the wholesale miss.

The cost ledger

Here is the full ledger for scoring 20 questions three ways. Our harness ran 60 system question pairs in 0.010 seconds, with zero API calls and zero cost, on a laptop, using only the Python standard library. Scoring the same 20 questions the RAGAS way needs 80 LLM judge calls and 20 embedding calls, because faithfulness needs two calls per question, answer relevancy needs one generation plus embeddings, and context relevancy needs one extraction. That is before retries, before rate limits, and before the judge model changes version and quietly moves every score. The judge route also has a failure mode the local route does not: when the judge API is down, your deploy gate is down. A local harness runs in CI on every commit, offline, deterministically, with the same numbers every time. That determinism is what makes it a gate instead of a report.

The gate: thresholds that block a deploy

A harness earns its keep when it can say no. We set three thresholds before looking at the results, and we did not move them after: recall at 3 of at least 0.90, faithfulness of at least 0.90, and substring accuracy of at least 0.80. System B passes all three and ships. System A fails recall at 0.85 and does not ship. System C fails faithfulness at 0.70 and recall at 0.85 and does not ship. The gate is a few lines at the end of the script, and it exits nonzero on failure so CI blocks the deploy.

GATE = {"recall@3": 0.90, "faith": 0.90, "substring": 0.80}

def gate(aggregate):
failed = [m for m, t in GATE.items() if aggregate[m] < t]
return (not failed), failed

passed, failed = gate(aggregate)
print("GATE PASS" if passed else f"GATE FAIL: {failed}")
raise SystemExit(0 if passed else 1)

Three rules keep the gate honest. First, thresholds are set before the run, never after, or the gate becomes a rubber stamp. Second, the gate runs on the same frozen test set every time, and the test set only grows: new failures get added as regression questions, never edited away. Third, the gate prints the failing questions by name. A red build that says recall dropped to 0.85 is useful. A red build that says the shipping question now returns the free shipping sentence instead of the 12 dollar sentence is actionable.

Our opinionated take

Most teams we see do evaluation backwards: they buy the fanciest judge, run it once, quote the number, and never look at a single answer. That is theatre. The order that works is the reverse. First, build the cheap deterministic harness and run it on every change, because it is free and it catches retrieval and selection failures the same day you introduce them. Second, read the five worst examples on every run, because the aggregate will lie to you and the examples will not. Third, run the LLM judge weekly on a sample, not on every commit, and calibrate it the ARES way: about 150 human labelled examples, confidence intervals on the scores, and a check that the judge still agrees with people on your domain. Fourth, keep the two layers separate in your head. The harness answers did we break retrieval. The judge answers does the answer read well and stay grounded. One number cannot answer both questions, and any tool that promises it is selling you the average of two different failures.

We also think the test set matters more than the metric. Twenty hand written questions caught three real failure modes and one adversarial trap, because the corpus had distractors built in: two cities, two bag prices, two shipping sentences in one passage. A test set without distractors will give you perfect scores and teach you nothing. If your test set has never failed, it is not a test set. It is a demo.

What to do this week

Start with 20 questions on your own corpus, each with a short gold answer and the passage that contains it. Add one distractor per question type: a second city, a second price, a second date that a careless retriever could pick. Implement recall at 3, token F1, substring accuracy, and the faithfulness proxy from this part, in one file, standard library only. Run it against your current retriever and read every miss by name. Then set three thresholds before your next change, wire the script into CI with a nonzero exit on failure, and add every new failure to the test set as a regression question. After that, and only after that, run RAGAS or DeepEval on a weekly sample, and compare the judge scores against your harness scores on the same questions. Where they disagree, trust the harness until a human reads both answers.

What is Next in This Series

Deep Part 9 takes the retrieval stack into LangGraph, where state and memory decide what the agent remembers between turns. We will build a small agent that keeps working notes across turns, measure how much memory helps and how much it costs, and show where memory becomes a liability: stale facts that outlive the documents they came from. The eval harness from this part comes along, because a memory change is a retriever change, and it goes through the same gate.


Originally published on LogicStack. Run the code examples right in your browser with our free interactive Python and SQL playgrounds — no setup, no sign-up.

Top comments (0)