Your RAG is lying to you : why your eval metrics are (probably) wrong

I changed my chunking from 800 to 1,200 tokens, I reran the eval, and my hit@1 jumped +12%. For thirty seconds, I thought I’d found the optimum. For the next thirty, I realized I’d simply broken the routing between my corpora, and that my eval was now comparing German MSS60 questions against a French MSS54 index.

The +12% wasn’t an improvement. It was an artifact. A rising metric, produced by a broken eval pipeline, looks exactly like a rising metric produced by a genuine improvement. That’s precisely why no one notices.

This piece is devoted to the three silent traps that lead most RAG teams to make their decisions on metrics that measure nothing. These aren’t textbook cases: they’re the three holes I plugged in my own pipeline, on a corpus of 65 modules of automotive technical documentation (Bosch MSS54 / Siemens MSS60), after realizing I’d been lying to myself for weeks. The context is automotive, but the mechanics transpose strictly: replace MSS54/MSS60 with any case where you have two distinct technical sources, product doc v1/v2, two legal entities, two languages, two brands. The traps are the same.

RAG eval is a system, not a benchmark

If you’ve been reading the RAG literature for eighteen months, you’ve seen thousands of words go by on chunking, embeddings, hybrid search, rerankers. You’ve probably read my own RAG explained like no one else does. You’ve seen two or three public benchmarks, MTEB chief among them, and you may have drawn the idea that evaluating a RAG is like evaluating a model: you take a dataset, you compute a score, you conclude.

That shortcut is the root of the problem. The evaluation of a retrieval-augmented system isn’t a score, it’s an infrastructure. It’s a pipeline that must itself be versioned, monitored and debugged. A RAG metric without its eval context is a thermometer you read without knowing where it’s stuck.

The three traps that follow all share the same structure: they produce numbers that look correct, but that measure something other than what you think you’re measuring. They are perfectly invisible as long as you don’t go looking for them.

The invisible prerequisite: manufacturing the ground truth

Everything that follows assumes an already-existing questions.yml file. In reality, manufacturing it is often the real bottleneck of a RAG project, the one no one talks about because it’s less glamorous than a reranker.

My workflow is nothing original but cost me a few mistakes before reaching its current form. For each module of the corpus, I pass the source markdown to Claude Sonnet with an instruction asking for five precise technical questions, their reference answer, and three to five keywords expected in a good answer. Output in structured YAML. Then, a non-negotiable step, I manually review each entry, I reword the ones that are poorly framed, I correct the reference answers (often too optimistic or too general), I adjust the keywords. Of my 255 questions, about 30% were modified after generation, and a dozen added by hand to cover cases the LLM hadn’t seen, typically the cross-corpus questions that require knowing both ECUs.

The major trap is to let the LLM generate the question, the answer, and then judge its own generation. In that configuration, you’re measuring the model’s internal consistency, not the quality of your RAG. Any score obtained is strictly decorative. Human validation on the reference answers is the irreducible minimum, it’s what anchors the eval in a truth external to the model being evaluated.

Beyond this first wave, questions.yml has to live. When a real production query puts the system in difficulty, it enters the file after annotation. When a module is added to the corpus, it comes with its five questions. A frozen eval dataset that never grows lies more and more over time, exactly as a software test suite that’s no longer enriched drifts in coverage without any red light coming on.

Trap #1: Multi-corpus routing, or how to evaluate MSS54 against the MSS60 index without realizing it

As long as you have a single corpus, you’re safe. That’s the overwhelming majority of RAG tutorials: a folder of PDFs, an index, a retrieval script, done.

The problem begins at the exact moment you have two distinct corpora. For me, it was the moment when, after ingesting 39 modules of the Siemens MSS60 documentation (the ECU of the BMW M3 E92 and M5 E60), I added a second corpus for the Bosch MSS54 (M3 E46, S54 et al). Two corpora, two indexes, two technical universes that talk about related but not identical engines.

Here’s what happens by default, in 99% of the RAG pipelines I’ve seen in production: your eval script loads questions.yml (255 questions in my case), builds a vector client, and fires the queries. All the queries. Against the same index. The one that was instantiated when the script started.

Result: your MSS54 questions are evaluated against the MSS60 index. The retriever does its job, returns chunks, the LLM generates an answer, the judge assigns a score. Everything works. Except that the returned content has nothing to do with the question asked, it simply comes from the wrong corpus. And since the LLM is polite, it will answer something coherent on the surface, which gets a mediocre but not catastrophic score. Your metrics drop slightly, you conclude that your chunking is suboptimal, you spend two weeks iterating on the chunking.

You spent two weeks iterating on the wrong problem.

The solution is conceptually simple, but it requires rewriting the eval loop: each question must be evaluated against its native index, identified by its doc_id. In my code, that’s _build_doc_id_config_map() in eval/run_eval.py, a function that scans ./documents/ at startup, builds a routing table doc_id → module config.yml, and is consulted for each question. Without it, my metrics were weighted averages of comparisons that made no sense.

The extension that complicates the picture further: cross-corpus questions. A question of the type “what are the differences between the torque-management strategy on the MSS54 and on the MSS60?” has a reference doc_id, but it legitimately needs to cross both indexes. In my server (serve.py), it’s _COMPARE_RE that detects the pattern and activates hint_corpus_all = True. The eval has to reproduce this behavior, otherwise it unfairly penalizes the comparison questions, which are precisely the ones where the RAG shows its true value.

The trap is insidious because it scales linearly with the number of corpora. With two corpora, about 50% of your questions are routed correctly by chance. With ten corpora, it’s 10%. And no one sees it, because the metrics keep coming out, keep moving, keep giving the illusion of measuring something.

Trap #2: No history = no measurable progress

Once the routing is fixed, you breathe a little. The numbers are finally “true.” So you can iterate on the chunking, compare the embedding models, adjust the top-k. Except that’s when you discover a second lie, more subtle: you have no basis for comparison.

You change your chunking from 800 to 1,200 tokens. You rerun run_eval.py. You see faithfulness = 7.4 / completeness = 6.8. Better than last time? You have no idea, because last time is nowhere. It’s in your terminal window, already closed, or in a screenshot you pasted into Notion three weeks ago. You iterate blind.

This is the moment when most RAG teams build, in a hurry and badly, their own historization system. A CSV that grows, a shared Google Sheet, a save_run_to_db() function grafted onto the end of the script. Everyone rediscovers the same thing after six months: you need an append-only, versioned storage system that archives every run with its full context (embedding model, chunking, generation model, date, retriever configuration), and that automatically computes the diff with the previous run.

In my case, that’s eval_history.jsonl: an append-only file, one line per run, containing all the aggregated metrics plus the per-question detail. On each new execution, run_eval.py compares against the last archived results and displays a “regressions / improvements” diff: which questions were good and turned bad, and vice versa. Not a global average, but a list of concrete cases to investigate.

The side effect, and it’s the one that convinced me: a run that abruptly degrades keyword coverage on a specific module almost always signals that a chunk has “vanished” into the vector noise. It’s exactly the mechanism of semantic collapse: the bigger the chunks get (or the more the corpus expands), the closer their embeddings come to one another in vector space, the discriminating signal dilutes into the context, and the retriever ends up no longer telling the relevant chunk apart from the merely plausible one. Without history, this phenomenon is invisible until an end user reports back that “the bot can’t find the info on the lambda sensors anymore.” With history, the diff tells you before production.

A metric without its history measures nothing. It only tells you where you are, not whether you’re making progress. And in a system where every commit can affect retrieval quality, measuring only the position is condemning yourself to sail without a rudder.

Three real diffs I’d never have seen without this history

To make all of this concrete, here are three diffs actually surfaced by my eval pipeline in recent weeks. None would have been visible in a “one run, one score” approach.

Diff 1: The silent chunking regression (Apr 30 run vs Apr 24 run). After changing my chunk_size from 800 to 1,200 tokens, the global hit@1 went from 0.67 to 0.60. Seven points down, unpleasant but not catastrophic, and a hurried mind would have seen it as mere measurement noise. The per-question diff, though, tells a precise story: EVT-01 (CAN transmission delay for the intake-opening angle) goes from rr=1.0 to rr=0.5, MM-01 (Momentenmanagement) loses 13 points of keyword coverage. Both questions touch on short, dense numerical tables. By enlarging the chunks, I had drowned the discriminating signal in verbose context, exactly the mechanism described in Semantic collapse. Without the per-question diff, I’d probably have attributed the drop to “a bit of variance” and persisted in the wrong direction.

Diff 2: The massive gain from the multi-corpus routing fix (May 9 run vs May 5 run). After wiring _build_doc_id_config_map() into the eval loop, the MSS54 hit@1 jumps from 0.73 to 0.80, the faithfulness goes from 7.43 to 8.07. These aren’t marginal variations: they’re seven points of hit@1 recovered by fixing a single plumbing function. The lesson is brutal: for weeks, my “score” was measuring a blend of real quality and routing noise. Everything I’d optimized in the meantime rested on poorly framed comparisons. A single question remains resistant after this fix, EDKSI-01 (kinematics of the electronic throttle), with a faithfulness that climbs (6→7) but a hit_1 still at false. It’s now the only case I know I have to investigate manually, instead of fumbling around globally.

Diff 3: The asymmetry between corpora (MSS60 run of May 7 vs MSS54 run of May 5). At identical configuration (hybrid + rerank + top_k=5), MSS60 caps out at faithfulness=6.25 / completeness=5.83, against 7.43 / 6.91 on MSS54. The two questions that sink (DTH-03 on lambda-sensor diagnosis and LFR-01 on fuel-pressure regulation) point to information scattered across several MSS60 modules, typical of less consolidated source documentation. Without the history, I might have believed my pipeline had regressed. With the history, I know it isn’t my pipeline that’s the problem, it’s the MSS60 corpus that needs deeper editorial work on the markdown cross-references. The diagnosis changes completely.

None of these three discoveries comes out of an aggregated score. All three come out of the per-question delta between two successive runs. That’s exactly what eval_history.jsonl makes possible and what “one score, one point” approaches make impossible.

Trap #3: Hit@k and MRR without an LLM-as-judge measure the wrong thing

The first two traps concern the eval infrastructure. The third concerns what we evaluate. And here we reach the heart of the misunderstanding.

Almost all public RAG benchmarks measure hit@k (is the right document in the first k results) and MRR (where is it, on average). These metrics come from the world of classic information retrieval, where one assumes there’s a “canonical answer” identified by a doc_id and that the system’s task is to find it.

The problem: a RAG isn’t a search engine. It’s a system that generates a natural-language answer from retrieved documents. The relevant question isn’t “is the right chunk in the top-3,” it’s “is the final answer correct.” And these two questions aren’t equivalent.

A system can reach hit@1 = 0.95 and produce wrong answers 30% of the time, because the LLM ignores the right chunk in favor of a bad one. Conversely, a system with hit@1 = 0.6 can produce correct answers 90% of the time, because the LLM knows how to compose a correct answer from three moderately relevant chunks.

Hence the necessity of a dual criterion measured directly on the generated answer:

• Faithfulness: is the answer faithful to the retrieved chunks? (Detects hallucinations.)
• Completeness: does the answer properly cover the question asked? (Detects correct but partial answers.)

In my case, these two scores are assigned by Haiku (Claude Haiku 4.5), on a scale of 0 to 10 for each. The judge’s prompt receives the question, the reference answer (from questions.yml), the answer generated by the system, and the chunks used. It returns two integers and a short justification.

On the cost side, over 255 questions, about $0.15 per full run. At a realistic pace of fifteen to twenty runs per month in an active iteration phase, you’re looking at $2 to $3/month. Turn on Anthropic’s prompt caching on top, on the judge’s system prompt (which doesn’t change between questions), and you divide that by three again. There’s no economic excuse for going without it.

The usual technical objection: “the LLM judge is biased.” Yes. But (a) it’s biased in a stable way, which keeps relative comparisons between runs valid, and (b) it’s less biased than a tired human annotating 255 questions on a Friday night. The LLM judge isn’t the truth, it’s a reproducible proxy, which is exactly what you ask of a measurement tool.

Cases where you can do without it: during local chunking iteration, the lightweight eval (no judge) is enough. In my case, that’s what POST /eval/run does in serve.py, fast metrics via SSE, with no LLM cost, for short feedback loops. Cases where it’s mandatory: before any merge to main, before any deployment, and systematically when you change a major component (embedding, retriever, generation model).

“And what about Ragas, LangSmith, DeepEval in all this?”

A legitimate question, and one to address head-on: these tools exist, they’re serious, and yet they solve none of the three traps described above. Not because they’re bad, but because they answer an adjacent problem.

Ragas provides a catalog of well-thought-out RAG metrics (faithfulness, context_precision, context_recall, answer_relevancy) with their built-in LLM judge prompts. That’s valuable: it saves you from writing your own eval prompts. But Ragas knows nothing about your corpus → index topology. If you hand it a dataset of 255 questions and a retriever, it will compute the metrics on whatever the retriever returns. If the retriever queries the wrong index, Ragas will very conscientiously compute metrics on irrelevant chunks. Trap #1 is upstream of Ragas, not inside it.

LangSmith is excellent for tracing: seeing, on each call, which chunks were retrieved, which prompt was sent to the LLM, which answer came back. It’s the tool you want for debugging. But LangSmith doesn’t force you to version your runs append-only with their full context (embedding model, chunking, retriever config). It can do it if you instrument correctly, but the default is to aggregate by session, not by reproducible eval run. Trap #2 isn’t solved by default, it depends on your instrumentation discipline.

DeepEval, Promptfoo, TruLens: same categories. Good tools for observability and unit metrics, not designed to handle multi-corpus routing nor to automatically compute the regressions/improvements diff between two successive runs.

The general rule: these frameworks provide the bricks (LLM judges, traces, metrics). The eval pipeline, that is, the way the bricks are assembled to produce numbers you can make a decision on, remains your responsibility. That’s precisely what the three preceding sections describe. You can perfectly well implement the doc_id → index routing, the append-only history and the per-question diff using Ragas as the LLM judge. It’s even probably what I’d do if I had to rewrite my code today.

The architecture that lies less

Assembling the three corrections, you get an eval pipeline whose general shape is the following:

The three colored zones correspond exactly to the three traps handled above: the routing (red) repairs trap #1, the append-only + diff (blue) repairs trap #2, the LLM judge (orange) repairs trap #3. If any one of the three is missing, the rest produces numbers that lie.

Step by step:

1. Loading questions.yml, a YAML file containing for each question: the prompt, the reference answer, the target doc_id, and the keywords to check.
2. Building the doc_id → config routing table at startup, by scanning ./documents/. This is the guarantee against trap #1.
3. Parallel RAG generation (8 workers in my case): each question is embedded with input_type=query, the retriever queries the index corresponding to its doc_id, the LLM generates the answer. The hybrid BM25 + vector mode with Cohere rerank is applied systematically.
4. Evaluation by LLM judge (Haiku) on the two faithfulness/completeness axes.
5. Computing the aggregated metrics: hit@1, MRR, KW coverage, average faithfulness, average completeness.
6. Append-only archiving in eval_history.jsonl with full context (models, chunking, date, duration).
7. Diff with the previous run: regressions and improvements, listed by question.
8. CSV and HTML export for manual analysis.

The code is roughly 800 lines, and it’s probably the most useful component of my whole pipeline. More useful than my chunking, more useful than my embedding choice, more useful than my reranker. Because it’s the thing that tells me whether all the rest is doing its job.

What this approach still doesn’t capture

Intellectual honesty demands noting what we continue not to measure, even with this pipeline.

The judge’s bias. Haiku can hallucinate its judgments, overrate certain answer styles, underrate laconic but correct answers. The countermeasure: a manual audit of a sample (10% of the questions) at each release, to make sure the judge isn’t drifting. It’s slow, but it’s the only real safeguard.

The absence of ground truth on multi-step reasoning. If the right answer requires combining three chunks from three different modules, the judge can’t say whether the combinatorial reasoning was correct, only whether the conclusion was. It’s a limit intrinsic to any evaluation by final output.

Model drift. The same eval run on Claude Sonnet 4.6 and Claude Sonnet 4.7 doesn’t give the same scores, even with the same embeddings and the same chunking. This means the judge itself has to be versioned in eval_history.jsonl, and that an apparent regression can be an effect of the judge, not of the system being evaluated. It’s a meta-problem that probably deserves an entire article.

The representativeness of questions.yml. You only ever evaluate what you’ve put in the dataset. If your users’ real questions in production are structurally different (vocabulary, length, intent), all the eval effort is off target. And there’s an aggravating factor: the quality of the corpus upstream. If your PDFs are badly parsed and arrive in the index as typographic mush, no eval metric, however sophisticated, will make up for it. The countermeasure: periodically collect a sample of real production queries and inject it into questions.yml after annotation, and audit the parsing quality in parallel.

RAG evaluation isn’t a metric, it’s a measurement system

We’d like the evaluation of a RAG system to look like that of a classification model: a dataset, a score, a decision. This analogy is dangerous because it masks the fact that a RAG system is a composite pipeline in which every stage can introduce its own errors, and whose evaluation must therefore itself be a pipeline.

If, for each metric you look at, you don’t know:

• which index the query was actually routed against,
• what the score was on the previous run and how much it moved,
• whether the generated answer was judged by a human or by a stable LLM proxy,

then you’re not measuring the quality of your RAG. You’re manufacturing conviction. And conviction without measurement, in a system where every commit can affect the semantic quality of retrieval, is exactly what produces the RAGs that collapse silently in production, six months after the initial deployment.

The bad reflex is to answer this observation with “it’s too complicated, we’ll do it later.” The good one is to realize that the eval infrastructure isn’t a luxury for a mature team: it’s the minimum foundation for the rest to make sense. Built early, it spares you weeks of blind iteration. Built late, it reveals months of false convictions.

Fifteen dollars for six months of serious iteration in LLM-judge calls, one append-only jsonl file, one function that scans your corpora at startup. It isn’t a revolution. It’s simply the cost of ceasing to lie to yourself.

To go further on the adjacent traps: Semantic collapse, From RAG to CAG, The embedding model no one really chooses and Turning a complex PDF into truly RAG-ready markdown.