Your RAG is lying to you : why your eval metrics are (probably) wrong
I changed my chunking from 800 to 1,200 tokens, my hit@1 jumped +12%, and for thirty seconds I thought I’d found the optimum. In reality, my questions about the MSS60 were now being evaluated against the MSS54 index. The routing between corpora had been broken from the start, without anything in the metrics flagging it. A rising metric produced by a broken eval pipeline looks exactly like a rising metric produced by a genuine improvement. Three silent traps, multi-corpus routing, the absence of versioned history, and hit@k and MRR with no LLM judge, mean that most RAG teams make their decisions on numbers that measure nothing.