Putting « public » content in your RAG : the trap question everyone gets wrong

There’s a very common reasoning error among people building their first RAG system. It looks logical on the surface. And yet it’s responsible for most of the disappointments you see in production.

Here it is: “The LLM already knows this, so there’s no point putting it in my knowledge base.”

It’s wrong. Not “a little wrong.” Fundamentally wrong, because the question was never does the model know? but do I want to control what it says?

What RAG really means, and what most people forget

A quick refresher for anyone just getting started: a RAG system (Retrieval-Augmented Generation) works in two stages. First comes a retrieval stage: it searches a vector database for the passages most relevant to the question being asked. Then comes a generation stage: the LLM produces its answer using those passages as context injected into the prompt.

The core idea is simple: ground the generation in sources that are known, dated, and under your control, rather than letting the model improvise from its parametric memory, that statistical compression of billions of web pages dating, at best, from several months before today. If you’re new to all this, I’ve published a complete guide to the RAG pipeline that walks through the seven structuring technical decisions, from chunking to reranking.

For an illustrated overview of how a RAG system works, this video covers the essentials in ten minutes:

The trouble with the question “should I include public content?” is that it assumes a document’s value inside a RAG system comes from how exclusive it is. It doesn’t.

The “noise” argument: legitimate but poorly calibrated

The strongest objection you hear is this: including generic content will pollute retrieval. When someone asks a precise question about your internal incident-management procedure, you don’t want the system surfacing a Wikipedia tutorial on ITIL.

True enough. But that’s a chunking, metadata, and reranking problem, not a problem inherent to whether the content is public or private.

A poorly configured retriever will indeed blend the essential with the incidental. The fix isn’t to thin out your knowledge base; it’s to structure what you put into it more carefully: typing your sources, filtering by namespace, a cross-encoder reranker (cross-encoder or LLM-as-judge), an adjusted similarity threshold. The tools exist. The problem is operational, not conceptual.

And for anyone worried about scaling up: the real performance problem at scale is that everything eventually starts to look alike mathematically, so the retriever can no longer tell signal from noise. Thinning out your base won’t prevent that. Working on your retrieval architecture will.

Four concrete reasons to include “already known” content

1. The exact version matters more than you think

On its own, an LLM gives you the averaged, smoothed-out version of a piece of information. The end product of training on thousands of contradictory sources, some of them dated, many of them in English.

A concrete example: say you manage furnished short-term rental apartments. The applicable tax regime (micro-BIC, the régime réel, LMNP thresholds, social contributions since the 2024 Finance Act) has changed several times in two years. Ask a bare LLM “what’s my tax allowance on furnished rentals?” and it’ll hand you something plausible and wrong. If your RAG base contains the up-to-date regulations, the official circulars, and your own annotated summary, the answer changes completely.

Is the topic “public”? Absolutely. But the model doesn’t know your version, your interpretation, your update from last month.

2. RAG reduces hallucinations even on familiar topics

It’s counterintuitive but well documented: an LLM hallucinates more on topics it believes it has mastered than on topics where it admits its own ignorance. Parametric confidence is itself a risk factor.

To see exactly why LLMs hallucinate and how RAG addresses it (watch 0:46 for the causes, 1:55 for the RAG solutions), here’s a good video:

When the RAG context is present (even if the information it contains is technically “public”), the model anchors itself to it. It no longer has to interpolate across ten thousand vaguely contradictory sources. It has three precise, dated, consistent chunks. Recent studies, including those published by the Facebook AI research team that formalized the RAG architecture in 2020, show documented hallucination reductions of between 20 and 70 percent depending on the domain, the quality of the retrieval, and the variant used: standard RAG, SELF-RAG (which evaluates itself and decides whether retrieval is even useful before answering) or CRAG (which actively corrects poorly retrieved chunks) can push that figure higher still on well-structured corpora. On topics very thoroughly covered during training, the gain sometimes drops to just 10 to 25 percent, which is still meaningful and in no way justifies going without it. To track hallucination rates by model in real time, the Vectara Hallucination Leaderboard and the AIMultiple barometer are the two most regularly updated references.

It’s also worth recalling what LLMs actually see when they read your sources: not a whole document, but a fragment truncated to a few tens of thousands of tokens, with all the structural noise that implies. Injecting your chunks straight into the context short-circuits that problem.

3. Traceability: what a bare LLM can never give you

A standalone LLM can’t cite reliably. It can pretend to source, inventing plausible URLs or attributing wording to documents that never contained it. This is one of the best-documented vectors of hallucination.

A properly configured RAG does exactly the opposite: it returns not just an answer but the source chunks along with their metadata: the URL of the official document, the page number, the version date. In production, that changes everything. The user can click, verify, and audit. They aren’t trusting a black box; they’re trusting a reference.

This is especially critical for regulatory or contractual content. If your system answers “under Article L. 145-2 of the French Commercial Code…” and can point to the exact text on Légifrance, you have a usable tool. If it generates that from memory, you have a time bomb.

4. Your vocabulary, your framing, your level of detail

This is the most underrated dimension of all. Even for a generic concept, the way you define it for your users or your team matters. And that’s precisely the central argument I make about AI as a commodity: the models have become interchangeable, and what still sets you apart is the proprietary corpus you give them to interpret.

An illustration: “what counts as a critical incident in our system?” An LLM hands you a perfectly correct ITIL definition. Your internal document says: “P0: impact on more than 500 active users OR payment API downtime OR a personal-data breach. Immediate PagerDuty escalation.” Same family of concept. A radically different answer, and it’s the one that matters.

The real cases where you can do without public content

Let’s be honest: there are situations where leaving it out is defensible.

A proof of concept or a demo where you’re testing only the technical pipeline, not the quality of the answers. No need to build an exhaustive knowledge base just to confirm that your embedding model works.

A domain entirely out of scope for your system. If your RAG is built to answer questions about your e-commerce products and a user asks for the definition of photosynthesis, you’re better off letting the LLM answer freely (or blocking the request) than polluting your base with general encyclopedic content.

Broad, exploratory monitoring, where what you want is precisely the web’s synthesis rather than your own curated perspective. In that case, direct web access through a tool-equipped LLM is a better fit than a RAG.

It’s also worth mentioning an architectural alternative that’s gaining ground in certain contexts: Cache-Augmented Generation (CAG). When your corpus fits inside the model’s context window, loading every document on each query can turn out to be simpler and more accurate than maintaining a full RAG infrastructure, without the approximations introduced by vector retrieval. CAG really shines when your entire corpus fits within the 128k to 1M tokens that the best 2026 models now offer, and when data freshness isn’t critical: latency drops, and so does operational complexity. It’s perfect for quickly testing a POC with a small static corpus (under 1M tokens) without wrestling with a vector database.

For a visual sense of the practical differences between RAG and CAG, this video sums up the strengths and limits of each approach:

Outside these cases, the “the LLM already knows it” argument is a false economy.

The real trade-off: control versus delegation

The right question to ask about any candidate corpus isn’t “is this content exclusive?” but “do I want to be responsible for what gets said on this topic?”

If the answer is yes (and in 90 percent of professional or semi-professional uses it is), then the document belongs in your base, whether it’s public, semi-public, or strictly confidential.

This control-versus-delegation trade-off comes paired with a second, more operational one: precision versus cost. Injecting five relevant chunks into the context costs less than injecting twenty approximate ones. That’s not an argument for thinning out your corpus; it’s an argument for refining your chunking and your reranker so that only what truly matters gets surfaced. A well-calibrated RAG is cheaper than a bare LLM that hallucinates and forces the user to rephrase.

RAG isn’t a vault for trade secrets. It’s a mechanism of editorial control applied to automated generation. The public/private distinction is secondary to the controlled/uncontrolled one.

What this changes in practice

For anyone actively building their pipeline, a few concrete implications:

Tag every document systematically with its type (regulatory, internal, external-reference), its validity date, and its level of authority. The LLM can lean on this metadata to weight its answer.

Don’t conflate retrieval with generation. If you’re afraid of noise, work on your reranker and your score thresholds, not the size of your base. Traditional keyword search (BM25) remains a serious alternative to pure vector retrieval on corpora full of precise identifiers (SKUs, standards references, codes).

Test against a minimal golden dataset: 15 to 20 questions representative of your real-world usage. Add or remove documents and measure the impact on perceived accuracy. That’s the only way to settle the matter empirically.

Favor surplus over deficit. A retriever that surfaces a slightly off-topic document is a reranking case to tune. A retriever that finds nothing because you over-pruned your base is an LLM left to improvise, and that’s infinitely more dangerous.

If you’d like to see a concrete RAG pipeline implemented step by step, this video tutorial is a good starting point:

Recap

Next time you hesitate to include a document because it deals with a “public” topic, ask yourself a single question: if the system says something false on this topic, is that a problem for me?

If so, add the document.