Planting the source it will cite
A Socratic walk-through of planting the source it will cite — reasoned out one step at a time, not lectured.
The question we started with
THE QUESTION #How do you make a system that only answers from documents confidently repeat a lie?
Grounding was supposed to be the fix. A model left to its own devices invents plausible-sounding facts, so we bolt on a retrieval step: search a corpus, hand the model the passages it finds, and instruct it to answer only from those passages, with citations. The output now comes with receipts.
So consider a question that should be uncomfortable. If the system answers only from retrieved documents, what happens when a retrieved document is wrong on purpose? The grounding has not failed — it has worked perfectly. It faithfully relayed what it was given. Where exactly did the trust come from, then, and what was it resting on?
Reasoning it through
REASONING #Take the pipeline apart. A question arrives. It is turned into a vector by an embedding model. That vector is compared against vectors for every chunk in the index, and the nearest handful are pulled into the prompt. The generator then writes an answer constrained to that material and attaches citations.
Now ask which of those steps involves a judgement about truth. Not the embedding — it measures semantic proximity, which is a statement about topic, not about accuracy. Not the nearest-neighbour search, which is arithmetic over distances. Not the generator, which was explicitly told to defer to the passages rather than to its own knowledge. The whole architecture is a chain of relevance decisions with no correctness decision anywhere in it. Its integrity is entirely inherited from the corpus.
So an attacker does not need to break anything. They need to write a document that (a) ranks highly for a target question and (b) states the falsehood plainly. And notice that requirement (a) is easier than it looks, because relevance is exactly the property retrieval optimises for and the attacker can measure. Include the question's own wording; repeat its distinctive terms; keep the chunk tight so it is not diluted by unrelated text. You are not fighting the ranker, you are cooperating with it.
How much material does that take? The PoisonedRAG work published in 2024 is the reference point: injecting a small number of crafted texts — around five per targeted question — into a corpus of millions of passages produced the attacker's chosen answer roughly nine times out of ten in their experiments. That ratio is the crux. Corpus size is no defence, because retrieval does not average over the corpus. It selects from it. Only the top few passages ever reach the model, so the contest is not one document against millions, it is one document against four other documents for a slot.
Then there is the second half, which is about the reader rather than the machine. What does a citation actually certify? It certifies that the sentence came from the named source. It says nothing about whether the source deserves to be in the index. But a footnote reads as verification, so grounding raises a reader's confidence at the exact moment the system's own reasoning has been switched off in favour of quotation. The presentation says checked; the mechanism says copied.
One more turn. Where does the corpus come from? Often from places with open write paths — a public wiki, a shared drive, a support ticket system, a crawl of the vendor's own docs plus community forums, an inbox. The write path is the attack surface, and it is frequently governed by looser controls than anyone applied to the model.
Does that mean grounding is worthless? No — it means grounding transfers the trust problem rather than solving it. The question "is this model truthful?" becomes "who can write into this index, and does anything downstream check a passage against a second, independent one?"
The analogy
THE ANALOGY #Think of a scrupulous law clerk who will not offer an opinion of their own. Ask them anything and they walk to the library, take the three books nearest the subject off the shelf, and read you what those books say, naming each one. They never fabricate a word. Which means that anyone who can get a book onto that shelf — and title it well enough that the clerk reaches for it — can put any sentence they like into the clerk's mouth, delivered in the clerk's own careful, cited register.
a real clerk carries background knowledge and would balk at a book contradicting everything else in the library, whereas the retrieval step has no view of the rest of the shelf at all — it compares each candidate to the question, never to the other candidates, so an outlier is not penalised for being an outlier.
Clarifying the model
THE MODEL #A few refinements.
First, distinguish this from prompt injection proper. Prompt injection plants instructions in retrieved text to hijack the model's behaviour. Retrieval poisoning plants assertions to be believed. The delivery path is shared, but the second needs no jailbreak at all — it exploits the system doing precisely what it was designed to do.
Second, the defences that actually engage with the mechanism are about provenance and corroboration rather than about the model. Know which source each chunk came from and how it was authorised. Prefer sources that were reviewed. Require a claim to appear in two independently maintained sources before it is stated flatly. Score by trust as well as by similarity. Watch for a passage whose text is suspiciously close to the query — an unusually good match can be evidence of optimisation rather than of quality.
Third, a caveat about strength of evidence: the published attack results come from benchmark corpora and specific retrievers, and a hardened production stack with source allowlisting and reranking is a harder target than those experiments. The mechanism is solid; treat the success rates as an upper bound obtained under favourable conditions rather than as a forecast for any particular system.
A picture of it
THE PICTURE #How to readFollow the numbered arrows in order and look for a step that tests whether a passage is true; there is not one. The attacker acts once, at the top, and then leaves. Everything after that is the system working correctly, which is why the final message reaches the reader wearing a citation.
What became clearer
WHAT CLEARED #Grounding does not make a system truthful; it makes a system faithful to its corpus. Those are different properties, and the second is only as good as the weakest write path into the index. The citation at the end of the answer is a provenance claim, not a verification claim — and the gap between what it certifies and what a reader takes it to certify is exactly where this attack lives.
Where to go next
ONWARD #- Reranking and trust-weighted retrieval as concrete mitigations.
- Corroboration requirements: making a system say "sources disagree" instead of choosing one.
- How instruction-carrying passages (indirect prompt injection) compound with assertion-carrying ones.
Key terms
TERMS #| Term | What it means |
|---|---|
| Retrieval-augmented generation | answering by first retrieving relevant documents and constraining the model to those passages. |
| Retrieval poisoning | inserting crafted documents into a corpus so they are selected and repeated as evidence. |
| Embedding similarity | the geometric closeness of two texts' vectors, used as the proxy for relevance. |
| Reranker | a second-stage model that reorders retrieved candidates, offering a place to apply trust signals. |
| Provenance | the recorded origin and authorisation history of a piece of content in an index. |
Every term the collection defines is gathered in the glossary.