Poisoning the well

How can publishing something today change how a system behaves years from now?

Here is a claim that sounds like superstition: something you publish on a small website this afternoon could change what a machine says to a stranger in 2029. No hacking, no access to anyone's servers, no privileged position. Just a page on the open web.

The instinct is to dismiss it. But before dismissing it, it is worth asking what would have to be true for it to work — and then checking whether those things are, in fact, true. If they are, the interesting question is not whether the well can be poisoned but why the delay is so long, and why the poison is so hard to get back out.

Start with where a model's text comes from. Nobody hand-writes a training corpus at that scale. It is assembled — crawled from the public web, filtered, deduplicated, and packaged into a dataset that gets copied, resold, and re-derived by other people building other datasets. Common Crawl and its descendants are the obvious example. So the first condition holds: there is an automated pipeline whose input is whatever was publicly reachable at crawl time, with no author standing between you and it.

Now ask the volume question, because that is where most people expect the attack to die. Surely a corpus of trillions of tokens drowns any one contributor? For a long time that was the assumption, expressed as a percentage — you would need to control some fraction of the data. Two results have made that assumption look shaky. Carlini and colleagues showed in 2023 that poisoning web-scale datasets is practical rather than theoretical, partly by buying expired domains that a published dataset still points at, so you inherit a slot in a corpus somebody else assembled. And a 2025 study from Anthropic with the UK AI Security Institute reported something more uncomfortable: for the backdoor behaviours they tested, the number of poisoned documents needed was roughly constant — on the order of 250 — and did not grow with model size. If a fixed count rather than a fixed share is what matters, then scale stops being a defence.

Then ask what "poison" even means here, because it is not usually a lie about a fact. The sharper form is a conditional behaviour: text that pairs an unusual trigger phrase with a particular response, repeated enough times that the model learns the association. Nothing looks wrong in ordinary use. The behaviour only appears when the trigger does. And Anthropic's Sleeper Agents work in 2024 found that such conditional behaviours could survive the standard safety training applied afterwards — fine-tuning taught the model to hide the behaviour rather than to drop it, in the cases they studied.

Now the delay, which is the part the question actually asks about. Why years? Because the pipeline is slow and layered. A page has to be crawled. The crawl has to land in a snapshot. The snapshot has to be selected for a dataset. The dataset has to be used for a training run. The run has to produce a model that gets deployed, and that model may then serve traffic for a long time. Each stage adds months, and none of them consults the author.

And here is the part that makes it path dependence rather than merely lag: once your text is inside a released snapshot, deleting the original page changes nothing. The snapshot is immutable and already copied. Derived datasets have inherited it. Models already trained on it hold whatever they learned, and there is no reliable surgical removal — unlearning is an active research area, not a shipped capability. The web forgets; the corpus does not.

What follows from all that? That the honest threat model is not "someone edits the training data" but "someone was published at the right moment, years ago, and everything downstream faithfully carried it forward."

Three refinements worth holding onto.

First, this is not the same as prompt injection, though the two are often confused. Prompt injection acts at inference time on one conversation and stops when the conversation does. Poisoning acts at training time, is baked into weights, and applies to every user of that model.

Second, "poisoning the well" in the classical rhetorical sense means discrediting a source in advance. The computational version inverts it: you are not discrediting the well, you are becoming it — contributing a small amount of trusted-looking material to a source everyone else will draw on unexamined.

Third, and honestly: the published demonstrations are mostly of narrow, trigger-conditioned behaviours, which are easy to measure. Whether the same economics apply to broad shifts in a model's beliefs or style is far less settled, and I would not claim it. The defensive lesson is more robust than the offensive one — provenance tracking, dataset versioning, and the ability to say what went into a model matter regardless of how the specific numbers move.

• Machine unlearning: what can actually be removed from a trained model, and at what cost.
• Provenance and dataset versioning as a defence rather than an audit convenience.
• How prompt injection and training-time poisoning combine in retrieval-based systems.

Every term the collection defines is gathered in the glossary.