Model collapse

Why does a model trained on the output of earlier models get steadily worse instead of compounding their skill?

Here is a hope with a respectable pedigree. A model learns the patterns in human writing. It writes. A successor trains on that writing, which is cleaner and more on-distribution than the raw web, and should therefore learn faster. Repeat, and each generation stands on the last — compounding, as in every other place where knowledge is written down and inherited.

What happens instead, when the loop is run in the laboratory, is that the descendants get worse, and worse in a particular way. So the interesting question is not "why is synthetic data bad", because sometimes it is excellent. It is: what exactly does a model lose when its teacher is a model, and why is that loss cumulative?

Begin with something that has nothing to do with machine learning. Take a large population, sample a thousand of them, then sample a thousand from that sample, and keep going. What disappears first? The rare types. A trait held by one person in ten thousand is probably absent from your first sample, and once absent it can never return, because every later generation is drawn only from what survived. Sampling is a one-way ratchet against the tails.

Now notice that training a model on another model's output is exactly this. A finite training set is a finite sample from the teacher's distribution. Rare constructions, unusual facts, minority dialects, odd but valid phrasings — each has some chance of simply not appearing, and what does not appear cannot be learned. Do it again and the surviving distribution is narrower again. This is what the Nature paper on recursively generated data calls statistical approximation error, and its striking claim is that it alone suffices: even a perfect learner with a perfectly expressive model collapses, purely from finite sampling. I am recalling that framing rather than deriving it, so treat the attribution as the softer part.

Two further errors ride along. The model family may not represent the true distribution exactly, and the training procedure — objective, optimiser, regularisation — introduces its own bias. Both are small per generation, and both are inherited and re-amplified.

Then a mechanism that is easy to miss and, I think, does a great deal of the work. Generation is not faithful sampling from what the model learned. We do not want faithful output, we want good output, so we generate with reduced temperature or truncate to the most likely few tokens — and every one of those knobs deliberately deletes the tail before a word is written. The corpus a successor trains on is narrower than its parent's own beliefs, on purpose, by a margin its builders chose for quality.

Now test the account against what is observed. If the story were simply "errors compound", descendants should get noisier — more diffuse, wronger in more directions. That is not the reported signature. Late-generation models become less varied: repetitive, bland, converging on a few modes, confidently generating things the original data never contained. Shrinking variance is the fingerprint of tail loss, not of accumulating noise, and that discriminating observation rules the vaguer account out.

Let us try to falsify it properly. If recursion were intrinsically poisonous, distillation should fail — yet training a small model on a large one's outputs works, and works well. And if generational degradation were unavoidable, then simply keeping the original data around should not help. Follow-up work in 2024 found that it does: if each generation accumulates real and synthetic data rather than replacing the old corpus, the degradation is bounded rather than compounding. That is the crucial control, and it relocates the cause. The culprit is not the presence of synthetic data. It is the discarding of the real data — closing the loop.

Which suggests the general condition. A recursive loop is safe when something outside it constrains it: a distilled student is anchored by a teacher stronger than itself, generated code by the unit tests it must pass, generated proofs by a prover that is not a language model. Where such an anchor exists, synthetic data is among the most useful things available. Where the only judge is another model trained on similar output, there is nothing left to correct toward.

It is worth separating this from two nearby failures. Overfitting is a model memorising a fixed sample instead of the pattern behind it: the data is fine and the fitting is wrong. Collapse is the reverse — the fitting may be impeccable while the data itself narrows underneath it. And a recommender's narrowing runs through a loop in the outside world, where items never shown gather no evidence, so exposure decides what is learnable. Model collapse needs no users and no exposure; the loop closes entirely inside the sampling.

The honest caveat is about scope. Nearly all the clear demonstrations are closed-loop laboratory setups replacing the corpus wholesale at every generation, which is not how anyone actually trains. Real pipelines mix, filter, deduplicate, weight by quality, and retain enormous archives of pre-existing human text. Whether the real web is degrading measurably from contamination is genuinely unsettled, and the strong public version of the claim outruns the evidence. What is solid is the mechanism and the condition under which it bites.

There is also a distributional point with teeth. Because the tails go first, collapse is not a uniform dulling: it falls hardest on whatever was rare — minority dialects, unusual subject matter, uncommon but correct constructions. A model can look barely changed on average benchmarks while having quietly stopped representing the edges of its own domain.

• Why provenance metadata and watermarking are being pursued as infrastructure rather than features.
• How reinforcement learning from a verifier differs, formally, from training on a model's own samples.

Every term the collection defines is gathered in the glossary.