What a list of numbers remembers

Why can a row of numbers made from a sentence be turned back into the sentence?

A sentence goes into an encoder and a row of, say, 768 floating-point numbers comes out. The numbers look like noise. There is no obvious word in them, no order you can read. So a very common architectural decision follows: the original text is treated as sensitive and locked down, while the vectors are shipped off to a hosted index, logged, cached, and shared, on the reasoning that they are "just embeddings".

The assumption underneath is that the transformation is one-way. It is worth asking where that belief comes from. Nobody proved it. It rests on the fact that the numbers are unreadable to us — and unreadable is not the same claim as irreversible.

Try a counting argument first, because it settles what is even possible. Take a short sentence of about 32 tokens. Each token is one choice from a vocabulary of maybe 50,000, so an upper bound on the information in it is roughly 32 times 16 bits — around 500 bits, and in practice much less, because language is redundant. Now look at the vector: 768 numbers at 32 bits each is about 24,000 bits of container.

So the vessel is dozens of times larger than the message. Is there any information-theoretic reason the sentence must be destroyed? No. Nothing about the shape of the output forces the loss. That does not prove the map is invertible, but it removes the reflexive defence — "it must be lossy, look how small it is" — because it is not small.

Then ask why the map feels one-way. Because it is hard to invert analytically. You cannot rearrange a transformer's forward pass to solve for its input, and gradient-following through a discrete token sequence is awkward. That difficulty is real. But notice what kind of difficulty it is: it is the absence of a formula, not the absence of information. And when information is present but no formula exists, there is a standard move — learn the inverse.

So consider the setup an attacker actually has. They hold a target vector. They usually also have query access to the same encoder, because it is a public or purchasable model. That second fact is the whole ballgame. It means they can check their guesses: take a candidate sentence, embed it, and compare its vector to the target. Distance in vector space becomes a scorable objective.

Now the method almost writes itself, and it is essentially what Morris and colleagues published in 2023 as Vec2Text. Train a model to propose an initial guess at the text from the embedding. Re-embed that guess. Compare the guess's vector with the target vector, and feed a corrector model both the current guess and the gap between the two vectors, so it can revise the wording in the direction that closes the gap. Repeat. Each round moves the hypothesis closer, and the loop is what does the work — the first guess alone is mediocre.

The reported results are the uncomfortable part. On 32-token inputs their system recovered the exact original text in roughly 92% of cases on one dataset, and produced near-matches otherwise — and where exactness failed, names, dates, and clinical details often survived, which is what actually matters for a privacy claim. Earlier work by Song and Raghunathan in 2020 had already shown embeddings leak substantial attribute and word-level information; the 2023 result pushed it to full reconstruction of short text.

What are the limits? Recovery degrades as the text gets longer — the vessel stops being dozens of times bigger than the message. The attack in that form assumes access to the same encoder for re-embedding, so it is strongest against widely used public models and weaker against a private one. And adding noise to the vectors reduces recovery, at a measurable cost to retrieval quality. So this is not "all embeddings are plaintext", it is "short text under a known public encoder is often recoverable, and you should assume so unless you have specifically defended against it".

Three refinements.

First, the mechanism is not decryption and it is not a lookup table. Nothing memorised the sentence. The attack succeeds by search with a scoring function, and the encoder itself supplies the score. This is why publishing the encoder is not a neutral act.

Second, the misconception worth naming: "the embedding is not the data, it is a derivative of the data." True, and irrelevant. What matters for a privacy regime is whether the personal information can be recovered, and a lossy but sufficient derivative is still personal data in any sensible reading. The practical rule is to give the vector store the same classification, access controls, retention limits, and breach obligations as the source text.

Third, an honest note on where the picture is unsettled. Defences exist — adding calibrated noise, projecting to fewer dimensions, fine-tuning the encoder to resist inversion — and they trade off against retrieval quality in ways that are still being characterised. I would not tell you there is a settled recipe. The settled part is the direction of the burden of proof: irreversibility must be demonstrated, not assumed from unreadability.

• Noise-based and dimension-reduction defences, and the retrieval quality they cost.
• Membership inference against vector stores: asking whether a specific document is in the index.
• Whether the same iterative-correction trick inverts image and audio embeddings.

Every term the collection defines is gathered in the glossary.