Redundancy of written text

Why can a sentence survive losing half its letters, yet resist being compressed below a fixed size?

Take a sentence, delete every vowel, and hand it over: th qck brwn fx jmps vr th lzy dg. Most readers get it. Delete letters at random instead and a surprising fraction still comes through. Written English seems to be carrying far more letters than it strictly needs, which invites an obvious conclusion: the letters are mostly filler, so squeeze them out and the text becomes tiny.

But that is not what happens. Compress that sentence with any lossless compressor and it shrinks — to roughly a quarter of its size, if the compressor is good and the text is long enough — and then it stops. Push harder and nothing more comes out. So the same text is simultaneously wasteful enough to survive losing half its symbols and stubborn enough to refuse compression past a fixed floor. Both facts have to be explained by the same thing.

Begin by asking what a letter of English is actually worth. If all 26 letters plus a space were equally likely and independent, each one would carry about log2(27), roughly 4.76 bits. That is the ceiling, and it is a fiction — nobody writes by rolling a 27-sided die.

So relax the fiction one step. Letters are not equally likely: e and t are common, q and z are rare. Counting single-letter frequencies alone brings the figure to about 4.03 bits. Now relax it again: letters depend on their neighbours. After q comes u almost without exception; after th the next letter is very likely e. Shannon's pair and triple statistics put it near 3.3 and then roughly 3.1 bits per letter.

Notice what is happening. Each time we let context in, the number falls. So push context to its limit and ask a different question — not "how often does this letter occur" but "how well can a fluent reader guess it?" That is exactly the experiment Shannon ran in 1951: show a person a passage up to some point, have them guess the next character, and record how many guesses it took. From those guess statistics he derived bounds, and with about a hundred characters of preceding context the estimate landed somewhere near 0.6 to 1.3 bits per letter. Later work betting on text — Cover and King in 1978 — put it around 1.25 bits. The exact value is still argued over, and it depends on the text and the guesser, but the order of magnitude is not in doubt.

Now both halves of the puzzle fall out of that one number. If a letter carries roughly one bit where it could carry nearly five, then about three quarters of the written symbols are, in the information sense, not telling you anything you could not have inferred. That is the deletion result: you can knock out a large share of the characters because the surviving context already predicts them. Vowels go first not by accident but because English vowels are the most predictable class given the consonant skeleton — which is why abjad scripts get away with leaving them out.

But the same number is the floor. If a hundred letters of typical English genuinely carry about a hundred bits of unpredictable content, no encoding can represent them in fewer. This is Shannon's source coding theorem: the average code length cannot fall below the entropy of the source, and any scheme that claims otherwise is either losing information or has smuggled the missing content into the decoder. Ask the sharp version of it — could a clever enough compressor beat the floor? — and the answer is that a compressor which is better at predicting English simply has a better model, which is to say it has revised the entropy estimate downwards, not defeated it. Modern neural language models compress text further than gzip precisely because they predict it better, and they too stop somewhere.

The word "redundancy" is doing precise work here, and it is easy to hear it as "waste." It is not waste; it is the error protection. A channel that carried English at exactly its entropy would be maximally compact and maximally brittle — every corrupted symbol would be an unrecoverable loss, because nothing else in the message would imply it. Natural language pays about a 75% overhead and buys back the ability to be read in bad handwriting, heard across a noisy room, or reconstructed after a smudge.

Two clarifications worth keeping. First, entropy is a property of a source and a model of it, not of a string. "The entropy of English" is shorthand for the entropy of a particular text sample under a particular predictor, which is why published figures vary and why the honest statement is a range rather than a constant. Second, the floor applies to lossless coding. Lossy schemes — abbreviating, summarising, deleting vowels and hoping — can go below it, at the cost that the original is no longer exactly recoverable, only usually guessable.

• How Shannon's guessing experiment actually derives upper and lower bounds from guess-number statistics, and why the two bounds are so far apart.
• Whether large language models' compression rates constitute a better estimate of English entropy, and what that would mean for the claim that entropy is model-relative.
• The same trade in engineered systems: error-correcting codes deliberately re-adding redundancy after compression has removed it.

Every term the collection defines is gathered in the glossary.