The Day Claude Shannon Used Human Brains as Language Models
What a 75-year-old experiment reveals about compression, prediction, and modern AI.
A CEO receives 10,000 emails a day.
One assistant forwards all 10,000.
Another sends a one-page summary:
Three customer escalations need attention.
Revenue is tracking 4% below forecast.
Legal needs approval by Friday.
Everything else can wait.
Which assistant is smarter?
Most people immediately pick the second one.
Why?
Because the second assistant compressed a mountain of information into something useful.
But hold on.
Spam filters compress information too.
Dashboards compress information.
KPIs compress information.
Even a zip file compresses information.
So what exactly impressed us about the assistant?
Was it the compression itself?
Or was it the ability to identify what mattered?
That distinction turns out to be surprisingly important.
Especially if you’re trying to understand modern AI.
Claude Shannon’s Strange Definition of Information
In 1948, Claude Shannon laid the foundations of information theory with a deceptively simple question:
What is the ultimate limit of compression?
To answer it, he did something that still feels a little unsettling.
He stripped information of meaning.
In everyday conversation, we think information means knowledge, truth, or insight.
Shannon didn’t.
To him, information was simply a measure of surprise.
A highly predictable event contains little information.
A highly unexpected event contains a lot.
If someone tells me:
The sun rose in the east this morning.
I learn almost nothing. The event was overwhelmingly expected.
If someone tells me:
A giant asteroid just landed in downtown Minneapolis.
I learn quite a bit.
Not because it’s useful.
Not because it’s true.
But because it would be wildly unexpected.
In Shannon’s world:
Information is surprise.
That single idea changed computing forever.
Because if information is surprise, then predictable things require fewer bits to describe.
And that leads to one of the most beautiful insights in computer science:
Prediction and compression are mathematically equivalent.
The better you can predict something, the more efficiently you can compress it.
The Question Shannon Couldn’t Answer
Now comes the part that fascinated me.
Suppose you want to know how compressible English really is.
How would you figure that out?
At first glance, this sounds like a statistics problem.
Count letters.
Count words.
Analyze books.
Calculate probabilities.
Done.
Except Shannon discovered something awkward.
Language isn’t just a collection of letters.
The probability of the next character depends heavily on everything that came before it.
To estimate the true compressibility of English, you need to estimate how predictable English is.
And to estimate predictability, you need a predictor.
So Shannon did something wonderfully clever.
He asked people.
In one experiment, he showed participants text and asked them to guess the next letter.
Then the next.
Then the next.
The better they could predict upcoming characters, the more compressible the language appeared to be.
Think about that for a moment.
The best language model available in 1950 was not a computer.
It was a human brain.
Shannon wasn’t merely measuring text.
He was probing intelligence itself.
Not because he was trying to build AI.
Because he was trying to measure compression.
And that’s where things start getting interesting.
Fast Forward Seventy-Five Years
Today we call it next-token prediction.
Back then Shannon called it estimating the entropy of English.
Different era.
Same underlying problem.
Modern language models perform a task that Shannon would immediately recognize:
Given everything you’ve seen so far, what comes next?
The scale is different.
The hardware is different.
The mathematics is remarkably familiar.
The better a model predicts language, the better it can compress language.
And here’s the part that makes people uncomfortable:
To become a better predictor, the model often has to discover increasingly sophisticated patterns.
Grammar.
Causality.
Human preferences.
Social conventions.
Programming languages.
Scientific concepts.
Business jargon.
The model learns compressed representations of all of these because they help prediction.
And better prediction means better compression.
Suddenly, compression and intelligence start looking suspiciously related.
But They’re Not the Same Thing
This is where many discussions go off the rails.
People notice that intelligence often produces compression and conclude:
Compression is intelligence.
That feels a bit like saying:
Footprints are walking.
A Netflix recommendation engine has a compressed representation of your viewing habits.
But the profile isn’t you.
A map is a compressed representation of a city.
But the map isn’t the city.
A financial dashboard is a compressed representation of a business.
But the dashboard isn’t the business.
Compression can preserve something important without becoming the thing itself.
That’s why I think both extremes miss the point.
The statement:
Compression is intelligence
goes too far.
But the statement:
Compression has nothing to do with intelligence
misses something profound.
Every time we try to compress something complicated—language, images, science, even our understanding of the world—we seem to be forced into building better models.
And better models often look suspiciously like what we call intelligence.
The Question Worth Asking
Maybe the interesting question isn’t:
Is compression intelligence?
Maybe the interesting question is:
Why does the pursuit of better compression keep pushing us toward increasingly intelligent behavior?
Claude Shannon was asking about compression.
He ended up needing human intelligence.
Modern AI researchers are building predictors.
They keep discovering capabilities that look increasingly intelligent.
Perhaps that isn’t a coincidence.
Compression may not be intelligence.
But it might be one of the clearest footprints intelligence leaves behind.
And footprints are worth paying attention to.
Just don’t mistake them for the traveler.
Further Reading
Claude Shannon, A Mathematical Theory of Communication (1948)
Claude Shannon, Prediction and Entropy of Printed English (1951)
David J. C. MacKay, Information Theory, Inference, and Learning Algorithms
Recommended Viewing
This article was inspired by the excellent 3Blue1Brown video:
Reinventing Entropy | Compression is Intelligence (Part 1)
If you’d like the mathematical intuition behind entropy, information, prediction, and compression—and a much deeper treatment than I could fit into a Medium article—I highly recommend watching it. It’s one of the clearest explanations I’ve seen of why a seemingly abstract theory from the 1940s ended up sitting at the heart of modern AI.