Fallacies of Machine Understanding
The Chinese Room and its descendants
A man sits in a room with a rulebook. Slips of paper covered in Chinese characters come in through one slot; he looks up the symbols in the book, follows its instructions, and pushes appropriate responses out through another slot. He does not understand a word of Chinese. To the native speakers outside, the room converses fluently. Does anything in this scenario understand Chinese?
John Searle built this room in 1980 and answered no: the man is just manipulating symbols, and since he is doing everything the computer does, no purely computational system — however sophisticated — can genuinely understand anything. The argument became one of the most famous thought experiments in philosophy and the ancestral form of a whole family of objections to machine minds. Concepts aren’t vectors, so language models can’t really reason. Intelligence must be water-based, so silicon can’t really think. Training a model on text is really just copying it, so the machine is a plagiarism engine with extra steps.
These four arguments span half a century, three disciplines, and at least one federal courtroom, and they are the same argument. Each one inspects what a system is made of — a man with a rulebook, arrays of numbers, dry silicon, other people’s sentences — finds the ingredient unimpressive, and concludes that the system’s performance must be counterfeit. Each mistakes what a thing is made of for what it does. This chapter takes them in turn, because each version fails in an instructive way, and the instruction is cumulative: by the fourth, the shared anatomy is impossible to miss.
The Man Is Not the Mind
Start with the room, and with the strongest reply to it — the one Searle heard immediately and never adequately answered. The system reply points out that the man was never the candidate for understanding. He is a component. The claim under examination is that the system — man, rulebook, room, and the process running across them — understands Chinese. Searle directs our attention to the component, invites us to notice that it lacks the system’s properties, and presents this as a discovery.
But of course the component lacks the system’s properties. That is what it means to be a component. No single neuron in your brain understands English; the conclusion that brains therefore cannot understand English is not a bold insight but an obvious blunder, and it is structurally identical to Searle’s. Understanding, memory, and perception are properties that emerge from the interaction of parts that individually have none of them — that is the only way such properties ever exist in a physical universe. Demanding that understanding show up inside a part is a misplaced level of analysis, the analytic equivalent of dismantling a clock to look for the time.
Searle’s rejoinder — let the man memorize the rulebook, internalize the whole system, and still he understands nothing — only relocates the mistake. Now the system runs on the man rather than around him, and the question is whether that implemented system understands Chinese, not whether its host does. The man’s sincere protest that he doesn’t understand a word settles nothing; he is reporting on the wrong process.
What keeps the argument alive despite this is not logic but intuition: the felt certainty that mere symbol manipulation couldn’t really be understanding, whatever the behavior looks like. And here the room teaches its second lesson: intuition is not evidence. Our intuitions about “real understanding” were trained on a sample of exactly one kind of mind, and they track biology, familiarity, and warmth, not function. If a system consistently behaves as though it understands — answers, explains, extends, corrects, applies its knowledge in novel contexts — then the insistence that this is not genuine understanding needs some criterion beyond the insister’s discomfort. None is ever supplied. “Real” becomes an honorific that biological systems award themselves.
That points at the room’s foundation, which is a substrate bias Searle never states as a premise because stated plainly it would beg the question: only biological brains can host genuine understanding. This assumption flatly contradicts functionalism — the view that cognitive states are defined by their functional and informational roles rather than by their material implementation — and Searle offers no argument against functionalism, only the room, which assumes its falsity. His later writings made the circle explicit: what began as an argument about intentionality — about how symbols come to mean anything — drifted into the claim that meaning requires biological consciousness, substrate bias graduated from hidden premise to official doctrine. On the question the room originally raised, this book has an actual answer: meaning is a triadic relation that exists wherever an interpretant takes a sign to stand for an object, an account I developed in The Origin of Meaning — and nothing in that account checks what the interpretant is made of.
So the room fails on every load-bearing point. It looks for understanding in a component when understanding is a system property; it substitutes intuition for criteria; and beneath both, it assumes the biological conclusion it claims to demonstrate. What survives is only the pattern — inspect the ingredient, dismiss the performance — and the pattern has descendants.
Concepts Are Not Vectors
The modern descendant wears mathematical dress. The critique runs: concepts aren’t vectors — concepts don’t in general add, subtract, or scalar-multiply — and this prevents language models, which represent everything as vectors, from ever achieving general intelligence.
At first glance this seems not just reasonable but rigorous. Human concepts — justice, freedom, irony — are intricate and context-saturated. They are plainly not algebraic objects. You cannot multiply mercy by three.
But look at the inference. It moves from concepts are not intrinsically numeric to concepts cannot be adequately represented numerically, and that step is a fallacy — the same fallacy, in a new costume. It assumes that a representation must share the intrinsic nature of what it represents. I argued at the foundation of this volume that this is precisely what a model does not have to do: a model need not resemble its domain, be made of the same stuff, or share its structure beyond the relations that matter for the task. The London Underground map falsifies every distance in the city and is a superb model of its connectivity. Ask of a representation not what it is, but what mapping it computes and whether that mapping is adequate for the purpose at hand.
Music makes the point unanswerable. Music is emotional, cultural, embodied, and subjective — as far from an algebraic entity as anything human beings make. Melodies do not literally add or subtract. Yet computers compose, edit, and perform music every day, because melody, harmony, rhythm, and even expressive nuance can be represented as numeric data adequately for the work at hand. Nobody concludes from the fact that music isn’t numbers that digital audio is impossible; the counterexamples are on every device you own.
So the relevant question was never “are concepts intrinsically vectors?” — of course they aren’t. It is “can concepts be adequately represented as vectors to support the reasoning we care about?” And that is an empirical question with a rapidly accumulating answer. Embedding spaces already carry analogy, metaphor, and semantic relationship well enough to support reasoning tasks that were confidently declared impossible a decade ago. The adequacy of a representation is decided by performance, not by ontology. Vector representations may well have real limits — they may need to be supplemented with symbolic, causal, or hierarchical structure, and the causal question in particular gets its own chapter. But those limits will be discovered at the level of what the representations can do, not deduced from what concepts are.
The Wet Chauvinist
The third version comes from an unexpected quarter: not a philosopher of mind but a distinguished biologist. Denis Noble holds that true intelligence must be water-based. “Real intelligence isn’t just fast computation — it’s fluid, flexible and fuelled by randomness,” he argues. “That’s why all living organisms are water-based. Water is a virtually unlimited source of the random motion which drives creativity, consciousness, and thought.”
It is a poetic claim, and there is something right buried in it. Randomness does matter for creativity, because creativity is at bottom an evolutionary process: the generation of variation and the selective retention of what works. A system with no source of variation cannot explore; Noble is right that a purely deterministic lookup of the already-known is not the whole of intelligence.
Everything else in the argument confuses the requirement with one particular way of meeting it. Grant that intelligence needs randomness: nothing follows about water, because thermal noise in water is not a privileged source of randomness — it is one source among an embarrassment of cheap alternatives. Quantum fluctuations, electronic shot noise, radioactive decay, atmospheric turbulence; one internet infrastructure company famously seeds its cryptographic randomness from a wall of lava lamps. Digital systems harvest high-grade physical randomness routinely, because randomness is one of the least scarce commodities in the universe. The property Noble correctly identifies as necessary is trivially substrate-independent; only by conflating the property with his favorite carrier of it does the argument seem to go through.
Which is the room again, wetter. Intelligence is substrate-independent computation — information processing, decision-making, adaptive pattern recognition — and none of these operations specifies a medium, any more than arithmetic specifies whether it is done on fingers, beads, or transistors. Silicon-based systems are practically universal computers, able in principle to run any computable process given the resources; no theoretical result reserves the computations underlying intelligence for wet chemistry. Noble’s inference — terrestrial life is water-based, terrestrial life is intelligent, therefore intelligence requires water — is an extrapolation from a sample of one lineage on one planet, made while machines built from dry sand were already exhibiting the flexible, creative, adaptive behavior water was supposed to monopolize. Biological chauvinism does not become rigorous by being expressed in the vocabulary of biology.
The Machine as Photocopier
The fourth version has left the seminar room for the courtroom, where the fallacy acquires a docket number. The claim: training a language model on copyrighted works is unauthorized copying — the model is a storage-and-retrieval device stuffed with other people’s property, and every fluent paragraph it emits is laundered reproduction.
The confusion here is between two acts that law and common sense have always distinguished: copying and learning. Copyright exists to prevent unauthorized reproduction and distribution — to protect creators from unfair competition by exact or near-exact duplicates of their work. It does not, and could not coherently, prohibit learning from protected works. A student reads a textbook and internalizes its concepts. A critic watches a film and writes an original analysis. An engineer studies patented designs to improve on them. Every one of these people has extracted value from protected material without paying for anything beyond access, and none has infringed, because internalizing ideas, facts, and patterns — and producing new, transformative work from them — is exactly what the system is designed to permit. Copyright restricts duplication, not understanding.
Training a model is the same kind of act. A language model statistically encodes the relationships and patterns running through its training data — generalized structure, not archived text. It does not store the works, and it does not reproduce them; feeding a book into a training run is conceptually far closer to a student reading it than to a photocopier duplicating it. Both the student and the model come away changed — holding abstracted knowledge that will surface later in new combinations — and neither comes away holding a copy. The law has already worked through the nearest precedents: Authors Guild v. Google affirmed that scanning millions of books to build a searchable index was transformative fair use, and indexing is a far shallower transformation than the wholesale abstraction training performs.
Notice why the mistake feels compelling anyway: people intuitively model an AI system as a storage device — a very large hard drive with a chat interface — because storage is what computers have always been for. On that model, whatever went in must be in there, and whatever comes out must have been retrieved. But the storage model is simply the wrong model of what training does, and the copyright panic is what happens when the wrong model meets the law. It is the Chinese Room’s move inverted: where Searle looked at the mechanism and refused to credit the learning, the copyright maximalist looks at the learning and insists it must really be mechanism — mere retrieval, mere copying. Both refuse to believe that statistical machinery can do what students do.
That is the conceptual half of the copyright question, and it is the only half that belongs in this chapter. Whether copyright itself — the whole apparatus of manufactured scarcity around ideas — survives contact with machines that learn is a question about economics and enforcement, and I take it up in The End of Intellectual Property. Here the point is narrower: even granting copyright its full traditional force, training is learning, and learning was never the thing it forbade.
One Fallacy, Four Costumes
Line them up. The man in the room doesn’t understand Chinese, so the system can’t understand. Vectors aren’t concepts, so the model can’t reason. Silicon isn’t wet, so the machine can’t think. The training data was someone’s property, so the output must be someone’s property regurgitated. In every case the argument peers into the system, finds a constituent that lacks the interesting property — comprehension, conceptual richness, vitality, originality — and concludes the system must lack it too.
This is essentialism about minds, and it has the same defect essentialism always has: it looks for a property in the ingredients when the property lives in the organization. No ingredient of any understander has ever understood anything. Neurons don’t grasp meanings; sodium ions carry no semantics; the water Noble venerates is just as blankly mechanical as the silicon he disdains. If the presence of uncomprehending components disproved comprehension, the theorem would apply to us first. Every mind that has ever existed is a counterexample to the inference these arguments share.
The remedy is the discipline this volume started with: ask of any system not what it is made of but what it does — what mapping it computes, over what domain, to what standard of adequacy. That question has answers, and the answers are empirical, which is exactly why the essentialist arguments avoid asking it. Once it is asked, the interesting problems come into focus, because the real limits of these systems — and they have real limits — are functional too, discoverable by testing performance rather than by auditing ingredients. The sharpest of those tests is causal reasoning: Judea Pearl argued on principled grounds that statistical learners can never climb from correlation to intervention to counterfactual, and that claim — unlike the four in this chapter — engages the machine at the right level of analysis. It deserves, and gets, the next chapter.