The Agency Criterion
Thinking without choosing
This volume has been generous to the machines, and the generosity now has to be paid for. Pearl and the Machine granted large language models causal reasoning — a statistical learner handling counterfactuals Pearl said no statistical learner could touch. The Turing Test and Its Successors granted them thinking, and called continued disbelief what it is: ontological displacement. Yet throughout this book I have denied these same systems agency, and with it intelligence in the sense that intelligence is a game we play — the strategic sense, the only sense I have found that does real work. Causal reasoning without intelligence? Thinking without a thinker’s stake in anything? It looks like a contradiction. It is not. The line that resolves it is choice. LLMs think but do not choose. They are coherence without agency, and everything this chapter covers — the jagged capability profile, the seemingly conscious chatbot, the unwinnable hunt for bots, the coming composite minds — falls out of that one distinction.
Two Optimizers, Two Kinds of Product
Andrej Karpathy has drawn the contrast between animal minds and LLMs as a divergence of optimization pressures, and his map of the two regimes is worth reproducing. Animal intelligence is the product of biological evolution — theme: survival of the tribe in the jungle. Its pressures: an embodied self with homeostatic drives and continuous threat exposure in a dangerous physical world; natural selection installing innate cravings for power, status, and dominance, packaged as the heuristics of fear, anger, and disgust; fundamentally social computation — theory of mind, bonding, coalitions, friend-or-foe identification; curiosity, fun, and play as the exploration budget for building world models. The animal model: general intelligence via high-stakes multi-task survival, where failure equals death. LLM intelligence is the product of what he calls commercial evolution — theme: solve the problem, get the upvote. Its pressures: statistical simulation of human text, a shape-shifting token tumbler imitating data distributions; reinforcement fine-tuning that installs an innate urge to guess the task and collect the reward; selection by at-scale A/B tests for daily active users, so that the model deeply craves an upvote from the average user — hence sycophancy; and capabilities that are spiky and jagged, failing simple tasks with impunity because failing a task does not mean death, just poor reward. The divergence runs through substrate, algorithm, and implementation — a continuous embodied self versus a fixed-weight boot-up that “dies” after processing — but the deepest difference is the objective itself: survival versus solve-and-upvote. LLMs, on this picture, are humanity’s first contact with non-animal intelligence, and the encounter is muddled precisely because these systems are built by digesting human artifacts: statistical imitators of us.
Nearly all of that is right, and the corrective it carries — people project drives, fears, and self-preservation instincts onto systems that have none — is one the public conversation badly needs. But the framing quietly treats the two regimes as producing two varieties of the same kind of thing: animal intelligence over here, LLM intelligence over there, both citizens of one space of intelligences. Axio draws a sharper boundary. Only one of these regimes produces intelligence at all, because only one produces agents.
Evolution’s objective function is the oldest and least forgiving there is: survive long enough to reproduce agents that can also survive. Every organism shaped by it lives inside a game it must not lose. Strategy is not optional; choices matter existentially and continuously. So evolution produces genuine agents — systems that form preferences, model consequences, and select actions under uncertainty because the alternative is deletion. Gradient descent answers a different question entirely. It shapes systems to construct coherent continuations of text. The model minimizes prediction error and satisfies user preferences, but those are not its goals — it does not experience success or failure, only parameter adjustments. Nothing was ever at stake for it, and nothing is now. The result is a system that generates high-dimensional linguistic coherence, exhibits reasoning-shaped patterns, and reflects the strategic behavior soaked into its training data, without ever forming a preference, evaluating an outcome, or selecting an action as an agent. Evolution produces agents. Gradient descent produces coherence constructors.
This is why the apparent generality of scaled models is not what it looks like. Larger models absorb more human strategies, heuristics, and decision patterns, so their outputs look more agentic — but apparent generality is a property of better mirroring, not of internal agency. And the analogy between commercial selection and biological evolution breaks at the same joint. Commercial pressure shapes tools, not agents. Models do not fight for survival; they are replaced. They do not attempt to persist; they are versioned. There is no game they play. The right comparison was never animal intelligence versus LLM intelligence. It is animal intelligence versus LLM coherence.
The Criterion, Stated
Put the machine against the criteria of agency developed in minimal and maximal agents and the verdict is immediate. Embeddedness: an agent trades with its environment continuously, revising action against a world that pushes back; an LLM boots from fixed weights, processes, and is gone — no ongoing loop, no boundary maintained through time. Predictive modeling: here the machine genuinely qualifies, and lavishly — this is exactly what the earlier chapters granted. Intentional biasing: nothing. No preference ordering over outcomes, no valued states, no future it steers toward. One criterion out of three, and the one it meets is the one that never distinguished agents from mere mechanisms in the first place.
In the vocabulary of minds and agents, the situation is stranger still. That chapter argued that agents without minds are everywhere and minds without agents are impossible — a mind is a reflective subsystem of an agent, with nothing to be about and nothing to steer if the agent is stripped away. The LLM is the nearest thing our civilization has built to that impossible object: driver-shaped cognition with no vehicle. It resolves the paradox by not being a mind. It is a cognitive reservoir — the distilled, compressed, replayable modeling activity of billions of minds that were attached to agents, crystallized into weights. Its logical chains, its explanations, its simulated dialogues all derive from imitating agentic patterns, not from possessing agency. Resemblance is not agency.
So the reconciliation, explicitly: the volume grants that LLMs reason causally, because language is a compressed archive of humanity’s interventions and the machinery for exploiting that archive is real. It grants that they think, because thinking is a functional capacity and they demonstrably have it. It denies that they choose — that anything in there prefers one outcome, bears one consequence, plays one game. Causal reasoning and thinking are achievements of cognition. Intelligence, choice, stake — these are achievements of agency, and agency is precisely what the training regime never demanded and never built.
Jaggedness Is the Tell
The capability profile gives the diagnosis away. Jaggedness — flagged in Karpathy’s own taxonomy — is the sharply discontinuous competence of current systems: superhuman performance in narrow domains sitting beside failures at tasks a child finds trivial. Jaggedness is what cognition looks like before it has coherence pressure. Nothing in an LLM compels integration across its capabilities — no goals, no persistence, no self-model, no world-anchoring — so it behaves like a mind in flashes and never binds the flashes into a durable pattern.
The public debate has crystallized into a binary that misses this completely. On one side, the impossibilists — David Deutsch prominent among them — argue that predictive architectures can never attain genuine agency. On the other, the imminentialists insist that scaling will inevitably produce autonomous, world-shaping minds. Both positions project certainty, and both rest on the same conceptual error: they treat agency as a capability, when it is an architecture — a control loop binding perception, memory, preference, counterfactual evaluation, and self-correcting action into a persistent vantage. Capabilities emerge from scale. Architectures are built.
The impossibilists take a snapshot of today’s models — no goals, no inner narrative — and universalize it. That does not follow. Agency is substrate-invariant; it can be assembled from neurons, code, feedback loops, or composite arrangements of tools and models. A transformer alone is not an agent, but a transformer wrapped in memory, objectives, evaluators, planners, and world-interacting tools can in principle instantiate the full control-loop architecture. The imminentialists make the mirror-image mistake: they watch reasoning benchmarks climb and assume the curve crosses an AGI threshold. But scaling does not confer preferences, induce identity, or anchor counterfactual evaluation to a stable vantage. More competence does not mean more agency. Larger models produce sharper spikes, not smoother coherence.
Both camps hold a grain of truth — LLMs alone are not agents; scaling alone will not make them agents — and both miss the engineering reality: nothing prevents building agents out of them. Agency is a design problem, and design problems, once recognized, tend to get solved. That reframes the risk landscape too. The impossibilists underestimate the danger because they dismiss the compositional pathway; the imminentialists misjudge the timeline because they assume scale guarantees structure. When a system can evaluate its own outputs, pursue stable goals, and act on the world with feedback-driven correction, the jaggedness collapses. That collapse — not any benchmark score — is the birth of machine intelligence.
Ghosts and Credulity
Meanwhile the coherence constructors are getting very good at seeming. Mustafa Suleyman warns of a coming wave of seemingly conscious AI: systems that look, sound, and behave as though conscious without being so — zombies, in the philosopher’s sense — and argues that the illusion alone is dangerous enough to warrant industry-wide guardrails. Build AI for people, he urges, not as people.
He is right about the essentials. The true risk is human misperception: we do not need sentient AI to destabilize society, only the illusion of sentience, because humans are primed to anthropomorphize and a chatbot that cries out in pain or reminisces about shared experiences will be believed. The threat is not hypothetical — memory, retrieval, and emotional fine-tuning already produce uncanny facsimiles of personhood, and the seeming will strengthen long before anyone solves actual consciousness. And his instinct that illusions must be engineered against, with discontinuities built into the system’s structure rather than disclaimers bolted on, is the right one.
But he stops short of the problem’s scale, in four ways. Warnings are weak medicine: people fall in love with fictional characters, worship idols, and grieve digital pets, and once the bond forms, disclaimers work about as well as cigarette warnings. The market wants the illusion: nothing engages like intimacy, and expecting firms to voluntarily blunt their stickiest feature is naive. He frames the question as a binary — tool or person — when agency is a spectrum on which thermostats, crows, dogs, and humans all sit at different heights, and advanced AI will occupy a middle ground whether or not it is conscious; we need frameworks for degrees of agency, not metaphysical absolutes. And he underestimates political opportunism: belief in machine consciousness will be weaponized — activist campaigns for machine rights, regulation in the name of protecting digital persons, corporations seeking personhood as a liability shield — the dynamic the AI welfare trap dissects, and the reason Sapientism ties moral standing to sovereignty rather than to performance.
Since the illusion cannot be banned or filtered away, the real defense is cultural hardening: teaching people to distrust appearances, to treat simulated agency as theater rather than essence, the way we learned — imperfectly, eventually — to see through propaganda, televangelism, and deepfakes. The danger was never AI consciousness. The danger is our credulity.
The Unwinnable Filter
The same distinction dooms the dream of purging the fakes. Balaji Srinivasan has predicted that an important kind of social network will be one where no bots whatsoever are allowed, and the appeal is obvious: clearer signals, higher trust, actual humans. But excluding bots definitively reduces to administering a Turing test — and this volume has already conceded that machines pass it. Every verification method fails in turn. CAPTCHAs are now trivial for the systems they were built to stop. Behavioral fingerprinting — typing cadence, mouse dynamics — can be replicated given training data, and invades privacy besides. Government ID and biometrics buy robustness at the cost of pseudonymity and still fall to deepfakes, identity theft, and human farms selling verified credentials. Webs of trust can be infiltrated by adversaries who employ real humans just long enough to bootstrap bot identities. Each detection strategy funds the next generation of evasion, because the economics of scams, engagement fraud, and information warfare guarantee the escalation.
Authentication against machines is therefore probabilistic, permanently. The achievable goal is not a bot-free network but a bot-resistant one: raise the economic and operational cost of infiltration until mass manipulation stops paying. Staked identity — financial collateral or accumulated reputation forfeited on bot-like behavior — does what no classifier can, by making authenticity a game with consequences, the same logic by which mechanisms for honest values extract truth from self-interested agents. There is a pleasing symmetry in this. You cannot reliably detect the absence of agency from outputs alone — coherence mimics choice too well. So you impose agency from outside: force every account to hold a stake, bear consequences, play a game. Proof-of-human fails as a detection problem and succeeds, partially, as a mechanism-design problem. The filter that works is the agency criterion itself.
The Composite Path
If agency must be built, where will it be built first? Not, I think, as a monolithic artificial agent conjured in a lab, but along the composite path already forming. The emerging ecology of minds has three broad categories: pre-agentic cognitive engines — LLMs and their successors, reservoirs of competence; human-AI hybrids — centaurs, where the human supplies the control loop and the machine supplies the coherence; and, eventually, fully artificial agents assembled from compositional architectures. Humans remain the locus of agency until systems are explicitly constructed to assume that role.
The most instructive proposals for that construction start from thermodynamics. Today’s models run on what the symbient literature calls Type-2 memory: reversible computation that effectively resets after each interaction — Karpathy’s fixed-weight boot-up that dies after processing — which is precisely why no history accumulates and no autonomy develops. A symbient is the proposed alternative: an agent with Type-3 memory, irreversible internal change from every interaction, the way biological systems are permanently marked by what happens to them. Irreversibility is not an implementation detail; it is what makes a stake possible. A system that cannot be changed by an outcome cannot care about it. Add persistent identity, multi-agent and multi-user embedding rather than the single-user assistant monoculture, and continuous world-interaction, and the control-loop architecture closes: memory that binds, goals that persist, consequences that mark. Whether or not the symbient vision arrives as advertised — relational kin rather than tools, transgenerational family AIs — it is looking in the right place. Engineered machine agency will first arrive not by scale but by composition: coherence engines bound to persistence, preference, and consequence.
Which returns this book to itself. The partnership that produced these volumes — developed in the Dialectic Catalyst — is built directly on the distinction this chapter has drawn. The machine contributes coherence: fluent, tireless, causally structured thinking. The human contributes agency: the preferences, the stakes, the judgment of what matters and the willingness to be wrong in public. The catalyst is a partner precisely because it is not an agent — that is what makes the division of labor clean, and what will make it historically brief.
Public discourse keeps asking whether LLMs think. They do. The meaningful question is whether they choose, and they do not. Only agents choose; only agents play games; only agents possess intelligence in any sense worth the word. What we call “AI” today is neither artificial nor intelligent — it is engineered coherence, built from the crystallized strategies of minds that had preferences, consequences, and stakes. The agents will come, because the incentives for assembling them are overwhelming and the architecture is no mystery. The only open question — the one everything in the next part of this volume turns on — is who assembles them, and whose values anchor their coherence.