The Architecture of Agency › Volume 8 › Exploiting the Best in Us

Exploiting the Best in Us

Hope and dread as recruitment fuels

This chapter is a review — it is readable but still changing.

In 1949, in the first issue of Monthly Review, Albert Einstein published an essay called “Why Socialism?” It is not the writing of a naïf. He diagnoses the “predatory phase of human development,” the way private capital concentrates until it captures the sources of information and the machinery of politics, the “crippling of the social consciousness of individuals.” His moral instrument was calibrated to a fine pitch. And then, having seen all that clearly, he wrote: “I am convinced there is only one way to eliminate these grave evils, namely through the establishment of a socialist economy.” He even raised, in the same breath, the objection that would bury the century: “how is it possible, in view of the far-reaching centralization of political and economic power, to prevent bureaucracy from becoming all-powerful and overweening? How can the rights of the individual be protected?” He posed the question — and then set it aside as a problem for later, a wrinkle to be ironed out by education and good will.

That is the whole tragedy in miniature. Einstein’s conscience worked. His model of how institutions behave under concentrated power did not. He recoiled from fascism, from militarism, from the psychopathic power politics of his era, and he assumed that benevolent planners could do what real planners never can: allocate resources without suppressing agency, distribute knowledge without collapsing its quality, and wield coercion without becoming addicted to it. His mistake was not kindness. It was the assumption that kindness scales. It does not.

I open with Einstein because he is the type specimen for the most underrated attack in political life, the one that closes the discourse chapters of this book and opens its chapters on capture. Every mechanism catalogued so far — the weaponized labels, the corrupted categories, the enforcement machinery — assumes a target who resists and must be worn down. The exploits in this chapter are more elegant. They do not overcome the target’s better nature. They recruit it. There are two of them, and they run on opposite fuels: hope and dread.

The Empathy Exploit

The thesis is this: opaque authoritarian systems conscript the virtuous first, because moral trust is the easiest resource to metabolize into power.

The most dangerous ideologies are not the ones that appeal to our vices. Those trip every alarm we have. The dangerous ones weaponize our virtues — compassion, high trust, moral seriousness, intellectual idealism — the very traits that make a civilization humane. They do this by advertising a moral horizon while concealing an institutional failure mode, so that the ideology’s stated aims and its structural dynamics point in opposite directions. The stated aim is inclusion, equality, shared purpose, the abolition of suffering. The structural dynamic is the annihilation of the distributed incentives that generate knowledge, coordinate action, and protect the individual against the planner.

The reason virtue is the attack surface, and not merely one target among many, is that empathic minds reason about intentions, and intentions are the camouflage layer. When you value cooperation over conflict and reason over domination, you instinctively read a moral claim as a claim about people’s hearts. You do not instinctively model tacit knowledge, emergent coordination, bureaucratic ratchets, coercion equilibria, or incentive-compatible failure. Those live in the domain of institutional design, and they are invisible at the level of moral language. So the high-trust mind sees a benevolent goal and grants provisional legitimacy, exactly where a colder mind would ask the boring question — by what mechanism, and what happens when the mechanism is captured? An ideology that signals cruelty activates defensive cognition. An ideology that signals compassion triggers compliance. Einstein saw a moral horizon where there was an institutional trap, and this blindness is not a personal failing peculiar to him. It is the predictable output of a well-functioning conscience meeting an ideology engineered to speak conscience’s language.

This is why opaque systems out-recruit transparent ones. They outsource their moral justification to the people least equipped to see the structural trap, and those people are the most credible messengers a movement could want. Empathy is socially contagious. When a figure of Einstein’s stature endorses an ideology, he exports his moral legitimacy along with it; the appearance of decency becomes the vector by which a system spreads that, once instantiated, consumes the very freedoms that made such decency possible. High-trust populations produce more moral capital than low-trust ones, and opaque ideologies parasitize the surplus. They do not manufacture sincerity. They metabolize it.

Which yields the asymmetry that is the heart of the matter. The propaganda of transparent tyrannies depends on fear; the propaganda of opaque tyrannies depends on hope. Fear mobilizes resistance. Hope mobilizes compliance. A regime that rules by terror announces itself as the enemy and provokes the immune response that eventually unseats it. A regime that rules by aspiration disarms the immune response before it fires, because the antibodies — suspicion, defiance, the refusal to be governed — read as cynicism, as a failure of solidarity, as a moral defect in the one who feels them. The better you are, the harder it is to feel them at all.

From there the descent can become a ratchet in four turns. It requires moral legitimacy to gain political leverage. Once leverage is secured, a plan that lacks continuing authorization may be enforced through credible threats against people who refuse it. Once that coercive mechanism is normalized, correction degrades because refusal carries a penalty. And when institutions systematically disable refusal and review, atrocity becomes a foreseeable selection pressure rather than a moral aberration: the system rewards whoever is willing to do what the plan requires. No stage of this needs a villain. The risk follows from ordinary institutional incentives, which is why good people can staff every level of it and why, looking back, no one can find the moment they became complicit.

I want the uncomfortable conclusion stated without softening, because the softened version is useless. The traits that make an individual admirable can make a society fragile. Compassion dulls the hostility-detectors. High trust blinds you to coercive drift. Moral seriousness misreads structural failure as sabotage, so that when the plan produces famine the plan’s defenders look for wreckers rather than for the flaw in the plan. Intellectual idealism assumes competence where none exists. This is not a story about sadists. The worst outcomes of the modern era were enabled less by cruelty than by idealism — by caregivers, not bullies. Opaque authoritarianism does not begin by attacking the wicked. It begins by conscripting the virtuous, because it needs their legitimacy before it can afford their coercion.

Opaque authoritarianism does not spread through bullies. It spreads through caregivers.

The Mirror Exploit: Dread

Hope is one fuel. Dread is the other, and it produces a mirror-image pathology — not compliance with a benevolent-sounding plan, but the manufacture of extremists by a movement that never intended to make any.

Consider a movement grounded in an extinction narrative: it proclaims nonviolence, and it also insists that some development — a technology, a policy, an industry — is a near-term path to the deaths of eight billion people, and that the institutions meant to stop it are incapable of responding. Hold those two commitments together and the second dissolves the first. If you have genuinely convinced someone that eight billion lives hang in the balance and that the ordinary channels are useless, it is not a surprise when one of them concludes that ordinary moral constraints no longer apply. It is the predictable output of the premises. The rhetoric is the catalyst; the pressure builds; and nonviolence, professed but never trained, becomes a slogan rather than a discipline. When an AI-safety group’s own cofounder is credibly accused of assaulting a colleague, the correct reading is not that the movement betrayed its principles. It is that the movement’s principles, combined with its threat model, produced exactly this.

The pipeline has four elements, and once they are all in place, violence is a foreseeable endpoint for whichever members internalize the narrative most literally:

Assemble these and you have built a machine that converts moral urgency into license for coercion, and it does not matter that the assembly was done by people who abhor violence. The structure repeats across environmental extremism, anti-nuclear agitation, doomsday religion, techno-skeptic movements — anywhere existential dread is made the animating energy of a group.

And it is the founders who radicalize first. This is the detail that distinguishes the catastrophe mindset from ordinary fanaticism, where we expect the drift to start at the disturbed margins and work inward. Here it starts at the center. Founders are temperamentally intense, identity-fused with the cause, and disposed to read every setback as further proof of the systemic failure they already believe in. The narrative they built for others intensifies fastest in the one mind most saturated by it: if everyone else is complacent, then perhaps preventing the disaster falls to me, by whatever means are left. The person most committed to averting catastrophe is therefore the most likely to become the present-day danger — which is the irony in full. A group formed to prevent a speculative future catastrophe manufactures a tangible present one out of ordinary human psychology. The threat was never only the hypothetical superintelligence. It was the social machinery the movement assembled around itself.

Nonviolence, in other words, cannot rest on good intentions. It requires training, explicit norms, and de-escalation protocols — it is a discipline, with the standing infrastructure a discipline demands, or it is decoration. A movement that saturates its members with extinction rhetoric and then points to its nonviolence statement as a safeguard has confused a wish for a mechanism. The remedy is not to abandon the concern; some catastrophes are real, and I take the strongest versions of the AI-risk case seriously enough to have worked through them elsewhere. This is the culture-side companion to that argument. Volume 3 asks whether the doom is true, weighing the cruxes and the risk arithmetic. This chapter asks what the discourse of doom does to the people who metabolize it, which is a separate question with a separate answer, and the two must not be collapsed. A claim can be well-calibrated and its surrounding culture still be a radicalization engine. The remedy for the catastrophe mindset is to revise the scaffolding: reduce the apocalyptic framing to what the evidence actually supports, affirm the basic dignity of the people on the other side of the issue, and build real internal mechanisms to contain radical drift before it reaches the founder.

The Common Structure, and the Fix

Set the two exploits side by side and the same skeleton shows through. Both target moral emotion, not moral failing. Both recruit the sincere rather than overcoming the resistant. Both work by binding a genuine feeling — hope for a better world, dread of a coming ruin — to a structure that will convert it into power or into violence, while keeping the structure itself below the threshold where moral cognition would notice it. Both are, in the vocabulary of this volume, the fuel that lets a memetic pathogen grow: the pathogen supplies the noble camouflage, and hope or dread supplies the sincerity it burns. And both are the engine room of what comes next, because capture is what an institution looks like after enough of its most conscientious members have been conscripted this way.

The tempting response is to conclude that empathy is the vulnerability and to prescribe less of it — to admire the cold, the suspicious, the unmoved. That is exactly wrong. Empathy is not the enemy. Empathy without institutional literacy is the enemy. The failure in every case above is not that people cared too much; it is that they let moral emotion answer a question it is not equipped to answer — the question of what a structure will do once it holds power, as opposed to what it says it intends. Einstein’s error was not that he felt the suffering of the poor. It was that he let the feeling stand in for an analysis of central planning under concentrated authority. The would-be radical’s error is not that he fears extinction. It is that his fear has been routed around every institution that might have checked it.

So the fix is not colder people. It is warmer people who have learned the boring questions and refuse to skip them: by what mechanism; who is coerced; what happens when this is captured; and what stops the plan when it is wrong? It is moral emotion embedded in structures that preserve agency and constrain coercion rather than moral emotion left to run on the open field of intentions. Empathy, as Volume 5 develops it, is a faculty of extraordinary power and no native sense of scale; it must be paired with the institutional knowledge that tells it when its instincts stop tracking reality. That pairing is the whole immunization. Diminish trust, compassion, or fairness and you have merely made a worse society that is still vulnerable. Teach a trusting, compassionate, fair society to ask what a structure does under load, and you have closed the attack surface without amputating the virtue.

Einstein was not a political idiot. He was a moral idealist navigating an opaque landscape with an incomplete map, and that mistake is forgivable in a way that repeating it, now that we have drawn the map, is not. A civilization that fails to understand these two exploits will keep producing the same result from opposite emotions: good people, moved by hope or by dread, empowering systems that eventually destroy the conditions that made their goodness possible. Understanding how the conscription works is the first move in refusing it — and refusing it is the subject of everything that follows.