Tool Bias

When intelligence flatters its own instruments

Robin Hanson believes the UFO evidence deserves to be taken seriously. Not as folklore — as data. Confronted with the recent corpus of “glints,” sensor tracks, and pilot testimony, he assigns less than a twenty percent probability to the hypothesis that it is all mundane noise, and drives his posterior toward the alternatives: alien probes, or a coordinated governmental deception sustained across decades. The reasoning is presented in impeccable Bayesian dress — priors stated, likelihoods weighed, updates performed in public.

This matters because of who is doing the updating. Hanson is one of the most original applied epistemologists alive. He invented prediction markets, the institution this book leans on whenever it needs beliefs to bear the cost of their own implications — a debt paid in full in Mechanisms for Honest Values. His Grabby Aliens model is a genuinely beautiful piece of work: economic reasoning applied to cosmic evolution, yielding sharp predictions about loud, expansionist civilizations from a handful of clean assumptions. If updating toward aliens on ambiguous sensor data were simply a mistake weak minds make, Hanson would be the last person to make it.

He is closer to the first. And that is the subject of this chapter: a failure mode that does not prey on weakness but on strength — that strikes hardest exactly where the analytical machinery is sharpest. I call it tool bias, and no account of intelligence is complete without it, because it is the characteristic way intelligence fails.

Every Lens Is an Attractor

Every mind has a preferred architecture. Economists see incentives; physicists see invariants; sociologists see hierarchy; evolutionary theorists see adaptation. These lenses are not passive descriptions waiting to be matched against the world — they are active attractors. When the data is rich, the world can push back and discipline the lens. When the data is ambiguous or low-bandwidth, the pushback fails, and the mind bends the world toward the tools it trusts.

Tool bias is intelligence updating toward hypotheses that maximize the relevance of its own machinery. The failure is subtle, not theatrical. It masquerades as rigor, because the internal reasoning is clean while the inputs are contaminated: every step of the calculation can be valid, and the conclusion still worthless, because the hypothesis space was curated — silently, before any arithmetic began — by asking which possibilities give my instruments something to do. In the vocabulary of Probability After Probabilism, this is a frame pathology, not an arithmetic one. The frame precedes the number, the carving precedes the calculus — and tool bias is a systematic distortion in the carving. It selects the event-space, the reference class, and the live hypotheses that flatter the analyst’s comparative advantage, then invites probability theory in to consecrate the result. Bayesian form over Bayesian substance.

Hanson’s UFO update is the ideal specimen, because the surface rhetoric is flawless and the structure underneath is warped in three distinct places. Each warp is instructive on its own. Together they trace the anatomy of the bias.

First Error: Correlated Channels Dressed as Independent Witnesses

The engine of Hanson’s update is the claim that the data carries too much structure to be mundane — multiple sensors, multiple observers, converging tracks. Independent confirmations multiply; that is the whole power of the likelihood term.

But classified sensor data is not a set of independent witnesses. Modern military sensing is a fused system: shared processing pipelines, shared timing signals, shared classification libraries, fusion algorithms that merge tracks before any human sees them, and operators primed by earlier cues to see what the last briefing taught them to see. A single software artifact can generate a multi-sensor “confirmation.” A fusion glitch can produce synchronized ghosts across radar, infrared, and optical channels at once. Treating these channels as independent evidence is a structural error — multiplying probabilities that should be discounted or collapsed into one.

Notice what kind of error this is. It is not a failure of calculation; the multiplication is performed correctly. It is a failure of evidence modeling — a mis-specified likelihood, invisible from inside the calculation, fatal to everything downstream. And notice which direction the mis-specification runs: toward structure, toward signal, toward a world with something in it worth analyzing.

Second Error: Silence Read as Coordination

The second pillar of the update is institutional opacity. The government’s decades of stonewalling, classification, and contradiction are read as evidence of sustained, strategic deception — a cover-up, which implies something worth covering.

But bureaucratic silence does not require intention. It requires only fear, compartmentalization, and inertia. The government does not maintain a coherent narrative about anything for seventy-five years — not public health, not foreign policy, not procurement. The idea that it has flawlessly maintained one about aliens misunderstands how institutions fail. Silence is the equilibrium state of risk-averse systems: no official was ever fired for classifying a document, and every ambiguous sensor track is someone’s potential embarrassment.

From the outside, confusion and conspiracy are hard to distinguish, because both produce opacity. From the inside, they could not be more different. Confusion is low-energy drift — the default. Conspiracy requires continuous strategic maintenance across changing administrations, personnel, and incentives — a costly, fragile achievement. The first is common; the second is exceptional. Reading opacity as coordination inverts that base rate. And it inverts it, again, in a specific direction: toward the hypothesis with agency in it. An entropic bureaucracy offers nothing to a theorist of incentives and signaling. A strategic deceiver offers everything.

Third Error: The Epicycle

The third warp is the most revealing, because it shows the bias operating on Hanson’s own best work.

Grabby Aliens predicts loud civilizations: expansionist, resource-hungry, visible across cosmological distances. UFO behavior is the opposite — quiet, local, inconsistent, strategically incoherent. Craft that flaunt themselves to fighter pilots while concealing themselves from telescopes fit no rational expansion strategy the model contemplates. The clean move, when a beautiful model meets discordant data, is to let the model speak: it says this is not what expanding civilizations look like, so the data is probably not civilizations.

Instead, Hanson reconciles the two with a Zoo Hypothesis: aliens born in our stellar nursery impose a non-expansion regime on us, watching quietly while declining to colonize. Nothing in the data demands this; it is an auxiliary story bolted on to keep both the model and the anomaly alive. Ptolemy’s astronomers knew the maneuver well. When elegance is sacrificed to salvageability — when a theory admired for its clean derivations starts accepting ornaments whose only function is rescue — the conceptual compass has already drifted.

The Mechanism Beneath the Mistakes

Three errors, one signature. Ask why a mind of this caliber over-updates on ambiguous evidence, and the answer is uncomfortable in its simplicity: the hypothesis he selected is precisely the one that activates his core competencies.

Noise gives him nothing to model. Sensor ghosts and bureaucratic entropy are dead ends for a theorist of strategic behavior — no incentives to trace, no equilibria to solve, no signaling games to unwind. But aliens and conspiracies place him back on native terrain, where incentives shape behavior, strategic silence becomes analyzable, and coordination dynamics can be mapped. In a world of correlated glitches, Hanson has no comparative advantage. In a world of covert alien governance or disciplined bureaucratic deception, he regains full relevance. The hypothesis that maximizes the applicability of his tools is the hypothesis his posterior drifts toward — and each of the three errors above is exactly the local distortion needed to keep it afloat.

This is not an accusation of bad faith; it is the opposite. Tool bias operates below the level at which honesty applies. No one decides to mis-specify a likelihood. The machinery simply finds structure congenial and noise barren, and in the absence of disciplining data, congeniality wins. That is why the failure masquerades as rigor so effectively: the specialist’s reasoning about the flattering hypothesis really is rigorous. The corruption happened earlier, in the choice of what to reason about.

And the bias respects no school. I am not exempt. Axio’s instruments are agents, models, and control loops, and a mind marinated in cybernetics will be tempted to find controllers in the clouds — to read regulation into every stability and purpose into every pattern. The penalty I am about to propose applies at home first.

The Discipline: Tax the Flattering Evidence

The Discipline of Updating established the general practice: test the frame, not merely the coherence inside it, because any frame can be storied into coherence, and the most dangerous frames are the ones that generate their own supporting evidence. Tool bias is that pathology in its most seductive costume — the frame under test is your own expertise, and the evidence it generates arrives wearing the colors of your best work.

So the corrective is a targeted extension of that discipline: when ambiguous evidence seems to energize your preferred tools, impose an additional epistemic penalty on it. Not a refusal to update — a surtax. The moment you notice that a hypothesis would make your machinery decisive, that observation is itself evidence about the interpreter, and it should be priced in against the interpretation. Minds naturally amplify data that makes them relevant; the discipline is to restrain that amplification deliberately, in proportion to the flattery.

The penalty bites hardest exactly where it is most needed: on low-bandwidth anomalies. Evidence emerging from classified, non-replicable, correlated systems should barely move a posterior, however much structure it seems to carry, because the structure cannot be audited and the channels cannot be counted. When such evidence moves a posterior dramatically, the cause is rarely the data. It is the geometry of the mind interpreting it — and the update is a self-portrait, not an inference.

A System Vulnerable to Its Strengths

Sophisticated minds do not fail by missing obvious patterns. They fail by seeing patterns where their tools fit too neatly, by mistaking correlated noise for independent testimony, by letting a theoretical aesthetic pull harder than the quality of the data. Hanson’s UFO update is not madness and it is not carelessness. It is a structural failure mode: a specialist applying a sharp, agent-centric toolkit to a domain dominated by noise and secrecy, and finding — as sharp toolkits reliably do — a world shaped like their handle.

That is the completing clause in this part’s account of intelligence. Intelligence is a game we play, a portfolio of instruments rather than a mystical essence — and every instrument sharp enough to be worth owning is sharp enough to carve its own likeness into ambiguous data. The stance that survives this fact is restraint: do not infer high-order agency from low-fidelity stimuli; do not project your model into ambiguous inputs; and never let the tools you wield most fluently determine the shape of the world you think you see.

Nor is the pathology confined to judging evidence. It governs how we judge minds — including the new ones. When philosophers and engineers argue about whether machines understand, each faction arrives wielding its favorite instrument: the philosopher’s intuitions about symbols and semantics, the biologist’s convictions about substrates, the lawyer’s categories of copying and storage. Each finds, in the ambiguous interior of the machine, exactly the structure its tools were built to detect — and the debates about machine minds have been bent around those instruments for fifty years. Anatomizing the four great fallacies that result is the business of the next chapter.