Fallibilist Bayesianism

Bayes within models, criticism between them

A diagnostic system considers three diseases, assigns them prior probabilities, and updates flawlessly as each new test result arrives. Its arithmetic is impeccable; its posteriors are coherent; it grows more confident with every observation. The patient has a fourth disease, absent from the model. Because normalization forces the probability mass to sum to one across the three represented hypotheses, every scrap of evidence — including the evidence that should be screaming none of these — is redistributed among false alternatives. The system does not merely fail. It fails while becoming more sure of itself, and its confidence is generated by the exhaustiveness assumption that reality declined to honor.

This is the failure that no amount of conditionalization can catch from the inside, and it marks the seam this whole volume has been working along. Bayesian conditionalization is exact within a model. Given a hypothesis space, a prior, a representation of evidence, and a likelihood function, it says precisely how the evidence should redistribute belief. What it cannot do is tell you whether the hypothesis space omits the truth, whether the evidence was carved at the right level, or whether the likelihoods encode a false causal picture. A model can be internally perfect and externally wrong. The update rule moves probability mass among possibilities that have already been represented; inquiry also has to create possibilities, re-represent evidence, and discard the models within which probabilities were assigned in the first place.

So the stance this chapter names rests on a division of epistemic labor that can be stated in five words: Bayes within models, criticism between them. The volume has practiced it throughout. It is time to name it.

The Tribes and Their Common Mistake

People argue about probability as if they were all discussing one thing. They are not. One person means a degree of belief; another a long-run frequency; a third a physical tendency in the world; a fourth a formal measure over possibilities carrying no interpretation as belief at all; a fifth denies that a scientific theory is even the kind of thing that takes a probability. The familiar labels — Bayesian, frequentist, likelihoodist, falsificationist — compress a disagreement whose real structure only appears once four separate questions are pulled apart.

The first is ontological: what is probability? A credence, a frequency, a propensity, a measure over states, or several distinct things sharing a common mathematics. The second concerns scope: what receives a probability? A future event, a parameter, a whole theory, a branch of the wavefunction, an agent’s own uncertainty. The third is evidential: what does evidence do? Update a credence, shift a likelihood ratio, raise the severity of a test, improve predictive calibration, or expose a malformed hypothesis space. The fourth is methodological: how does inquiry progress? By conditionalization, model comparison, conjecture and refutation, error control, or conceptual reconstruction.

These questions interact but do not collapse. A statistician can run Bayesian computations while denying that probability is credence. A machine-learning researcher can optimize probabilistic predictions while staying serenely indifferent to whether the numbers are chances, credences, or mechanisms. Most usefully for what follows: Karl Popper held an objective-chance view of probability — a propensity theory, on which a physical setup possesses a genuine tendency to produce its outcomes — while simultaneously denying that a scientific theory has any probability of being true. His anti-Bayesianism about theory confirmation and his realism about physical chance sit on different axes entirely. Critical rationalism and propensity are not the same position, and conflating them has confused a century of the argument.

Each school earned its place by seeing something real. Subjective Bayesians (Ramsey, de Finetti, Savage) tied belief to coherent action. Objective Bayesians (Jeffreys, Jaynes) sought constraints on priors stronger than mere coherence. Frequentists secured objective performance guarantees. Likelihoodists separated evidential support from belief. Predictivists demanded out-of-sample success. Error statisticians (Mayo) asked whether a claim had survived a test capable of exposing its flaws. Critical rationalists insisted that knowledge grows by conjecture and refutation, not by accumulating confirmation. Pluralists refused to force credence, chance, frequency, and measure into one metaphysical box.

The recurring pathology is not any one of these insights. It is imperial expansion: taking a school’s genuine local discovery and inflating it into a total theory of rationality. Credence becomes all probability. Conditionalization becomes all learning. Long-run error control becomes all warrant. Falsification becomes all scientific judgment. The most instructive case is the Bayesian one, because it is the most sophisticated. Confronted with the objection that a finite agent’s hypothesis list omits most possibilities, the imperial Bayesian reaches for a universal prior — a Solomonoff mixture assigning weight to every computable hypothesis — and declares the space complete in principle. This secures formal universality by removing operational content. No embodied agent carries an explicit distribution over every possible ontology, and “included in principle” does no work when the hypothesis has yet to be conceived. The difficulty has not been solved; it has been pushed into the prior and relabeled.

The Named Stance

The resolution is not a new school to set beside the others. It is a settlement among their true parts, and it has a slogan:

Bayesian about updating, pluralist about probability, critical-rationalist about knowledge.

Bayesian about updating. Wherever a probabilistic model has earned its authority, conditionalization is the right way to redistribute confidence, and an agent facing a decision under uncertainty needs some integrated representation of what it believes and what is at stake. Bayesian credence remains the most developed general instrument for that job. This is the ground defended in defense of Bayes: a credence about a theory is a fact about the agent’s uncertainty, not a physical chance attached to the theory, and the division of labor holds — abduction proposes, criticism culls, Bayes keeps the books.

Pluralist about probability. Credence, frequency, physical chance, and quantum measure share a mathematics but not a substance. The QBU module’s separation of objective Measure from subjective Credence is one instance of a wider discipline: whenever probability is invoked, ask which kind. Objective structure — stable frequencies, symmetries, physical measures, tested causal models — constrains rational credence without being identical to it. An agent who assigns 0.9 to heads on a fair coin is not offering a valid alternative perspective; the world supplies structure that defeats him.

Critical-rationalist about knowledge. Every probability is conditional on a frame that had to be built, and every frame remains open to attack. The machinery for that attack is the pancritical rationalism of rationality without foundations: no belief, including this one, sits on an exempt foundation; positions are held rationally when they are held open. This is what supplies the “between models” operation that conditionalization structurally cannot. The frame is earned or unearned before the first number is written, as probability after probabilism argues, and the difference between an internal error the calculus can catch and an external error it cannot is exactly the difference between arithmetic inside the frame and the adequacy of the frame itself.

The position accepts every school’s demand and refuses every school’s imperialism. It keeps the mathematics and rejects its expansion into a total philosophy of mind.

The Cycle

Stated as an operation rather than a doctrine, rational inquiry runs a loop the closed Bayesian picture leaves out. The closed picture goes prior → evidence → posterior → await more evidence, forever within one representation. The open cycle is:

construct, infer, predict, criticize, reconstruct.

Construct. Hypotheses are invented, not read off a menu. Theories arrive through analogy, recombination, and attention to anomalies that existing frameworks classify as irrelevant; a new causal model can reveal that several unrelated phenomena share a mechanism, split one category into several, or turn evidence once treated as noise into the decisive signal. Bayes can compare geocentric with heliocentric astronomy once both are specified. The conceptual labor of formulating the heliocentric model lies outside the comparison.

Infer. Here, and only here, conditionalization governs the credences: estimate parameters, compare represented hypotheses, allocate confidence. Action then follows by a separate rule — expected-utility choice, which takes those credences together with a utility function. Modern methods stretch this stage impressively — hierarchical models, Bayesian nonparametrics, an unbounded number of clusters under a Dirichlet process — but an infinite model class is still a model class. It can infer another cluster; it cannot discover from its own formal resources that the observations should have been represented as trajectories, or as artifacts of selection, in the first place.

Predict and criticize are where the loop opens. A posterior can be razor-sharp while the model systematically misses the data — residuals that retain temporal structure, predictions miscalibrated across subgroups, a fit to historical observations that collapses under intervention. Posterior predictive checks and structured residual analysis ask the question conditionalization never asks: can this model reproduce the relevant patterns at all? A model earns criticism not from idle doubt but from concrete failure — failed predictions, repeated ad hoc repair, collapse under a plausible boundary change. Criticism can even reject the question: “which personality type causes extremism?” presupposes stable personality categories and a unitary phenomenon, and a crisp number can create the illusion that the conceptual work is finished when it has not begun. The worked ledgers of Bayes in the wild show the pathology in the flesh — a posterior near 0.90 riding on a single fragile likelihood ratio, precise to the decimal and unearned at the root.

Reconstruct. Criticism becomes productive when it points to the missing distinction, the false assumption, the omitted cause — and builds the next frame, which re-enters the cycle as a fresh construction. No final model escapes this. A model can become extraordinarily reliable, sustained by stable measurement and integration with broader theory, and its authority remains conditional on the practices that sustain it. Fallibility is about revisability, not uniform doubt: some models have survived vast ranges of criticism, others are verbal sketches, and treating them as equally uncertain is its own error.

Instruments That Narrow the Gap

Part of the critical work can be formalized, and it cuts against a lazy reading in which “criticism” is a mystical human residue the mathematics can never reach. Structural causal models make causal assumptions explicit, distinguish observation from intervention, support counterfactual reasoning, and reveal exactly when an ordinary conditional probability cannot answer a causal question. Causal-discovery algorithms search over candidate graphs, using independencies or interventions to eliminate structures. These instruments narrow the distance between formal inference and model criticism.

They do not close it. Every structural causal model begins with a choice of variables, admissible graph structures, and measurement assumptions; causal discovery leans on conditions — acyclicity, causal sufficiency, faithfulness — that may be justified, approximate, or false. An algorithm searches within a specified family of structures; nothing in it guarantees that the chosen variables capture the relevant ontology or that a latent process has not been omitted. Formal causal inference gives criticism sharper tools, and the tools themselves remain open to criticism. The gap narrows; it does not vanish.

The same holds for the agent doing the criticizing. The distinction between inference, criticism, and reconstruction is functional, not a distinction between machines and people. Inference changes values within a representation; criticism evaluates the adequacy of the representation; reconstruction changes it. A system participates in criticism whenever it detects a representation’s failure against a standard and uses that failure to revise the representation or the space of alternatives — mere adaptation does not count. Automated systems already run posterior predictive checks, search for counterexamples, compare causal graphs, and propose hypotheses; theorem provers expose inconsistent assumptions; language models generate candidate explanations and adversarial tests. Humans currently do much of this work because they bring broad background knowledge and cross-domain transfer, but nothing in the argument grants those capacities a biological monopoly. A sufficiently capable artificial agent could practice fallibilist Bayesianism more consistently than most humans do. Criticism is also frequently distributed — across scientists, engineers, and institutions — because agents grow adapted to the frames they helped build, and a community can expose errors no isolated reasoner would find. The same social machinery can also entrench a shared mistake once prestige and incentive protect a frame past its empirical expiry, which is why fallibilism has to apply to systems of inquiry, not only to minds.

The Economics of Reopening

Criticism is not free, and a stance that demanded it everywhere would never let an agent act. Reopening a model consumes time, computation, attention, and access to alternatives, so rational inquiry depends on provisional closure: some assumptions are held fixed because revisiting them would cost more than the errors they are likely to cause. This gives fallibilism an operating constraint rather than a counsel of perpetual doubt. Newtonian mechanics is the right model for bridge design because relativistic corrections are negligible at that scale; a triage heuristic can beat a richer model when the decision must be made in seconds; a software team may tolerate a known architectural defect until its operational cost exceeds the cost and risk of redesign.

So the choice between repairing a model and replacing it is comparative, not algorithmic. A local repair is justified when it preserves the model’s established mechanisms, explains the anomaly with independently motivated structure, and keeps generating successful predictions; it becomes suspect when it adds unconstrained flexibility, invents a fresh exception for each failure, or protects the model by making it harder to test. A replacement frame earns preference when it explains the anomaly while preserving the predecessor’s successes, reduces arbitrary assumptions, and generates new discriminating tests. This is neither a decision procedure nor an invitation to relativism. The absence of a universal threshold does not make every judgment equally defensible; a new frame gains authority by outperforming the old one under shared constraints the world imposes. Inference inside a fixed representation can itself be expensive, but reconstruction poses a harder problem, because the target representation is not yet known and the search cost is open-ended. Bounded agents must decide where to spend their criticism. The operative question is rarely whether a model is incomplete — every useful model omits structure — but whether its inadequacy has grown large, systematic, and costly enough to be worth replacing.

The Beginning

Rationality, then, is not the possession of a probability for every proposition. A rational agent knows how to formulate propositions worth evaluating, distinguishes uncertainty inside a model from uncertainty about it, recognizes when evidence challenges a hypothesis and when it challenges the language the hypotheses were written in, updates when updating is warranted, and reconstructs when the frame has failed. There is no view from nowhere in any of this: every criticism draws on some background knowledge, vocabulary, and purpose, and those resources are themselves open to attack. That is not paralysis and not infinite regress. It is the ordinary condition of inquiry, which proceeds by local reconstruction from a current position rather than by descent to a foundation — the working posture the whole volume has been arguing toward and now, finally, names.

What that posture asks of a life, once the machinery is set down and the practice remains, is the beginning of wisdom.

Bayes within models.

Criticism between them.