Artificial Judgment
Ask a language model a hard question and you get five options. Ask an expert and you get one answer. The difference looks stylistic — models hedge, people commit — but it points at something deeper than style. It points at what may be the binding constraint on artificial intelligence.
Start with an intuition about mediums. We have Turing completeness for what is computable; it is tempting to want "expression completeness" for what is expressible. Modern AI seems to invite the concept: models with effectively unbounded cognition, unbounded inputs through tools, and one narrow output channel — text. The medium is the message, and the medium looks like a bottleneck.
But the intuition doesn't survive inspection, at least not in that form. Text is technically expression-complete; anything computable can be serialized as tokens. The real constraint is cost, and the real limit sits on the receiving end: a human reads three hundred words a minute and holds a few chunks in working memory. Expressiveness is a property of the channel plus the decoder. And once tools let a model render images, build interfaces, and act in the world, the output channel isn't bounded in any hard sense. The bottleneck, if there is one, must be somewhere else.
It is. Consider the abundance problem: models enumerate where authors conclude. An author facing a terminal medium — no follow-up turn, no repair — is forced to make calls. This thesis, not those five. Chat, endlessly repairable, never applies that pressure. But the medium explanation only goes so far, because enumeration and selection are different cognitive acts. Listing options requires knowing the option space. Choosing one requires a view about what matters — and a willingness to be wrong. A model that hands you every option has, in a real sense, expressed less than one that says "do this, here's why." It has offloaded the hard part back to you. Enumerating options is the easy part. Collapsing them is the work.
That work is judgment, and judgment has a peculiar structure: it resists articulation. Hayek saw the same structure in prices. The price of tin doesn't tell you why tin got scarce; it tells you what to do about it, with the explanation compressed away. A price aggregates knowledge that cannot be centralized because it was never written down — the factory manager's hunch, the local circumstance no report captures. An editor who says "cut this chapter" emits the same kind of signal: a scalar decision backed by a thousand absorbed cases that were never articulated and largely can't be. Polanyi's phrase covers both: we know more than we can tell.
Here is the problem for machine learning. Models learn from text — the articulated residue of human knowledge — while judgment lives precisely in what never got articulated. Reinforcement from human feedback looks like a fix, and it does compress thousands of raters' preferences into a policy. But notice the disanalogy. Prices work because every participant has skin in the game; the signal is backed by people bearing the cost of being wrong, updating on the margin, in a specific place and moment. Averaging stakeless ratings into one frozen global policy is nearly the opposite of a market. It smooths away exactly the situated information that made the signal valuable. A judgment function built by averaging will naturally output the whole distribution rather than a committed point — which is to say, it will hedge.
If judgment can't be articulated into a model, it has to be grown the way markets grow prices: through consequence-bearing feedback in specific contexts. And this is where our current mode of delegation fails. We hand AI decomposed tasks — write this function, draft this email — and the consequence signal is absorbed by whoever owns the loop. The human sees whether the deal closed or the customer churned; the model sees "task completed." Decomposition severs credit assignment. An assembly-line worker doesn't develop a founder's judgment no matter how many years on the line, because the feedback that teaches is attached to the whole, not the parts. The economic version is familiar too: a firm that owns its P&L gets the full price signal, while a cost center gets a KPI — a proxy someone articulated, which is exactly what tacit knowledge cannot survive being reduced to. Metric-scoped delegation is central planning in miniature, and Goodhart follows.
There is an obvious tension. End-to-end ownership is what makes an agent consequential enough to be dangerous, yet we won't hand over loops until the model has judgment — which it can't form without holding a loop. Humans solved this bootstrapping problem long ago, and the solution is apprenticeship: graduated ownership of small loops, real outcomes, bounded blast radius. This customer, this campaign, this codebase. That is arguably where the agent trajectory is already heading, whether or not anyone designed it that way.
The conclusion inverts the opening intuition. The bounded thing in AI was never the output medium, and it may not even be intelligence. It is that models have stood outside the loops where consequences accumulate into knowing. Judgment isn't articulated. It's grown — by bearing the cost of being wrong.