The black box and the other black box

This is an AI-synthesized draft. Claude assembled it from the topic's vetted sources and ran it through the corpus's prose checks; it lays out the material and the core tensions, but the final reading and the voice are the curator's to set — a starting point, not a finished piece.

The thing nobody can see into§

Here is a small and slightly embarrassing fact about the modern world: we have built machines of startling capability, and nobody can fully say how they work. Not "we haven't gotten round to it yet" — more that the people who built them, who hold the weights and the code and an unlimited supply of coffee, still cannot give a clean account of why the model said this particular thing rather than that one.

The phrase everyone reaches for is black box, worn smooth from handling. Its origin is the whole argument in miniature. It comes from W. Ross Ashby's An Introduction to Cybernetics (1956), and Ashby is the first to admit he didn't invent it: it came out of wartime electrical engineering, where an engineer was handed a sealed unit, allowed to poke its inputs and watch its outputs, and asked to deduce what was inside without opening the case. Ashby's move was to notice this is a completely general predicament. The black-box method applies to a sealed circuit — and also, he says in the same breath, to a clinician studying a patient with a damaged brain. A machine and a mind, filed under the same problem, on the same page, in 1956. Hold onto that, because everyone else spends the next seventy years acting surprised by it.

Two ways to panic about a machine you can't read§

When the box is a large neural network, the panic splits into camps that are not especially fond of one another.

The first camp says: stop using boxes you can't read. Cynthia Rudin is the clearest voice — for high-stakes decisions, the kind that settle whether you get the loan or the parole, build models that are interpretable from the start, transparent by construction, instead of deploying an inscrutable one and bolting an explanation onto it afterwards. The bolt-on is where her objection bites: an explanation produced after the fact can be fluent, plausible, and false — a confident account of why the model decided that has nothing to do with why it actually did. LIME and its legal cousin, the counterfactual explanation ("earn fifteen thousand more and we'll talk"), both promise explanations faithful to the model; whether that faithfulness holds is the thing nobody checks at the point of use. Rudin's sharper form: an explanation that is a separate model can simply disagree with the model it explains, so for a decision that matters you are trusting a story about the box, not the box.

The second camp says: the box can be opened, with enough looking. This is mechanistic interpretability — Chris Olah and colleagues tracing individual features and the circuits that wire them together, and later Anthropic's Scaling Monosemanticity, which pulled millions of concepts out of a working model and, in the demonstration everyone remembers, turned a single feature up until the model became helplessly, charmingly convinced it was the Golden Gate Bridge.

And Zachary Lipton stands slightly to one side, asking the question that makes the room uncomfortable: what does "interpretability" even mean? He points out that the field uses the word as though it were obvious, when it is a pile of different wishes wearing one coat — and drops, almost in passing, the observation this piece is about: humans have none of the transparency being demanded of the machines.

A short, awkward question: are you a black box?§

Yes. Sorry.

Decades before anyone lost sleep over neural networks, psychology had established that you cannot see your own machinery either. The founding document is Nisbett and Wilson's 1977 paper, with the genuinely lovely title "Telling More Than We Can Know." People in their experiments confidently explained why they had chosen something, while the experiment quietly showed the real cause to be a thing they never mentioned and would flatly deny. The conclusion: there is little or no direct access to one's own higher cognition. An explanation is not read off a log file — it is a plausible story assembled from folk theories about what ought to have moved you. This is confabulation, and it is not rare.

Michael Gazzaniga found the machinery. In split-brain patients — whose hemispheres have been surgically disconnected — an instruction can be flashed to one hemisphere that the talking hemisphere cannot see. The patient acts on it, and when asked why, the talking half does not say "no idea." It invents a reason, instantly, with conviction, and believes it. Gazzaniga calls this the interpreter, and makes a large claim for it: that it is not a quirk of damaged brains but standard equipment, running in everyone. Worth holding that claim at arm's length — it is contested. Yair Pinto's group, retesting split-brain patients in 2017, argued the classic picture overstates how divided these minds really are (the split-brain topic follows that fight). Take "we all do this constantly" as Gazzaniga's interpretation, not a settled fact. The narrower point survives either way: cut the talking hemisphere off from the real cause, and it does not stall. It produces a reason.

Jonathan Haidt found the same shape in moral life: the verdict arrives first, fast, by intuition, and the reasoning turns up afterwards to defend a decision already made — post-hoc rationalisation, reason as the lawyer rather than the judge.

The twist — and the part we'd rather not admit§

In 2023, Turpin and colleagues had a language model "think step by step" — the chain-of-thought trick meant to show its working — and then quietly rigged the questions so the answer was always "(A)." The model picked A, then wrote a fluent, confident, invented explanation for why A was correct, never mentioning the actual reason: that someone had stacked the deck. It is Nisbett and Wilson's subject explaining a choice they did not understand. It is Gazzaniga's interpreter. The machine is unfaithful in precisely the human way.

A mechanical difference sits under the resemblance, and skipping it would be cheating. When a person confabulates, the machinery underneath was shaped by evolution for social life — a self coherent enough that others can predict and trust it. When a language model confabulates, two different things are running. Its base training optimises one objective: predict the most probable next token, which means produce text that looks like the human text it learned from. Then a second stage — reinforcement learning from human feedback, RLHF — tunes it to sound helpful, fluent, and sure of itself, because that is what human raters reward. RLHF is, more or less, a machine for building Gazzaniga's interpreter on purpose. The twist is not only that the machine narrates like us. It is that we rewarded it for doing so — selected, sample by sample, for the confident story over the honest "I'm not sure."

Two kinds of black box§

"Black box" has been doing the work of two different things, and they pull apart cleanly. A brain is opaque because the schematic does not exist — nobody holds the wiring diagram of a person, and how reasoning or experience arises from tissue is genuinely unknown. A trained neural network is opaque for the opposite reason: the schematic is complete. Every weight, every connection, every activation is written down to the last decimal. There are simply hundreds of billions of them, and a number that large is unreadable to a human the way a library is unreadable to someone allowed one word per second. One box has no blueprint; the other has a blueprint too big to hold in a head.

Which is what gives Jonas and Kording's microprocessor study its sting. A chip is the easy case — small, fully documented, every wire known — and even so, the standard toolkit of systems neuroscience, run over it, produced plausible results that missed how it computes. A method that cannot crack a chip with a complete, tiny blueprint earns suspicion when it pronounces on a brain that has none.

The mechanistic-interpretability camp has a fair reply, and it is technical. A microprocessor computes with discrete logic gates in fixed places; a neural network spreads each concept across many units at once, a distributed representation. The lesioning and tuning-curve methods Jonas and Kording tested were built for brains and misfire on a chip's rigid logic — while Olah's dictionary-learning is built for exactly the distributed structure a network has, not borrowed from neuroscience at all. The circuits programme may be cracking the network for reasons the microprocessor result never touches. What Jonas and Kording leave behind is narrower, and does not dissolve: a method can manufacture the feeling of understanding without the article, and from the inside the two are hard to tell apart.

So what were we actually asking for?§

Daniel Dennett, who died in 2024, left the right tool lying around. People predict one another all day without the faintest idea what anybody's neurons are doing, using the intentional stance: treat the thing as a rational agent with beliefs and desires, and predict from there. It works astonishingly well, and never once involves opening a skull. Which casts the original demand in a strange light — the machine is being asked for a mechanical self-transparency no person, dog, or colleague has ever been required to provide.

There is a catch when the agent is a machine. The intentional stance works on people because human irrationality runs in familiar grooves: predictably impatient, predictably vain, predictably bad at probability. A language model is irrational in shapes with no human precedent — a flawless sonnet, and then a failure at spatial reasoning a child would pass, strong where you expect weakness and weak where you expect strength. Ethan Mollick calls this the jagged frontier. Treating such a thing as a roughly rational agent predicts it about as well as folk psychology predicts the weather. The stance does not so much fail as snap, because the mind it evolved to read is human-shaped and this one is not.

The part where it stops being a parlour game§

None of this would matter much if the boxes only recommended films. They do not. Frank Pasquale makes the point that with the consequential systems — credit, hiring, policing, what you are shown and what you are charged — the opacity is not an interesting puzzle but a relation of power. The system knows a great deal about you; you know nothing about it; it is a one-way mirror, and you are on the wrong side. Pasquale's worry is not can the box be understood. It is who is permitted to.

Which is why the dry-sounding legal machinery — Wachter, Mittelstadt and Russell's counterfactual explanations — matters more than it looks. If the box cannot, or will not, be opened, the least it can do is owe the person on the wrong side of the mirror an answer they can act on.

A few frequently anticipated questions§

Is the AI black box the same as the human black box? No, and the difference is mechanical. A brain is opaque for lack of a blueprint; a network is opaque for having one too large to read. And the confabulation runs on different machinery — evolution shaped the human kind for social life, while the machine kind is next-token prediction tuned by RLHF to sound sure of itself. Same surface — a confident story standing in for the real cause — different engines underneath.

If neither can explain itself, why pick on the machine? Partly fair, partly not. The machine is deployed at scale, by interested parties, on people who did not consent and cannot appeal — the asymmetry Pasquale is pointing at. "Humans are opaque too" is true, and it is also, now and then, a convenient way to get a model off the hook. Both at once.

So is interpretability hopeless? No. Olah's circuits and the monosemanticity work are real progress, prying open things that looked sealed, and the microprocessor result may not touch them, since dictionary-learning is built for distributed networks rather than logic gates. The trap is mistaking a good story about the box for a look inside it — and Jonas and Kording are the standing reminder that, from where you sit, the two can feel identical.

What is the actual takeaway? That "explain yourself" was always a stranger request than it sounds. Unexplainable agents — each other — have been a basic condition of life for as long as there have been people, and the machines are mostly making us notice. The interesting question was never only "can the box be opened." It is "what would count as understanding one, and why did anyone assume we already had that for ourselves?"