Could a Neuroscientist Understand a Microprocessor?

Eric Jonas was in the Department of Electrical Engineering and Computer Science at the University of California, Berkeley when this appeared; Konrad Paul Kording was a computational neuroscientist then at Northwestern University and the Rehabilitation Institute of Chicago, later at the University of Pennsylvania. Kording is known for Bayesian models of the brain and for sharp methodological critique of his own field, which is the register this paper is written in.

It was published in January 2017 in PLOS Computational Biology, a peer-reviewed open-access journal. The design is a thought experiment carried out for real: the authors treat a classic microprocessor, the MOS 6502 that ran the Apple II and the Atari, as if it were a brain, recording its "neural" activity while it ran three games and then applying the standard analytic toolkit of systems neuroscience — lesion studies, tuning curves, connectomics, dimensionality reduction, Granger causality. Because the chip's full wiring diagram is known, there is a ground truth against which every method's output can be checked, and the checking is unkind: the analyses surface structure and produce plausible-looking results without reconstructing how the processor actually computes.

The piece sits as a primary methodological study and a deliberate provocation aimed inward at neuroscience. It is the black-box method that Ashby formalised, run for once on a system where the answer is known in advance, which turns the method into a test of the analyst rather than the artefact. It cuts in two directions across this topic at once: it humbles the project of reading a biological brain from the outside, and it is a caution to machine interpretability — the warning that a method can reveal structure that feels like understanding without delivering it. In that second direction it is the standing challenge to Olah's claim that the right methods can reverse-engineer a network feature by feature; the circuits programme is in part an argument that interpretability has tools Jonas and Kording's neuroscientists lacked.

The authors' stake is reputational and methodological. The paper is a critique from inside the discipline, using a system with a known answer to expose where the field's standard methods fail, and its value to the authors is the credibility that a well-aimed self-critique earns. There is no commercial or ideological interest in the result; if anything the paper makes the authors' own enterprise look harder than the field tends to admit.