Index Errans

person in Black box

Konrad Paul Kording

Black-box methodInterpretability

in Black box

Konrad Paul Kording, computational neuroscientist. At Northwestern University and the Rehabilitation Institute of Chicago when the microprocessor paper appeared, later at the University of Pennsylvania. He is known for Bayesian models of the brain and for sharp methodological critique of his own field.

Stake§

Reputational and methodological — a critique from inside neuroscience, using a system with a known answer to expose where standard methods fail. The value to the authors is the credibility a well-aimed self-critique earns.

Kording's contribution to this topic, with Eric Jonas, is to turn the black-box method into a test of the analyst rather than the artefact. Applying lesion studies, tuning curves, connectomics, and dimensionality reduction to a chip whose wiring is known, they find the methods do not reconstruct its logic — which bears directly on whether machine interpretability can be trusted where there is no answer key, as there is none for a brain or a deployed model.

Works in this corpus§

their concepts on the territory
Black-box methodBlack-box method InterpretabilityInterpretability

2 concepts in this scholar's webopen the full territory →

excerpts

We show that the approaches reveal interesting structure in the data but do not meaningfully describe the hierarchy of information processing in the microprocessor.
— Could a Neuroscientist Understand a Microprocessor? (2017)

The test: take a system whose every wire is known — the MOS 6502 running Donkey Kong, Space Invaders, and Pitfall — and run standard neuroscience analyses on it. The methods produce results that look like findings but do not recover the logic that is sitting right there in the schematic.

on Black-box method

Ultimately, the problem is not that neuroscientists could not understand a microprocessor, the problem is that they would not understand it given the approaches they are currently taking.
— Could a Neuroscientist Understand a Microprocessor? (2017)

The problem the authors locate is in the methods rather than the target: a toolkit that cannot recover the logic of a chip with a known answer gives reason to doubt its verdicts on the brain, where there is no answer key.

on Interpretability

⤺ mentioned in

articles