Index Errans

person in Black box

Chris Olah

Mechanistic interpretabilityInterpretability

in Black box

Chris Olah, interpretability researcher. He led the Clarity team at OpenAI when "Zoom In: An Introduction to Circuits" appeared in Distill, the web journal he had founded and edited; in 2021 he co-founded Anthropic, where he leads the interpretability team. With his collaborators he is the originator of the circuits approach to neural-network interpretability.

Stake§

Reputational and intellectual, and total — the article states the thesis that neural networks are not irreducible black boxes, which the same people then spent years trying to vindicate, at OpenAI and then at Anthropic. Distill was nonprofit, so the early investment was in the idea rather than a product; the later Anthropic work carries the commercial dimension.

Olah's contribution to this topic is the claim that a network not built to be transparent can still be reverse-engineered from the inside — mechanistic interpretability. "Zoom In" traces individual feature detectors in a vision model and the circuits of weights that assemble them into complex ones, and the programme scales toward a frontier language model in Scaling Monosemanticity. The bet runs against Jonas and Kording's demonstration that standard analysis methods can miss a mechanism that is fully known.

Works in this corpus§

their concepts on the territory
InterpretabilityInterpretability Mechanistic interpretabilityMechanistic interpretability

2 concepts in this scholar's webopen the full territory →

excerpts

In contrast to the typical picture of neural networks as a black box, we've been surprised how approachable the network is on this scale.
— Zoom In: An Introduction to Circuits (2020)

The claim that frames the whole [[concept:mechanistic-interpretability|mechanistic interpretability]] programme: looked at closely enough — neuron by neuron, weight by weight — the network stops being a black box. The wager is that opacity is a function of where you stand, not a property of the system.

on Mechanistic interpretability

Features are the fundamental unit of neural networks. They correspond to directions. These features can be rigorously studied and understood. … Features are connected by weights, forming circuits. These circuits can also be rigorously studied and understood.
— Zoom In: An Introduction to Circuits (2020)

Two of the paper's three speculative claims (the third is universality across models). Features and the circuits joining them are offered as the units that make a network legible — the alternative to treating it as an irreducible whole.

on Mechanistic interpretability

For instance, we see that clamping the Golden Gate Bridge feature 34M/31164353 to 10× its maximum activation value induces thematically-related model behavior. In this example, the model starts to self-identify as the Golden Gate Bridge!
— Scaling Monosemanticity: Extracting Interpretable Features from Claude 3 Sonnet (2024)

The "Golden Gate Claude" demonstration: a single extracted feature can be turned up, and the model's behaviour changes in a way that names the feature's meaning. The claim is that the feature is a handle on the concept rather than merely a correlate of it — opening and steering the box at once.

on Mechanistic interpretability

⤺ mentioned in

articles