Index Errans

person in Black box

Shan Carter

Mechanistic interpretability

in Black box

Shan Carter, researcher in data visualization and machine-learning interpretability. He co-founded Distill with Chris Olah and worked at Google Brain's People + AI Research before joining Anthropic's interpretability team. He is known for interactive explanations of machine learning, including the Activation Atlas work with Olah.

Stake§

Reputational and intellectual — a co-author across the circuits programme from its Distill beginnings to the Anthropic feature work, with the professional investment in the thesis that internals can be made legible.

Carter is a co-author of both "Zoom In" and Scaling Monosemanticity, the first and the scaled-up paper of mechanistic interpretability for this topic. His particular contribution runs through the visual, interactive presentation of model internals — making features and circuits legible on the page, which is part of the claim that they can be studied at all.

Works in this corpus§

their concepts on the territory
Mechanistic interpretabilityMechanistic interpretability

1 concept in this scholar's webopen the full territory →

excerpts

In contrast to the typical picture of neural networks as a black box, we've been surprised how approachable the network is on this scale.
— Zoom In: An Introduction to Circuits (2020)

The claim that frames the whole [[concept:mechanistic-interpretability|mechanistic interpretability]] programme: looked at closely enough — neuron by neuron, weight by weight — the network stops being a black box. The wager is that opacity is a function of where you stand, not a property of the system.

on Mechanistic interpretability

Features are the fundamental unit of neural networks. They correspond to directions. These features can be rigorously studied and understood. … Features are connected by weights, forming circuits. These circuits can also be rigorously studied and understood.
— Zoom In: An Introduction to Circuits (2020)

Two of the paper's three speculative claims (the third is universality across models). Features and the circuits joining them are offered as the units that make a network legible — the alternative to treating it as an irreducible whole.

on Mechanistic interpretability

For instance, we see that clamping the Golden Gate Bridge feature 34M/31164353 to 10× its maximum activation value induces thematically-related model behavior. In this example, the model starts to self-identify as the Golden Gate Bridge!
— Scaling Monosemanticity: Extracting Interpretable Features from Claude 3 Sonnet (2024)

The "Golden Gate Claude" demonstration: a single extracted feature can be turned up, and the model's behaviour changes in a way that names the feature's meaning. The claim is that the feature is a handle on the concept rather than merely a correlate of it — opening and steering the box at once.

on Mechanistic interpretability