Interpretability

The property of a machine-learning model being understandable to a human — and a goal the field invokes without agreeing on its meaning. Lipton and Doshi-Velez and Kim both argued, around 2016–2017, that the term is underspecified. In Rudin's usage it is sharpened to mean a model transparent by construction, as opposed to a black box made legible after the fact.

Interpretability is the term the machine half of this topic argues over. Lipton separates the things people want from it — trust, the ability to find causes, transferability, informativeness, fairness — from the model properties proposed to supply it, splitting those into transparency (can a human simulate the model, decompose it, follow its algorithm) and post-hoc explanation. Lipton observes that humans exhibit none of the transparency the field demands of models.

Rudin uses the word narrowly and normatively: an interpretable model is one constrained to be understandable from the start, which she argues is the right tool for high-stakes decisions, against the practice of explaining black boxes after they have decided. The disagreement over what interpretability even denotes is part of why Miller turned to the social sciences for an account of what explanation, to a human, actually is.

Discussed in§

• The Intentional Stance
• Towards A Rigorous Science of Interpretable Machine Learning
• Could a Neuroscientist Understand a Microprocessor?
• The Mythos of Model Interpretability
• Explanation in Artificial Intelligence: Insights from the Social Sciences
• Zoom In: An Introduction to Circuits
• "Why Should I Trust You?": Explaining the Predictions of Any Classifier
• Stop Explaining Black Box Machine Learning Models for High Stakes Decisions and Use Interpretable Models Instead
• Scaling Monosemanticity: Extracting Interpretable Features from Claude 3 Sonnet