The Mythos of Model Interpretability

Zachary C. Lipton was a PhD student in the machine-learning group at the University of California, San Diego when he wrote this, and he later became an assistant professor at Carnegie Mellon. He was already known as a critic of loose terminology and hype in the field, writing the blog Approximately Correct, and the paper is of a piece with that reputation: a demand that a fashionable word be made to mean something before it is required of models.

The essay had a long publication life. It was posted to arXiv in June 2016 and presented that month at the ICML Workshop on Human Interpretability in Machine Learning in New York; the revised, peer-reviewed version appeared in Communications of the ACM in October 2018, a practitioner-facing venue rather than a research-archival one. Its method is conceptual rather than experimental: Lipton observes that "interpretability" is invoked axiomatically, then separates the things people want from it — trust, causality, transferability, informativeness, fair and ethical decision-making — from the model properties they propose to deliver it, which he splits into transparency (simulatability, decomposability, algorithmic transparency) and post-hoc explanation.

The paper sits as a critique and a taxonomy, secondary to the methods it sorts through and aimed at clearing conceptual ground beneath them. Its most consequential move for this topic is an aside that becomes a thesis: humans exhibit none of the forms of transparency the field asks of models, and the processes by which people decide may differ from the processes by which they explain. That observation links it to the human-confabulation literature — Nisbett and Wilson on the absence of introspective access, Haidt on moral reasoning as after-the-fact justification — and forward to Turpin's demonstration that a model's stated reasoning can misrepresent its actual computation. It shares Rudin's distrust of post-hoc explanation while declining to endorse the "interpretable" category she builds on.

Lipton's stake is reputational and intellectual. The piece is a contrarian methodological intervention, the kind that establishes a young researcher as the field's skeptic, and it cuts against rather than toward commercial interest — a paper arguing that "interpretability" is underspecified gives no comfort to vendors selling explainability tools. The argument's weight rests on definitional analysis and citation rather than on data the reader can re-run.