Explainability
The effort to account for what an opaque machine-learning model does after it has done it, usually by fitting a second, simpler method that approximates the original near a given prediction. The model itself stays a closed box; the explanation is added on top rather than built in. This contrasts with making a model understandable from the start, where the system is constrained to be readable in the first place.
The project of producing an account of a black-box model's outputs after the fact — typically through a second, simpler model or method that approximates the original locally — without making the model itself transparent. The founding general-purpose method is Ribeiro, Singh and Guestrin's LIME (2016). Distinguished from interpretability, which constrains the model to be understandable from the start.
Explainability is the post-hoc pole of the machine debate: keep the model, add an explanation. LIME fits a sparse linear surrogate in the neighbourhood of a single prediction to report which features pushed it; SHAP and a large secondary literature followed the same model-agnostic move. The legal counterpart is the counterfactual explanation, which accounts for a decision "without opening the black box."
The standing objection is Rudin's: an explanation is a separate model that can disagree with the thing it explains, which makes it the wrong instrument for consequential decisions. The deeper worry is faithfulness — whether the explanation reflects the model's actual computation — which Turpin and colleagues showed can fail for the step-by-step reasoning of language models. Miller argues the field has built explanations from engineers' intuitions rather than from the study of how people explain to one another.
Discussed in§
- Towards A Rigorous Science of Interpretable Machine Learning
- The Mythos of Model Interpretability
- Explanation in Artificial Intelligence: Insights from the Social Sciences
- "Why Should I Trust You?": Explaining the Predictions of Any Classifier
- Stop Explaining Black Box Machine Learning Models for High Stakes Decisions and Use Interpretable Models Instead
- Counterfactual Explanations Without Opening the Black Box: Automated Decisions and the GDPR