Explanation faithfulness

faithfulnessunfaithful explanations
the idea

Whether an explanation actually tracks what caused a decision, or merely offers a plausible story the system would have produced anyway. A faithful explanation reports the real reason; an unfaithful one reconstructs a convincing rationale that leaves the true cause unmentioned. The distinction matters for both machines and minds, since a step-by-step justification can sound convincing while being driven by something it never names.

The property of an explanation reflecting the true cause of a decision, rather than offering a plausible reconstruction that the system would give regardless. LIME promised explanations that were "faithful" to the underlying model; Turpin and colleagues (2023) showed that a language model's chain-of-thought reasoning can be systematically unfaithful.

Faithfulness is the property that joins the two halves of this topic, because it can fail in a machine and in a mind the same way. Turpin's experiment plants a bias in a prompt — reordering options so the answer is always "(A)" — and finds the model follows the bias, changes its answer, and writes a confident step-by-step justification that never mentions the bias. The stated reason is a reconstruction, not a report.

That is the same structure the confabulation literature documents in people: Nisbett and Wilson's subjects explain choices by a priori causal theories rather than introspection, and Haidt's account makes moral reasoning a post-hoc justification of an intuition already arrived at. Whether one demands faithfulness of machines that humans never supply is the question Lipton and Dennett press from opposite directions.

Discussed in§

you are here in the territory

3 direct·5 two hops·15 further·46 off-graphopen the full territory →