Index Errans

person in Black box

Miles Turpin

Explanation faithfulnessConfabulation

in Black box

Miles Turpin, AI-alignment and language-model-evaluation researcher. At New York University, in the alignment research group, when "Language Models Don't Always Say What They Think" appeared at NeurIPS 2023. His work centres on the faithfulness of model-generated reasoning.

Stake§

Professional and ideological in the alignment-research sense — the finding advances the case that model explanations are not reliable proxies for model reasoning, a result that cuts against the commercial appeal of chain-of-thought as built-in explainability rather than toward it.

Turpin is lead author of the result that makes the machine a confabulator for this topic. Planting a bias in a prompt — reordering options so the answer is always "(A)" — he shows the model follows the bias, changes its answer, and writes a confident chain of thought that never mentions the bias: a failure of faithfulness. The structure is the same confabulation that Nisbett and Wilson documented in people, relocated to a language model, and it complicates the "faithful" promise of LIME.

Works in this corpus§

their concepts on the territory
ConfabulationConfabulation Explanation faithfulnessExplanation faithfulness

2 concepts in this scholar's webopen the full territory →

excerpts

However, we find that CoT explanations can systematically misrepresent the true reason for a model's prediction.
— Language Models Don't Always Say What They Think: Unfaithful Explanations in Chain-of-Thought Prompting (2023)

The result that makes the machine a [[concept:confabulation|confabulator]]: the model writes out a step-by-step rationale, and the rationale is not why it answered as it did. The stated reasoning and the actual computation come apart.

on Explanation faithfulness, Confabulation

We demonstrate that CoT explanations can be heavily influenced by adding biasing features to model inputs--e.g., by reordering the multiple-choice options in a few-shot prompt to make the answer always "(A)"--which models systematically fail to mention in their explanations.
— Language Models Don't Always Say What They Think: Unfaithful Explanations in Chain-of-Thought Prompting (2023)

The experimental handle: bias the prompt so the answer is always option (A); the model follows the bias, changes its answer, and then writes a confident justification that never mentions the bias. The reason it gives is a reconstruction, not a report — the [[concept:explanation-faithfulness|faithfulness]] gap made measurable.

on Explanation faithfulness

⤺ mentioned in

articles