Julian Michael

Julian Michael, natural-language-processing and AI-safety researcher. At New York University at the time of the chain-of-thought faithfulness paper (NeurIPS 2023), where he worked on evaluation and on debate as a method for supervising models; he later led research at the NYU Alignment Research Group and at Scale AI.

Stake§

Professional and reputational, in the AI-safety and evaluation community; the work supports the case that model-stated reasoning needs checking rather than trusting.

Michael is a co-author of the unfaithful-reasoning result, which bears on this topic by showing that a language model's chain of thought can misrepresent the cause of its answer. His broader work on debate and scalable oversight is about extracting trustworthy judgments from systems whose internal reasoning cannot be taken at face value.

Works in this corpus§

Language Models Don't Always Say What They Think: Unfaithful Explanations in Chain-of-Thought Prompting

¶ excerpts

However, we find that CoT explanations can systematically misrepresent the true reason for a model's prediction.

— Language Models Don't Always Say What They Think: Unfaithful Explanations in Chain-of-Thought Prompting (2023)

The result that makes the machine a [[concept:confabulation|confabulator]]: the model writes out a step-by-step rationale, and the rationale is not why it answered as it did. The stated reasoning and the actual computation come apart.

on Explanation faithfulness, Confabulation

We demonstrate that CoT explanations can be heavily influenced by adding biasing features to model inputs--e.g., by reordering the multiple-choice options in a few-shot prompt to make the answer always "(A)"--which models systematically fail to mention in their explanations.

— Language Models Don't Always Say What They Think: Unfaithful Explanations in Chain-of-Thought Prompting (2023)

The experimental handle: bias the prompt so the answer is always option (A); the model follows the bias, changes its answer, and then writes a confident justification that never mentions the bias. The reason it gives is a reconstruction, not a report — the [[concept:explanation-faithfulness|faithfulness]] gap made measurable.

on Explanation faithfulness