Ethan Perez

Ethan Perez, AI-safety researcher at Anthropic, where he works on red-teaming, reinforcement learning from human feedback, and model-written evaluations. He completed a PhD at New York University before joining Anthropic, and is a co-author of the chain-of-thought faithfulness paper (NeurIPS 2023).

Stake§

Professional and ideological in the alignment sense — the result supports the safety community's position that model self-reports are unreliable, which informs how systems are evaluated rather than how they are sold.

Perez's contribution to this topic is as co-author of the unfaithful-reasoning result. His wider work on adversarial testing and automated evaluation of language models is the practical counterpart to the finding — ways of probing a system from the outside when its stated reasoning cannot be trusted.

Works in this corpus§

Language Models Don't Always Say What They Think: Unfaithful Explanations in Chain-of-Thought Prompting

¶ excerpts

However, we find that CoT explanations can systematically misrepresent the true reason for a model's prediction.

— Language Models Don't Always Say What They Think: Unfaithful Explanations in Chain-of-Thought Prompting (2023)

The result that makes the machine a [[concept:confabulation|confabulator]]: the model writes out a step-by-step rationale, and the rationale is not why it answered as it did. The stated reasoning and the actual computation come apart.

on Explanation faithfulness, Confabulation

We demonstrate that CoT explanations can be heavily influenced by adding biasing features to model inputs--e.g., by reordering the multiple-choice options in a few-shot prompt to make the answer always "(A)"--which models systematically fail to mention in their explanations.

— Language Models Don't Always Say What They Think: Unfaithful Explanations in Chain-of-Thought Prompting (2023)

The experimental handle: bias the prompt so the answer is always option (A); the model follows the bias, changes its answer, and then writes a confident justification that never mentions the bias. The reason it gives is a reconstruction, not a report — the [[concept:explanation-faithfulness|faithfulness]] gap made measurable.

on Explanation faithfulness