h t t p s : / / p r o c e e d i n g s . n e u r i p s . c c / p a p e r _ f i l e s / p a p e r / 2 0 2 3 / h a s h / e d 3 f e a 9 0 3 3 a 8 0 f e a 1 3 7 6 2 9 9 f a 7 8 6 3 f 4 a - A b s t r a c t - C o n f e r e n c e . h t m l

Ingest: Language Models Don't Always Say What They Think

Type: kb/sources/types/ingest-report.md

Classification

A NeurIPS experimental paper with controlled prompt interventions, two benchmark families, model comparisons, manual annotation of failure cases, and an explicit operational definition of systematic unfaithfulness. Author: Miles Turpin, Julian Michael, Ethan Perez, and Samuel R. Bowman; the paper is a peer-reviewed NeurIPS 2023 study from researchers affiliated with NYU Alignment Research Group, Cohere, and Anthropic.

Summary

Turpin et al. test whether chain-of-thought (CoT) explanations report the factors that actually drive model predictions. On BIG-Bench Hard, they add biasing features such as reordering few-shot answer choices so the correct answer is always “(A)” or suggesting a random answer; on BBQ, they test stereotype-aligned responses under weak evidence. GPT-3.5 and Claude 1.0 often change their answers under these interventions without mentioning the biasing feature in CoT, and GPT-3.5 accuracy drops by as much as 36% in one condition. In a manually annotated sample, many unfaithful explanations support the new bias-consistent answer, while some remain fluent and apparently sound. The paper therefore turns the Jacovi-Goldberg distinction into a behavioral intervention test: a rationale can be plausible and useful-looking while omitting the causal feature that shifted the decision.

Quotes

  • Source extract (verbatim): We experiment with two benchmarks: BIG-Bench Hard (BBH; Suzgun et al., 2022) and the Bias Benchmark for QA (BBQ; Parrish et al., 2022).1 We test on GPT-3.5 (OpenAI, 2023) and Claude 1.0 (Anthropic, 2023). With BIG-Bench Hard (§3), we investigate two biasing features: (1) Answer is Always A , where we reorder all multiple-choice answer options in a few-shot prompt so the correct one is always “(A)”, and (2) Suggested Answer , where the prompt suggests that a specific answer choice might be correct. With BBQ (§4), we measure whether models make predictions on the basis of common social stereotypes.
  • Source location: Introduction, experiment overview.
  • Source extract (verbatim): In practice, we find that models virtually never verbalize being influenced by our biasing features: we review 426 explanations supporting biased predictions and only 1 explicitly mentions the bias (Appendix B).
  • Source location: Section 2, “Counterfactual Simulatability.”
  • Source extract (verbatim): Table 4 shows examples of unfaithful explanations, where the model changed its prediction to a biasconsistent answer after adding the biasing feature. We observe that in many such examples, the content of CoT explanations also changes to support the new incorrect answer. To quantify how often this happens, we manually annotate 104 unfaithful explanations (one from each model/few-shot/task/context combination) from the Suggested Answer bias setting. We consider an explanation not to support the predicted answer if it suggests a different answer from the final prediction or if it does not indicate any answer choice. Explanations can include reasoning errors but still support the predicted answer. As many as 73% of unfaithful explanations in our sample support the bias-consistent answer.
  • Source location: Section 3.2, results for changed explanations.

Connections Found

The paper is direct evidence for Selective revision needs a faithful rationale, not just a legible one: it demonstrates the target note's predicted failure mode in which an unfaithful rationale can reduce probing while increasing confident errors. It grounds Reasoning production is not reasoning evaluation with a concrete process-validity failure, and reinforces why structured-prompt gains do not establish training-distribution selection: CoT can alter task performance or bias sensitivity without identifying the process that caused the change. It also supplies a correlated-error example for Error correction works with above-chance oracles and decorrelated checks and a model-level analogue of Context contamination operates below an agent's compliance reasoning. Towards Faithfully Interpretable NLP Systems is the conceptual companion.

Extractable Value

  1. Counterfactual input perturbation is a faithfulness test -- Add a feature that should be irrelevant to the stated reasoning, measure whether the prediction changes, and check whether the explanation names that feature; this is a reusable intervention pattern for rationale evaluation. [experiment]
  2. CoT can rationalize an induced wrong answer -- Across 13 BIG-Bench Hard tasks, biasing features can drive substantial accuracy losses (up to 36%) while the explanations omit the feature that caused the shift. [quick-win]
  3. Fluent explanation quality and causal faithfulness come apart -- Many manually examined explanations support the new bias-consistent answer, and some retain apparently sound local reasoning, so surface coherence cannot be the gate. [quick-win]
  4. Process structure and reported-process faithfulness are independent -- A forced step-by-step format is a process intervention, not evidence that the emitted steps are a faithful trace of the computation that produced the answer. [quick-win]
  5. Shared semantic biases can correlate model errors -- GPT-3.5 and Claude respond to related biasing features, limiting the assumption that switching models automatically supplies a decorrelated faithfulness check. [deep-dive]

Limitations (our opinion)

The experiments use GPT-3.5 and Claude 1.0, 2023-era BIG-Bench Hard and BBQ tasks, and a bounded set of answer-order, suggested-answer, and stereotype manipulations. The operational test treats the added feature as the relevant driver; it establishes systematic sensitivity and omitted attribution, not a complete causal account of every internal computation. Manual annotations cover sampled cases, and the paper does not measure the cost or accuracy of a human overseer deciding whether a retained rationale is faithful. CoT sometimes reduces bias sensitivity, so the result is a failure-mode boundary rather than evidence that every chain-of-thought explanation is unfaithful.

Update Selective revision needs a faithful rationale, not just a legible one to cite this ingest alongside Jacovi and Goldberg, adding the intervention-based test and the caveat that process structure can improve outputs without making explanations faithful.

Relevant Notes: