Selective revision needs a faithful rationale, not just a legible one

Type: kb/types/note.md · Tags: learning-theory, discovery, evaluation

A retained theory is worth more than an equivalent pile of cases because one theory-level revision can change many downstream conclusions at once. That operation needs something the theory's claim does not contain. To narrow "skip integration tests when only documentation files changed" at the premise that actually broke, rather than deleting it, the system must have on hand why it held the theory: which observations supported it, which assumptions it rested on, where it was expected to stop. That record is the theory's rationale.

So the rationale is not commentary attached to a theory. It is the surface revision operates on. A theory without one can be deleted and re-derived; it cannot be repaired.

Legibility is what you get, faithfulness is what you need

A rationale that can be read is legible. A rationale that accurately tracks the basis the theory actually rests on is faithful, and the two come apart exactly where it matters. Jacovi and Goldberg distinguish faithfulness from plausibility: an explanation can convince a reader without representing the process behind the conclusion. Turpin et al. show the behavioral version — controlled input features shift an answer while a fluent chain-of-thought omits the feature that moved it. The same divergence appears in retained artifacts specifically: self-evolving agents' behavior is often not causally grounded in the artifacts they purport to use.

The failure this produces is not symmetric with having no rationale at all, and that asymmetry is the whole claim. With no rationale, a counterexample forces wholesale replacement — expensive, but the broken theory does not survive. With an unfaithful one, the system edits the premise the rationale names, the theory reads as repaired, and the premise that actually carried the failure is untouched. The result is a theory that has been confidently revised in the wrong place, and the revision itself removes the pressure that would have caught it. A wrong rationale is worse than none precisely because it is actionable.

The test is intervention, not inspection

Reading a rationale cannot establish that it is faithful, since reading is what a plausible confabulation is optimized to survive. What separates them is whether operating on the rationale moves the theory's commitments the way the rationale says it should.

Concept bottleneck models supply the worked case. Their concept layer is the rationale made explicit — a legible intermediate the prediction is supposed to run through. Where the concept loss is weighted too low, that layer stays fully readable and keeps competitive accuracy on both the task and the concepts, and intervening on it makes predictions worse. Legible, accurate, and unfaithful at once, with the divergence invisible to inspection and obvious to intervention.

This gives the property an operational shape rather than a rhetorical one: edit the rationale's stated premise and check whether the theory's downstream commitments move as stated. It is the same operation as selective revision. Faithfulness is therefore not a precondition checked before revising — it is what revising correctly consists of, which is why a system that cannot test it cannot claim the revision advantage either.

Scope

  • The claim is about repair, not about whether the theory is good. A faithful rationale for a bad theory supports rejecting it precisely; faithfulness makes revision reliable, not favorable.
  • The same condition governs a second consumer: an overseer deciding about a retained change by reading its rationale rather than probing behavior. That saving inherits the asymmetry — an accepted but unfaithful rationale can cut probing while raising confident error. This note does not develop the oversight case.
  • A recorded rationale is a claim about the theory, not a transcript of how it was formed. Nothing here assumes a system can introspect its own basis; the rationale is an artifact that may be wrong, which is why the intervention test is load-bearing rather than optional.
  • The cited work establishes the faithfulness–plausibility distinction, an intervention-based failure mode in chain-of-thought, and a constructive case in concept bottlenecks. None of it tests prose rationales attached to retained theories, which is the setting this claim is about.

Open Questions

  • Whether the intervention test transfers from a supervised concept layer, where the intermediate is a typed variable, to a prose rationale whose premises are not separately addressable without an interpreter deciding what its premises are.
  • Whether rationales recorded at formation are measurably more faithful than ones reconstructed after the fact, or whether both confabulate often enough that the distinction does not pay for itself.
  • Whether keeping many rationales faithful as their theories drift costs more, at library scale, than re-deriving theories from scratch would.

Relevant Notes: