A retained-theory intervention isolates one surface, not the whole program theory

Type: kb/types/note.md · Tags: foundations, self-improving-systems, evaluation

A system may carry program-relevant understanding across explanations, architectural decisions, code, tests, operating records, model competence, and human participants inside the declared boundary. A retained-theory intervention varies a designated explicit theory state while holding specified background components fixed. It tests the contribution of that surface under those conditions, not the removal of every theory-bearing influence.

Explicit retention provides direct targets for selective revision. A claim can be withheld, replaced, or perturbed without deliberately changing the whole system. This convenience does not make the surface exhaustive. Code and records may support reconstruction of the same understanding, and the model or a human participant may supply relevant competence.

What the intervention identifies

At a selected decision, match the model, symbolic state, tools, evidence, task, and resource limits while varying the designated theory. Systematic predicted differences in proposal, diagnosis, evaluation, or recovery support a causal contribution. For stochastic systems, a difference between two individual runs can be sampling variation; repetitions or another justified estimation method must support the claimed effect.

This is stronger evidence than retrieval or citation alone: a citation is a mediation trace, not proof that the cited account determined the decision. The intervention still identifies only the contrast it actually creates, not every interpretation of that contrast.

If the theory is reconstructed from other records, removing its designated text may have no effect. That null result does not show that program theory is useless. It does count against an incremental advantage of keeping this text under those conditions, when the comparison has enough precision and includes reconstruction cost. Equivalent reconstruction is a possible explanation to test, not a reason to make every null result uninformative.

Influence is not yet explanatory guidance

A plausible wrong theory can cause a predicted error because the model obeys its instruction. That establishes influence without showing useful explanatory inference. Extra task facts, a shorter presentation, or a direct hint can also change performance.

Compare explicit theory with raw observations and a descriptive summary of the same observations. Keep access, context allowance, and other treatment features comparable. Test consequences not stated in the text and revision after evidence contradicts an assumption. These contrasts strengthen the explanatory-use interpretation without claiming to observe every internal reasoning step.

Separate assumption-preserving changes from assumption-breaking changes. A correctly applied but inadequate theory can initially cause worse decisions. Measure that misdirection separately from later recovery. A whole-run score can hide both the failure and the acquired correction.

Use, learning, and acquisition require different continuations

Supplying a theory tests its use. A later observation that causes a retained revision tests a further link. To test acquired capacity, resume matched continuations on untouched cases, retaining the revised state in one and restoring its earlier version in the other. State which other carriers remain fixed and which can reconstruct the lesson.

Acquiring a useful theory absent from the seed is a different claim. Preserve the observations from which it could be formed, omit the decisive account, and count the work of constructing and validating it. Success with a supplied theory does not establish that acquisition.

The deployed-system learning unit is wider than this treatment. It can learn through code, tests, tools, schemas, and runtime policy as well as theory text. A null theory-text effect does not exclude those paths, and a positive effect does not establish reliable operation of the complete house.

Scope

The protocol must name the theory version, fixed background, mutable state, outcomes, and comparison conditions. Holding symbolic state fixed is a local intervention choice, not a claim that code cannot carry understanding. During a longitudinal treatment, subsequent state can diverge as each continuation learns; that evaluates the treatment's later effects rather than a single isolated edit. Report failures, interventions, uncertainty, and total cost for the actual claim.

Relevant Notes: