How theory enters an improvement loop
Status: Working model. This note defines the theory-mediated addition used throughout the workshop. Detailed comparisons and controls are in the experiment design.
A proposal-selection improvement loop normally turns evidence into candidate changes, evaluates them, and either retains one candidate or keeps the current system. Theory-mediated learning inserts a scoped, criticizable theory between the evidence and at least one stage or decision in that loop. The theory can guide diagnosis, search, candidate choice, evidence acquisition, or outcome interpretation.
In this workshop, a working theory explains why the system behaves as it does by proposing a mechanism, invariant, or other explanatory relation. The artifact must state its assumptions, scope, and testable consequences. Researchers must record it before the stage or decision it is meant to guide and before they reveal the relevant held-out outcomes. A recommendation or post hoc rationale does not qualify.
The minimal loop
Let tau_n denote the working theory used in episode n. The system may construct it from current evidence or form it by applying a retained theory T_n to the current episode.
current system + objective + evidence
→ construct or retrieve working theory tau_n
→ diagnose and search
→ propose and prioritize candidate changes
→ choose evaluation evidence
→ accept a change or keep the current system
outcomes + audits + working theory tau_n + prior retained theory T_n
→ assess or revise the theory
→ retain T_{n+1} or keep T_n
The first path decides whether to change the system. When the experiment includes theory retention, the second path decides whether to change the retained theory. The paths share evidence but end in separate decisions.
Where theory can help
A working theory can affect four parts of an improvement episode:
- Diagnosis and search. It can explain the observed failure or opportunity, identify a likely intervention point, and direct search away from irrelevant components.
- Candidate proposal and choice. It can derive possible changes, state the premises on which they depend, prioritize them before evaluation, and help rank or reject them afterward.
- Evidence acquisition. It can predict which functions a candidate may affect and which observations would distinguish intended benefits, regressions, and rival explanations. The selective-evaluation model develops this role.
- Outcome interpretation and theory revision. It can explain how a result bears on a premise or scope condition and propose a narrower theory, a broader one, or a replacement for later episodes.
These roles can succeed or fail independently. A theory may locate a useful intervention while missing its regressions. It may identify a discriminating test while generating no useful candidate. Experiments should therefore test each role separately before combining them. A theory-derived prediction remains a claim to test, not additional empirical evidence.
Two decisions, not one
A candidate can work for a reason the theory gets wrong. Conversely, a failed candidate can expose a useful counterexample to the theory. Accepting a system change therefore does not accept its explanation. Rejecting the change need not discard everything learned from the episode.
A reusable theory needs a separate assessment and retention decision. Evidence should warrant only the claims and scope it actually tests: a checked outcome licenses the episode, not an abstracted explanation, and theory warrant should remain as narrow as the evidence.
On-the-spot and retained theories
An on-the-spot treatment constructs tau_n for the current episode and discards it afterward. This treatment tests whether theory mediation improves the current search or decision. If the theory guides a behavioral change that is accepted and later becomes operative, the episode can still count as theory-mediated learning even though the theory itself does not accumulate.
A retained-theory treatment starts from an addressable T_n, records whether the episode retrieves and uses it, and permits the system to retain a revision separately as T_{n+1}. This treatment tests the stronger claim that theory work in earlier episodes can improve later ones. Any benefit must outweigh the costs of retrieval, applicability checking, maintenance, and correction. It must also outweigh the risk that a false retained theory will misdirect several episodes. The theory-mediated self-improvement note develops this retained case.
What the theory is about
Most of this workshop concerns an object theory: an account of how the system will behave under a proposed change. An improvement-process theory instead explains how observations should become retained changes and how the system should later activate, revise, or retire those changes. Both theories can shape behavior, but evidence for one does not support the other. The Exo case develops this distinction.
The experiment design specifies how to distinguish genuine mediation from extra deliberation or post hoc explanation. The workshop README routes the HCL, SPADE, and Exo applications. Those systems supply settings in which to test this proposed addition; they are not evidence for the combined loop.