Case packet

Neutral case identifier: case-65f16a15010b0b

The possible directed relationship from Artifact A to Artifact B is under review.

Artifact A

Theory-mediated learning may improve sample efficiency under structured shifts

When a task changes, a learner can adapt by fitting new behavior directly to new observations. It can also infer a theory that explains the observations, then reuse or revise that theory when the task changes. Under shifts that preserve some underlying structure, the second route may need fewer target observations.

Call a pathway theory-mediated when it represents a candidate theory as an intermediate object and changes behavior by adopting, applying, rejecting, or revising it. The theory may be natural-language, a causal model, a program, or another representation. What matters is that the learning process can reason about what the theory says, not merely reproduce the behavior it induces.

The conjectured pathway is:

observations → theory search → assessment of explanatory-reach → retained addressable theory → reuse or targeted revision

The expected gain depends on the whole pathway. Addressability alone only makes a theory available for operation; the leverage comes from finding a theory with genuine explanatory-reach, and the capability that selects for that rather than for adaptive fit is [reach-assessment], which no amount of retention machinery supplies on its own.

Explanatory-reach supplies the leverage

A theory has [explanatory-reach] when it captures structure that supports correct conclusions beyond the observations that produced it. That structure may be a causal mechanism, invariant, modular decomposition, compositional rule, or reusable program.

A structured shift changes surface regularities while preserving enough of this structure for an earlier theory to remain useful. The learner may then need to identify only which premise, parameter, component, or applicability condition changed, rather than infer each target behavior independently. One theory-level revision can change many downstream predictions at once.

For example, a coding agent may observe that several documentation-only changes do not require integration tests. It could retain the correlation-shaped rule "documentation files are safe." Or it could infer the theory "a changed file cannot affect integration behavior when no executed process consumes it." If a build tool later begins reading one documentation file as configuration, one failure can support a precise revision: the exemption applies only to files not consumed by executable tooling. The correlation-shaped rule offers no such handle — it can only be deleted or narrowed by enumeration.

This is also the central risk. A broad but wrong theory produces broader negative transfer than a local association does, because it is wrong just as widely as it would have been useful — and [stating where a lesson stops] is a judgment, not something the evidence hands over.

The capability at issue is reasoning about theories

A collection of explicit rules is not enough. The learner must be able to construct candidate theories, derive their consequences, identify their assumptions, compare rival explanations, seek discriminating evidence, and revise a theory's content or scope. Search and assessment stay distinct stages, [since a proposal-selection loop requires search, evaluation, and operative retention]: a system can be starved by its search range while its evaluator is sound, or flooded with candidates its evaluator cannot rank.

LLMs make this pathway practical for theories that have not yet been formalized. They can operate on natural-language candidates: expose an assumption, compare mechanisms, propose a counterexample, or rewrite an applicability condition. How reliably they select theories with genuine explanatory-reach rather than plausible post-hoc stories is an open empirical question, and two recent benchmarks measure adjacent parts of it.

[DiscoverPhysics] places agents in simulated worlds with non-standard laws, has them design experiments on trajectory data, and collects both a natural-language explanation and a Python implementation of the discovered law. Its most relevant result is not the pass rate but the dissociation: predictive accuracy and explanation quality come apart. That is the separation this conjecture depends on, measured directly rather than assumed. [FalsifyBench] runs a Wason-style rule-discovery game across twelve models and reports that success is driven by test selection — models probing to falsify their current hypothesis outperform models probing to confirm it.

Both readings need care, and both captures are abstract-level. If DiscoverPhysics scored explanation quality with an LLM judge, its dissociation is a hard oracle disagreeing with a soft one rather than a measurement of understanding; and its worlds, while counterfactual as instances, are drawn from a space published physics already discusses. FalsifyBench's headline may be partly definitional: on 2-4-6, where the hidden rule is deliberately broader than the seed invites, probes outside the current hypothesis are also the only informative probes, so the correlation could be information gain rather than an epistemic disposition. Neither benchmark tests reuse of a retained theory across a controlled shift — which is exactly the gap the test design below proposes to fill — but together they establish that theory search and its assessment can be measured separately from predictive fit at all.

Purely parametric retention exposes no scope

A parametric learner can acquire reusable structure. Features learned in one network transfer to related tasks ([Yosinski et al. 2014]), meta-learned initializations adapt from a handful of examples ([Finn, Abbeel, and Levine 2017]), and gradient updates can produce behavior equivalent to revising a theory. The claim cannot be that weights contain only correlations or that they cannot encode theories.

The distinction is narrower and architectural. Where retention is purely parametric and the learning pathway exposes no semantic interface to what was retained, an individual theory is not a first-class update target. The system may acquire equivalent behavior, but it cannot deliberately retrieve one theory, compare it with a rival, replace one of its premises, or narrow its scope while preserving unrelated commitments; the extent and collateral effects of an update have to be discovered behaviorally. A theory is addressable when the system can identify it as a stable semantic unit and inspect or revise its relevant parts — and addressability comes in degrees, since a theory addressable only as an indivisible document can be replaced or deleted but not rescoped by one premise, which requires its content, assumptions, and applicability conditions to be separately reachable.

Two lines of work show this is a claim about a default rather than an impossibility, and the shape of their shortfall is more informative than the fact that they exist. [ROME] edits particular factual associations inside transformer weights, recovering an interface after training. What it delivers is content-addressability without scope-addressability: a rank-one edit revises a stored association but carries no applicability boundary, so rescoping remains unavailable. [Concept bottleneck models] take the other route, supervising a legible concept layer into existence before training so it can be intervened on — but the intervention is per-inference and nothing is retained, and where the concept loss is weighted too low the layer stays readable and competitive while interventions on it make predictions worse.

So the contrast is not neural versus symbolic, nor computational expressivity. It is purely parametric adaptation versus learning that can operate on theories as theories, and the axis it runs along is [representational form]. An LLM using retained natural-language theories is a hybrid on that axis: the weights supply the semantic interpreter, while the retained theories supply persistent identity and selective revisability.

Formalization buys a mechanical acceptance test

Symbolic systems implement the same pathway wherever the theory space and acceptance criteria are formalized: causal discovery searches for structure whose consequences survive specified interventions, and [DreamCoder] grows a library of reusable program abstractions that compound across a task family.

DreamCoder also shows what formalization buys, and it is the acceptance test rather than the search. Its retention gate is statistical — an abstraction is kept when it reduces description length — so the decision to retain is mechanical. Nothing plays that role for a theory still in natural-language form, where stating where the theory stops remains a judgment, which is why the natural-language route's advantage rests on a capability with no formal proxy. The cost of the formal route is a supplied language, variables, primitives, search procedure, and acceptance test, [worked out in the causal and proof obligations that formal systems check]. LLMs take the opposite trade: a far more open-ended theory space before formalization, with weaker guarantees.

What the negative results actually bound

Generic "off-distribution" predicts nothing, since some shifts destroy every regularity any system could have retained. The conjecture is conditional on structured shifts, and the transfer literature supports that conditional shape and no more: reusable causal mechanisms are proposed as what survives intervention-like change ([Schölkopf et al. 2021]), cross-environment stability is proposed as evidence for causal predictors ([Peters, Bühlmann, and Meinshausen 2016]), and speed of adaptation to such change can be made a training signal ([Bengio et al. 2019]).

Generalization under shift is hard and assumption-dependent for every method. Under the benchmarks and model-selection procedures studied in [DomainBed], the evaluated domain-generalization methods did not consistently beat carefully implemented empirical risk minimization. [Rosenfeld, Ravikumar, and Risteski] bound invariant risk minimization specifically: in the linear regime it needs more environments than the spurious-feature dimension, and outside it the objective permits a predictor that mimics the invariant one on training data and reverts to ERM at test, with a penalty exponentially small in that dimension. That second result is an existence proof about what the objective allows, not evidence that gradient training finds such a solution — a distinction the paper's abstract invites readers to lose, and one that keeps this result compatible with invariance-as-hypothesis-test rather than contradicting it.

The bitter lesson is often read as a third objection here, and it is not one — [what scale selects against is unearned reach, not structure], so a system that searches theory space and tests its candidates is running the general method rather than supplying a prior. That clearance is conditional on the acceptance test doing real work, which is the condition this whole note turns on. What survives is a narrower objection aimed at the retention layer: a sufficiently scaled system might run the same search implicitly in activations, with no externalized addressable object anywhere, making the artifact scaffolding rather than a load-bearing part of the pathway. Nothing here rebuts that, and [where the artifact has to earn its place against a longer context] is where the objection bites hardest.

Cost can erase the gain

The sample-efficiency hypothesis concerns new target observations only. Distinct from it is whether the advantage survives full accounting: theory discovery, [codification], retrieval, applicability checking, validation, application, maintenance, and correction on one side; pretraining, adaptation data, optimization, and evaluation on the other. Folding every entry into "sample efficiency" conflates the two.

The boundary must be drawn symmetrically. Counting the historical evidence that produced an explicit theory as free, while charging the parametric learner for all its training data, biases the comparison before it starts. Retrieval deserves separate emphasis: a theory nothing surfaces at the moment of need contributes nothing however good it is, while parametric retention avoids a separate retrieval step by residing in the operative substrate — which does not guarantee the relevant behavior activates or generalizes in context. Its discount is differently shaped, not absent. And validation has more routes than criticism: proof, simulation, model checking, causal analysis, counterexample generation, targeted experiment. Criticism is the judgment-heavy route whose reliability bounds unattended use, [since warranted autonomy is bounded by oracle domain].

What would test the conjecture

Matching "informational content" between an explicit arm and a separately trained parametric arm is probably not operationalizable, and is not needed. Hold the same base model, the same source observations, and the same inference budget fixed, and vary only the pathway:

  • an explicit theory with stated assumptions and scope, revisable at the theory level;
  • raw episodic memory of the same source observations, with no synthesized theory;
  • an explicit theory that may only be appended to, never rejected or rescoped;
  • the same pathway with the evaluator ablated to surface matching, removing reach-assessment;
  • parametric adaptation from the same observations;
  • a formal causal-discovery or program-synthesis baseline.

Then apply controlled shifts of distinct kinds: surface change preserving the mechanism; intervention preserving some causal modules; change invalidating exactly one applicability condition; change invalidating the theory entirely; and a deceptive correlation that held across every source observation. Target accuracy alone will not separate the arms. Measure target observations needed to recover performance, whether the correct mechanism was identified, scope calibration, whether one counterexample triggers proportionate rescoping rather than wholesale replacement, collateral damage to unaffected cases, false transfer from an overbroad theory, and the cost of discovery and validation.

The decisive prediction is an interaction, not a level:

A correct explicit theory should reduce target observations when the shift preserves the structure it names, while producing less collateral change outside that theory's scope. The advantage should disappear or reverse when the theory is wrong, when the shift breaks the structure, and — the sharpest arm — when the evaluator is ablated, since that arm keeps addressability and loses only reach-assessment.

A generic accuracy gain would not establish the mechanism. Every arm is a retention pathway wrapped around a fixed base model on an ordinary domain task, so the design needs no special architecture — only controlled shifts and honest measurement.

Open Questions

  • Whether a task family with controlled structured shift can be exhibited where the theory-mediated pathway measurably reaches fixed performance on fewer target observations — and whether one can be exhibited where it measurably fails to.
  • Whether the evaluator-ablation arm separates cleanly, or whether a base model capable enough to apply a theory also reach-assesses it implicitly and cannot be ablated without crippling the other arms.
  • Whether hybrid pathways — parametric adaptation guided by retrieved explicit theories — dominate both pure pathways, turning the contest into an engineering question about composition.
  • Whether a theory's explanatory-reach can be estimated from its addressable form before any shift tests it, or whether legibility supplies only a handle on reach established some other way.
  • Whether validating a theory's reach against an informally specified shift is itself target-data-free, or quietly consumes the observations the conjecture claims to save.
  • Whether discovery, codification, validation, and maintenance cost for a library of many theories grows faster than the smooth cost curve of parametric scaling as task families accumulate.
  • Why LLM-mediated evaluators appear able to reach-assess claims still in natural-language form, which no current theory explains and on which this pathway's advantage rests.

Relevant Notes:

Artifact B

A proposal-selection improvement loop requires search, evaluation, and operative retention

A proposal-selection improvement loop is the architecture of improvement in which candidate changes are generated, evaluated with a possibility of non-adoption, and selectively made operative. It is a named subtype, not the whole of the phenomenon: a [self-improving system] needs its changes to be responsive to evidence bearing on an improvement objective, and evidence may instead determine an update directly — gradient-, reward-, error-, or viability-driven — with no candidate ever standing to be rejected. What follows is the anatomy of the subtype, and it applies with full force exactly there.

A proposal-selection loop requires three functions: search brings a candidate change into consideration, evaluation supplies grounds for accepting or rejecting it, and operative retention preserves an accepted change with behavioral authority. Remove any one and the loop does not close — a change nobody proposed, nobody could reject, or nobody will ever act on.

The loop is therefore narrower than self-modification. A blind, accidental, or unconditional rewrite may change later behavior without applying any criterion; a transient rewrite may fail to preserve the result. Both can count as self-modification, but neither closes a proposal-selection loop. Conversely, the three functions can close the loop in a system that is not reflective at all.

A terminology note: the concept descends from Ashby's adaptation — his ultrastable system, examined below, is the conceptual ancestor even though it classifies outside the subtype — but it is named for what the loop aims at rather than by his word for it. Everyday adaptation is transient compensation, an eye adjusting to the dark, and retains nothing; retention is one of the three requirements. Where this note says adaptation or adaptive, it means Ashby's phenomenon. The architecture described here is named proposal-selection throughout.

A [reflective system] supplies one possible causal path into this loop. Through intercession — an operation that changes the system through its causally connected self-representation — it can modify a represented aspect of itself. Making that path available does not itself provide search, evaluation, or retention.

The independence runs both ways. A directly determined update can land on a self-representation as readily as on an opaque substrate — evidence can revise an explicit policy or a recorded lesson with nothing rejectable anywhere in the path — so neither architecture is the general form of reflective improvement.

Search determines what enters consideration

Search brings an unrealized change under consideration. It may include:

  • detecting a problem, opportunity, or adaptation signal;
  • selecting the aspect and operation to change;
  • generating one or more candidates;
  • allocating effort and deciding when to stop or escalate.

At minimum, search must produce a candidate from a space in which other possible changes remain unrealized. It need not compare several candidates at once or operate autonomously. A maintainer may choose the problem, a model may draft a candidate, and a script may enumerate alternatives within one declared socio-technical loop. Assigning those functions establishes the loop's boundary; it does not make the loop reflective.

Search range and evaluation strength are independent limits:

Evaluation cannot select a candidate that search never reaches.

A strong verifier can improve judgments within a narrow generator's range, but it cannot expand that range. [Automating KB learning is an open problem] gives one concrete search space—extract, split, synthesize, relink, regroup, reformulate, retire—whose judgment-heavy parts remain substantially human-driven.

Evaluation determines which changes may remain operative

Evaluation applies criteria to a proposed or already actualized change. Its result must be able to affect selection, rollback, or continued retention. Evaluation is non-vacuous only if some possible result permits rejection: an unconditional trigger is not an evaluator merely because it precedes a transition, and a conditional trigger whose only effect is to launch the next variation is not one either. The verdict must control an operation distinct from producing the next candidate — select, discard, block, roll back — so that rejecting a change and merely changing again are different events in the mechanism.

Oracle is shorthand for the component or procedure that supplies the evidence or judgment. It may be a proof system, test, validator, empirical measurement, rubric, model evaluator, human review, or some combination. The [oracle-strength spectrum] grades these mechanisms, while [the boundary of automation is the boundary of verification] explains why constructing an adequate oracle is often harder than generating candidates.

Any judgment remains scoped to what the check establishes. An oracle may accept a candidate under specified criteria without establishing that the change is globally beneficial. Search and evaluation may be performed by the same person or process, but they fail in different ways and improve by different means. They are analytically separable rather than independent: automating one changes the load on the other.

Operative retention makes the change consequential

Acceptance alone does not make a change consequential. Operative retention combines persistence with an authority path through which the retained result can affect later behavior. In [behavioral authority] terms, the change needs a consumer, a channel, and a force.

  • A reviewed note that no future reader or prompt-assembly step loads has no consumer.
  • An approved patch that is never merged has no channel.
  • A generated validator that no command invokes has no force.

In each case, search ran and evaluation passed, but the proposal-selection loop remained open: the artifact exists without becoming behaviorally consequential.

Artifact labels do not decide whether retention is operative. A knowledge artifact consumed as evidence or advice can affect later behavior, while a nominal system-definition artifact with no consumer cannot. The test is the [behavioral-authority] path: consumer, channel, and force relative to the objective and declared horizon.

For self-improvement, the accepted change must reach the system's own [behavior-determining organization]. Promotion into instruction, enforcement, or configuration is one way to strengthen that path, and may itself run as another proposal-selection instance — [the two-layer execution system] develops that promotion architecture, with recurrence as the trigger, pre-promotion verification as the gate, and methodology growth plus a coverage-test update as retention — but it is not universally required for reflective or operative change.

Repetition does not establish cumulativity

A proposal-selection loop can repeat on a timer or fresh request without using anything retained by an earlier iteration. Whether later improvement consumes or preserves earlier improvement-relevant information is cumulativity, whose criterion and counterexamples belong to [the informational-dependence test on the retained result]. Retained rationale can provide that dependence when later search or evaluation actually consumes it; [design rationale management in Commonplace] documents that path.

Boundary cases clarify the claim

Cybernetician W. Ross Ashby's ultrastable system marks the subtype's edge from just outside it, and its exclusion follows from the evaluation criterion above, not from a missing component. The electromechanical Homeostat has exactly one evidence-responsive transition: when essential variables leave viable bounds, the parameters jump to new random values ([Ashby 1960, chapters 7–8]). That single jump both discards the incumbent configuration and produces its successor — rejection is not an operation distinct from generation — and a configuration that restores viability persists through equilibrium, with nothing whose function is to accept it. The functions collapse into one trigger, so under the definitions here the machine is a non-reflective, direct viability-driven [self-improving system], not an instance of this subtype.

What the Homeostat does admit is a functional variation–selection–retention reading: configurations vary, viability determines whether variation continues, and the survivor persists through non-displacement. That reading is an analyst's reconstruction, not architecture, and its value is to mark the floor of each function — search as a draw from a random-number table bearing no relation to the problem, evaluation as a one-bit viability boundary that ranks nothing, retention as equilibrium, a configuration surviving because nothing is left to displace it. Read this way, the Homeostat is the cheapest demonstration of what a stronger generator and a real oracle actually buy. Reflection is still not a premise of the decomposition. An evolutionary strategy supplies the genuine non-reflective instance: it runs an explicit generate-and-select loop over parameters nothing inside it can read.

The Homeostat's contrast with a gated system is architectural, not a difference of gate strength: a gradient learner has no evaluator either — [online gradient descent] adopts every step the revealed cost dictates, with no accept/reject anywhere (Zinkevich 2003) — and Zinkevich's Greedy Projection/GIGA result supplies the technical counterexample to treating an acceptance gate as universal. The Homeostat stands with it, on the excluded side of the boundary just drawn. [Gödel machines] sit inside the subtype at its formal extreme, a proof-mediated gate rather than none at all; that architecture is developed in their own note.

Reflection is a separate axis from this exclusion: the Homeostat is also non-reflective, and [what that costs is addressability, not category membership] — evidence-responsive operative change to the system's own organization, with or without a self-representation and with or without a gate, is what makes a [self-improving system].

What the decomposition claims

The three functions are analytically separable, not architecturally separate. One process may perform several of them — a maintainer who notices a problem, drafts the fix, and merges it performs all three — and evaluation may run before a candidate becomes operative or after. Co-location has a floor, though: the functions must remain causally distinguishable even when one process performs them — rejection, in particular, must be an event distinct from the arrival of the next candidate. Where they collapse into a single evidence-triggered transition, as in the Homeostat, the loop is not weakly present; it is absent, and the pathway is direct. The decomposition specifies what the loop must accomplish, not a sequence, a component diagram, or a division of labour. Its use is diagnostic: when a loop stalls, ask which of the three is missing rather than which component failed.

The status claimed here matches how the neighboring self-adaptive-systems field treats its own loop models: MAPE-K — introduced in [Kephart and Chess's autonomic-computing vision], which itself supplies no membership test — and its relatives are presented as reference models for engineering adaptation, not as the definition of it ([Weyns, Software Engineering of Self-Adaptive Systems]), and a systematic review of that literature finds no settled formal definition from which any single loop architecture would follow ([Petrovska, Erjiage, and Kugele 2025]). The proposal-selection decomposition is offered in the same spirit — a conceptual model of one architecture, with the [category membership question] settled elsewhere.

Open Questions

  • Whether search range can be measured or bounded for a socio-technical loop in the way oracle strength can be graded.
  • Whether a fallible evaluator can govern changes to its own acceptance criteria without either an external criterion or the axiomatization that buys formal closure.

Relevant Notes:

Under-review context phrase

the search/evaluation split that keeps theory generation distinct from reach-assessment