Theory refinement may improve sample efficiency under structured shifts

Type: kb/types/note.md · Tags: learning-theory, discovery

When a task changes, a learner can adapt by fitting new behavior directly to new observations. It can also learn model-based: infer a model that explains the observations, then reuse or revise that model when the task changes. Under shifts that preserve some underlying structure, the second route may need fewer target observations.

This note concerns theory refinement: the case where the model is a theory in the sense of the definition, with consequences a case can contradict and parts available for candidate repairs, and the learner keeps it, reuses it, and revises it when evidence contradicts it. The learner can derive what the theory implies, expose an assumption, compare a rival, or narrow a scope, rather than only reproduce the behavior the model induces. The theory may be natural language, a causal model, a program, or another representation.

The conjectured pathway is:

observations → theory search → assessment of explanatory-reach → retained addressable theory → reuse or targeted revision

This pathway contains two separable hypotheses. Reuse and revision: retaining a useful theory and selectively revising it may reduce target observations under structured shifts. Selection: among candidates with comparable fit to the source evidence, preferring estimated explanatory-reach may select theories that transfer better. Reach-assessment is the proposed capability behind the second hypothesis. A useful theory can deliver the first benefit even if it was supplied or selected without reach-assessment; retention alone does not establish the second.

Explanatory-reach supplies the leverage

A theory has explanatory-reach when it captures structure that supports correct conclusions beyond the observations that produced it. That structure may be a causal mechanism, invariant, modular decomposition, compositional rule, or reusable program.

A structured shift changes surface regularities while preserving enough of this structure for an earlier theory to remain useful. The learner may then need to identify only which premise, parameter, component, or applicability condition changed, rather than infer each target behavior independently. One theory-level revision can change many downstream predictions at once.

For example, a coding agent may observe that several documentation-only changes do not require integration tests. It could retain the correlation-shaped rule "documentation files are safe." Or it could infer the theory "a changed file cannot affect integration behavior when no executed process consumes it." If a build tool later begins reading one documentation file as configuration, one failure can support a precise revision: the exemption applies only to files not consumed by executable tooling. The correlation-shaped rule does not state the dependency condition, but the learner could still discover it from the failure and revise the rule. Retaining the explanation supplies a candidate basis for diagnosis; whether it saves observations compared with reconstructing that basis is part of the conjecture.

This is also the central risk. A broad but wrong theory produces broader negative transfer than a local association does, because it is wrong just as widely as it would have been useful — and stating where a lesson stops is a judgment, not something the evidence hands over.

The capability at issue is reasoning about theories

Testing the whole pathway requires more than storing explicit rules. The learner needs operations for constructing candidates, deriving consequences, identifying assumptions, comparing explanations, seeking discriminating evidence, and revising content or scope. The reuse hypothesis can instead be tested with a supplied theory, isolating its use and revision from discovery and selection. Search and assessment stay distinct stages, since a proposal-selection loop requires search, evaluation, and operative retention: a system can be starved by its search range while its evaluator is sound, or flooded with candidates its evaluator cannot rank.

LLMs make this pathway practical for theories that have not yet been formalized. They can operate on natural-language candidates: expose an assumption, compare mechanisms, propose a counterexample, or rewrite an applicability condition. For a natural-language theory, though, what the theory implies is what the interpreter says it implies, so a contradiction is a reading and a located defect is a judgment. The classical loop ran three inference modes over a rule set: deduction to apply it, abduction to hypothesize the missing premise behind an unexplained case, induction to fill the gap. In the LLM realization all three are the model's judgments, and abduction is the exposed one, since it is where new content enters and a plausible post-hoc story is easiest to mistake for a located defect. How reliably LLMs select theories with genuine explanatory-reach rather than such stories is an open empirical question, and two recent benchmarks measure adjacent parts of it.

DiscoverPhysics places agents in simulated worlds whose laws deliberately differ from our universe's physics, has them propose experiments and observe trajectory data, and collects both a natural-language explanation and a Python implementation of the discovered law. Its abstract-level capture reports that "Strong predictive accuracy doesn't guarantee quality conceptual explanations" (DiscoverPhysics, Findings, verbatim). The capture does not reveal how explanation quality was scored or quantify the dissociation. FalsifyBench runs a Wason-style rule-discovery game across twelve models. Its abstract reports that "The primary driver of success is the capacity for negative testing" (FalsifyBench, abstract, verbatim): models that test to falsify a hypothesis outperform models that seek confirmation.

Both readings need care. DiscoverPhysics's captured result separates two submitted outputs inside one benchmark, not theory search from reach-assessment in general. FalsifyBench's headline may be partly definitional: on 2-4-6, where the hidden rule is deliberately broader than the seed invites, probes outside the current hypothesis are also the only informative probes, so the correlation could be information gain rather than an epistemic disposition. Neither benchmark tests reuse of a retained theory across a controlled shift — which is exactly the gap the test design below proposes to fill — but together they show adjacent behaviors that such a test can separate from predictive fit.

Purely parametric retention exposes no scope

A parametric learner can acquire reusable structure. Features learned in one network transfer to related tasks (Yosinski et al. 2014), meta-learned initializations adapt from a handful of examples (Finn, Abbeel, and Levine 2017), and gradient updates can produce behavior equivalent to revising a theory. The claim cannot be that weights contain only correlations or that they cannot encode theories.

The distinction is narrower and architectural. Where retention is purely parametric and the learning pathway exposes no semantic interface to what was retained, an individual theory is not a first-class update target. The system may acquire equivalent behavior, but it cannot deliberately retrieve one theory, compare it with a rival, replace one of its premises, or explicitly target a scope condition. The extent and collateral effects of an update have to be discovered behaviorally. Explicit theories make candidate edits identifiable too, but their locality does not guarantee preservation of other behavior: shared premises can affect many conclusions, so regression checks remain necessary. A theory is addressable, in the sense the definition requires, when the system can identify it as a stable semantic unit and inspect or revise its relevant parts — and addressability comes in degrees, since a theory addressable only as an indivisible document can be replaced or deleted but not rescoped by one premise, which requires its content, assumptions, and applicability conditions to be separately reachable.

Two lines of work show this is a claim about a default rather than an impossibility, and the shape of their shortfall is more informative than the fact that they exist. ROME applies a rank-one update to transformer weights to change a selected factual association — "factual associations correspond to localized, directly-editable computations" (ROME, Overview, verbatim) — and reports specificity and counterfactual-generalization tests. Calling this content-addressability without scope-addressability is the comparison here: the edit target is addressable, but the edited artifact exposes no premise or applicability boundary to rescope. Concept bottleneck models take the other route, supervising a legible concept layer into existence before training: "By construction, we can intervene on these concept bottleneck models by editing their predicted concept values and propagating these changes to the final prediction" (Concept bottleneck models, abstract, verbatim). The correction is per-inference rather than a persistent theory revision. The paper also shows that task and concept accuracy alone do not predict intervention benefit; with too little concept-loss weight, correction can increase error.

So the contrast is not neural versus symbolic, nor computational expressivity. It is purely parametric adaptation versus learning that can operate on theories as theories, and the axis it runs along is representational form. An LLM using retained natural-language theories is a hybrid on that axis: the weights supply the semantic interpreter, while the retained theories supply persistent identity and selective revisability.

Formalization can buy a mechanical acceptance test

Symbolic systems can implement the same pathway wherever the theory space and acceptance criteria are formalized: causal discovery searches for structure whose consequences survive specified interventions, and DreamCoder iteratively extends a domain-specific language with reusable symbolic abstractions while training neural search guidance.

Formalization can make acceptance mechanical when the system supplies an explicit retention objective or proof obligation. The current checksum-pinned DreamCoder snapshot is abstract-only: it supports iterative library growth and compositional reuse, but it does not state a description-length criterion or a discrete retention gate. DreamCoder therefore cannot ground that stronger example until a fuller primary capture is available. The general cost of the formal route remains a supplied language, variables, primitives, search procedure, and acceptance test, worked out in the causal and proof obligations that formal systems check. LLMs take the opposite trade: a far more open-ended theory space before formalization, with weaker guarantees.

What the negative results actually bound

Generic "off-distribution" predicts nothing, since some shifts destroy every regularity any system could have retained. The conjecture is conditional on structured shifts, and the transfer literature supports that conditional shape and no more: reusable causal mechanisms are proposed as what survives intervention-like change (Schölkopf et al. 2021), cross-environment stability is proposed as evidence for causal predictors — "we collect all models that do show invariance in their predictive accuracy across settings and interventions. The causal model will be a member of this set of models with high probability" (Peters, Bühlmann, and Meinshausen 2016, abstract, verbatim) — and speed of adaptation to such change can be made a training signal (Bengio et al. 2019).

Generalization under shift is hard and assumption-dependent for every method. Under the benchmarks and model-selection procedures studied in DomainBed, the evaluated domain-generalization methods did not consistently beat carefully implemented empirical risk minimization. Rosenfeld, Ravikumar, and Risteski bound invariant risk minimization specifically: in their linear Gaussian setting it needs more environments than the environmental-feature dimension, and with nonlinear featurizers the objective permits a predictor that is near-identical to the invariant one on the training distribution and reverts to ERM on most sufficiently shifted test points, with an exponentially small penalty. That second result is a formal existence construction. The paper also reports confirming synthetic fits from its own model, but not how often practical optimization reaches the constructed failure outside that setting. This keeps the result compatible with invariance-as-hypothesis-test rather than contradicting it.

The bitter lesson is often read as a third objection here. The linked case-level conjecture diagnoses a structured method's loss as a requirement-to-objective proxy used beyond its assessed scope, and it disclaims any inverse guarantee that assessed structure survives scaling — so the clearance below is this note's own conjecture rather than a result carried over: a system that searches theory space and tests its candidates is running search and learning rather than supplying a hand-specified prior. That clearance concerns the effectiveness of the complete search-and-selection procedure; the benefit of reusing a supplied useful theory can be tested separately. What survives is a narrower objection aimed at the retention layer: a sufficiently scaled system might run the same search implicitly in activations, with no externalized addressable object anywhere, making the artifact scaffolding rather than a load-bearing part of the pathway. Nothing here rebuts that, and where the artifact has to earn its place against a longer context is where the objection bites hardest.

Cost can erase the gain

The sample-efficiency hypothesis concerns new target observations only. Distinct from it is whether the advantage survives full accounting: theory discovery, codification, retrieval, applicability checking, validation, application, maintenance, and correction on one side; pretraining, adaptation data, optimization, and evaluation on the other. Folding every entry into "sample efficiency" conflates the two.

The boundary must be drawn symmetrically. Counting the historical evidence that produced an explicit theory as free, while charging the parametric learner for all its training data, biases the comparison before it starts. Retrieval deserves separate emphasis: a theory nothing surfaces at the moment of need contributes nothing however good it is, while parametric retention avoids a separate retrieval step by residing in the operative substrate — which does not guarantee the relevant behavior activates or generalizes in context. Its discount is differently shaped, not absent. And validation has more routes than criticism: proof, simulation, model checking, causal analysis, counterexample generation, targeted experiment. Criticism is the judgment-heavy route whose reliability bounds unattended use, since warranted autonomy is bounded by oracle domain.

What would test the conjecture

Use two experiments so that the contribution of a useful theory is distinguishable from the contribution of its selector.

Reuse and revision experiment. Start with the same base model and source observations, and hold the inference budget fixed. Generate a source-derived theory once and keep its initial content fixed across the explicit-theory arms. Compare retaining and revising it with retaining it unchanged or allowing additions only. Include raw episodic memory from the same observations as a baseline. This tests the contribution of the retained synthesis and, separately, the ability to revise it. If an experimenter supplies a theory from knowledge unavailable in the source observations, report that arm as a controlled capability test; it does not establish an end-to-end learning advantage. Parametric adaptation and formal causal-discovery or program-synthesis systems can provide additional comparisons, with their training and search costs reported separately.

Selection experiment. Give each selector the same candidate theories, source evidence, and selection budget. Compare a selector that prefers estimated explanatory-reach among comparably fitting candidates with selection by source fit alone, using a declared tie-break rule. Keep the downstream application and revision procedure fixed. The selector sees no held-out target outcomes; score its choices on controlled shifts afterward. Include candidate sets with similar source fit but different transfer behavior, so that source fit alone does not settle the choice. This tests whether an LLM's reach judgments add useful selection information beyond source fit. It does not assume that changing an instruction removes all implicit reach-assessment from the model.

Apply controlled shifts of distinct kinds: surface change preserving the mechanism; intervention preserving some causal modules; change invalidating one applicability condition; change invalidating the theory entirely; and a deceptive correlation that held across every source observation. Measure target observations needed to recover performance, mechanism identification, scope calibration, collateral damage to unaffected cases, false transfer, and discovery and validation costs. Evaluate both experiments over repeated task and candidate sets; a lucky theory choice cannot establish a selector's reliability.

The predictions differ:

  • Reuse and revision: a useful retained theory should reduce target observations relative to episodic memory on shifts that preserve its structure. Where a scope condition changes, permitting targeted revision should improve adaptation relative to freezing that same theory. Incorrect or invalidated theories test the risk of negative transfer; they need not defeat every repair procedure.
  • Selection: reach-based selection should choose theories with better held-out transfer, or less harmful overgeneralization, than source-fit selection under the matched conditions. A null or negative result would count against this selector's proposed benefit in the tested setting, even if useful theories still help once supplied.

Removing the explicit reach-based selector therefore need not erase the reuse benefit. A generic accuracy gain from the full pathway would establish neither contribution separately.

Open Questions

  • Whether a task family with controlled structured shift can be exhibited where the theory-refinement pathway measurably reaches fixed performance on fewer target observations — and whether one can be exhibited where it measurably fails to.
  • Whether the declared selection policies produce measurably different choices, and how much implicit reach-assessment remains in the source-fit comparator or downstream revision procedure.
  • Whether hybrid pathways — parametric adaptation guided by retrieved explicit theories — dominate both pure pathways, turning the contest into an engineering question about composition.
  • Whether a theory's explanatory-reach can be estimated from its addressable form before any shift tests it, or whether legibility supplies only a handle on reach established some other way.
  • Whether validating a theory's reach against an informally specified shift is itself target-data-free, or quietly consumes the observations the conjecture claims to save.
  • Whether discovery, codification, validation, and maintenance cost for a library of many theories grows faster than the smooth cost curve of parametric scaling as task families accumulate.
  • Whether, and under which conditions, LLM evaluators rank natural-language theories by explanatory-reach reliably enough to improve held-out transfer beyond source-fit selection; if they do, what accounts for that capability.

Relevant Notes: