A search controller is tested by what it brings to stronger evaluation
Type: types/note.md · Tags: evaluation, self-improving-systems, improvement-loop, learning-theory, goal-prioritization
A search controller decides which branches or probes receive further work. It does not itself establish that those branches are correct or worth adopting. Its evaluation should therefore follow what it routes into stronger evaluation.
A useful controller brings forward branches that later produce valuable candidates, discriminating evidence, informative failures, or improved recovery. A false lead does not individually refute it, just as one successful branch does not establish it. The comparison concerns the distribution of consequences produced by its routing decisions.
The controller can be compared with alternatives under matched tasks, resources, and downstream evaluation. This does not require knowing the best branch in the full counterfactual search space, which is normally unavailable in the open-ended setting.
Two refinements follow from the decision-theoretic account of the value of computation (Russell and Wefald). A routed branch or probe earns credit only when it changes a later decision; one that raises confidence without changing a choice brings nothing forward and still costs what it cost. And where two controllers' costs differ materially, as they often do when controllers run unattended, the matched-resources comparison can be joined by one on net value, benefit minus cost on one scale, reported across a range of exchange rates rather than at one assumed rate. Matched-resources comparison stays valid in every regime; the net-value form is an added option, not a replacement, and it needs cost per controller run to be recorded.
Scope
- Stronger evaluation may be immediate, delayed, empirical, or formal. The claim does not select one evaluator.
- Search-controller quality does not establish that any accepted result was adequately warranted; acceptance remains a separate judgment.
Relevant Notes:
- Lightweight search control allocates further search without licensing adoption — grounds: supplies the limited-authority controller whose output is being evaluated
- Open-ended improvement must allocate search before decisive evaluation is available — grounds: explains why comparison cannot assume exhaustive evaluation of every branch
- Backtracking keeps lightweight search control provisional — mechanism: explains how a mistaken branch can still improve recovery while remaining provisional
- A failure explanation becomes search control only when it changes a later branch decision — mechanism: identifies when an informative failure actually changes subsequent routing
- Diagnostic richness constrains outer-loop learning quality — grounds: explains why inspectable evidence and failures are valuable downstream consequences apart from acceptance
- An experiment identifies only the contrast it actually runs — grounds: limits controller attribution to comparisons that vary the controller rather than a larger treatment bundle
- Principles of Metareasoning (Russell and Wefald, 1991) — evidenced-by: the net-value rule (credit only for a changed decision, every computation pays its cost) and the exchange-rate sweep for controllers of unequal cost