Boundary cases assessed by deployment evidence
Status: Retained report, frozen as of 2026-09-17. Produced in the theory-builder workshop as its boundary-case assessment and retained here because the library definitions of theory builder and its externally tested, reflective, and autonomous variants stated their boundary cases in their own words and this is the exact record behind them. On 2026-09-25 the variants were retired and theory builder was redefined; the record below is unchanged and uses the terms of its date. These are source-bounded readings, not architecture-wide verdicts or independently reproduced experiments. The Commonplace downstream case remains a proposed arrangement; the first-downstream-run workshop owns it.
Reading the assessment
The boundary includes the roles that produce, diagnose, revise, select, and maintain the retained knowledge or machinery. An operator doing that work belongs inside the assessed builder for that episode. Supplying demands or judging completed outputs can be an outside role, even when the same person also does internal work. Role separation does not establish statistical independence or remove evaluator bias.
External assessment, evidence access, and continuing responsibility are separate questions. A fixed objective can be externally assessed. An internally represented utility can refer to externally supplied outcomes. Conversely, a separate judge does not make adaptively reused evidence an untouched test. Continuing responsibility means maintaining and revising the relevant commitments across later work; neither an indefinite runtime nor mere archive retention is its test. A bounded experiment can show such responsibility within its horizon. Where a paper does not establish it, the answer below remains unknown.
“Observed” below means reported in the inspected source, except where the row explicitly identifies local documentation or a proposal. None of the external implementations was executed for this assessment. External task success also does not establish that every retained object is a theory, or that a retained explanation caused the success.
Commonplace's retained reflection case reports a two-way causal path through the tag-README change, with human design judgment and improvement bounded by the adopted criterion. That local evidence is distinct from the proposed downstream experiment. The lack of a run under the new protocol does not withdraw the earlier bounded reflection claim; this assessment does not reaudit its commit trace.
Assessment
| Case and evidence | Declared system boundary | Observed evaluation and judges | Evidence access and protocol | Continuing responsibility | Scoped conclusion and unknowns |
|---|---|---|---|---|---|
| FORTE — retained ingest, Quotes at §§2 and 4.2; paired snapshot unavailable | The retained task statement supplies an incorrect initial theory and consistent positive and negative instances, and asks for a minimally revised theory correct on those instances. The verified scope here is that refinement task. | The retained quotations establish correctness on supplied instances as the task and termination when no improving revision is generated. They do not establish the experimental judge or a held-out protocol. The ingest's broader experimental account remains unverified in this assessment. | Supplied initial theory and instances are supported by the retained quotations. Train/test access, experiment sampling, and downstream feedback cannot be checked without the pinned snapshot. | The stopping condition supports a bounded refinement invocation. It does not prove that FORTE could not be placed under continuing maintenance responsibility. No such deployment is established by the retained quotations. | A supplied-example refinement task is supported. The earlier categorical “not persistent; no unresolved issue” is too strong for the evidence available here. Ongoing responsibility and independently assessed generalization remain unverified; this is an access limit, not an impossibility result. |
| Gödel machine — ingest (snapshot required), §§2.1–2.4, 3.2, 4, 7 | A formal machine contains an environment-interacting policy and a proof searcher, with axioms for hardware, initial code, environment assumptions, and utility. The environment is outside that machine. This is a construction, not a reported deployment. | The paper establishes conditional formal results about proof-governed switching. It reports no experiment with an output judge. The proof checker authorizes rewrites relative to the axioms; that is not an observed downstream acceptance study. | Inputs and reward signals can arrive from the environment. state2theorem admits time-labelled observations into proof reasoning. The theorem's assumptions and the selected utility govern what a rewrite must prove; no empirical train/assessment protocol is supplied for a deployed instance. |
The construction assigns the machine ongoing control of its policy and proof search during its lifetime. Whether an implementation maintains empirically revisable theories in the workshop's sense is not established by an observed episode. | Classification as an observed externally tested theory builder stays open. Encoding utility internally does not establish absence of external acceptance: a deployment would have to specify what supplies its rewards and judges consequences. The formal construction alone neither demonstrates nor excludes that arrangement. |
| Darwin Gödel Machine (DGM) — ingest (snapshot required), §§3–4.4, Appendix C.3 | The assessed search system includes the self-editing coding agents, separate o1 diagnosis, archive, and fixed selection controller. Benchmark tests supply task scores. Keeping the controller fixed is an update limit, not proof that outcomes lack external assessment. | Authors report staged coding-benchmark scores, component ablations, and transfer tests after 80 search iterations. SWE-evolved and Polyglot-evolved agents improve on the alternate benchmark, which the authors report was never accessed during their respective searches. Benchmark test harnesses judge task solutions; the authors conduct the study. | Search uses evaluation logs and scores. The SWE diagnosis prompt includes private test patches withheld from agents during task evaluation. Stages expand from 10 tasks through a larger subset, with 200-task SWE evaluation concentrated on high-scoring leaders. Those measurements participate in adaptive search. The alternate benchmark is a separate, untouched-by-search transfer test. | Later iterations consume and modify inherited agent code, so the archive is operative. That shows repeated development within the run, not merely storage. Responsibility for revisable theories, as distinct from code and search records, and continued responsibility beyond the study are not established. | Bounded external assessment of coding outcomes is reported; cross-benchmark transfer supports a stronger generalization claim than the adaptive within-benchmark curve. Neither limited duration nor a fixed objective invalidates that evidence. Theory-builder qualification, sustained improvement, and attribution to retained explanations remain open. |
| EURISKO: fleet design and heuretics — ingest (snapshot required), §§2–3 | Use the joint program–Lenat builder: heuristic generation, simulation, worth adjustment, and Lenat's repeated culling and rewarding all shape the product. Traveller tournament play is outside that production loop. Heuretics has a different assessment boundary. | Lenat reports fleet wins in the 1981 and 1982 tournaments run by Game Designers Workshop; rule changes intervene between them. For heuretics, reported checks include the program's judgments and the researcher's inspection and correction, including a heuristic that claimed credit for others' discoveries. | Fleet development consumes recorded battles, new simulations, and human intervention. The source describes culling roughly every 12 machine hours. Tournament results supply an additional outcome beyond those development signals, but this is not a randomized or preregistered comparison. No common held-out protocol spans all reported domains. | The joint builder repeatedly evaluates, revises, removes, and reuses heuristics; the source describes development under changed tournament rules. This supports bounded continuing responsibility for that heuristic repertoire. It does not establish autonomous responsibility or make every heuristic a theory. | External fleet outcomes support the joint builder's tournament performance. They do not independently warrant its heuretics or mathematics claims. Attribution among encoding, search, simulation, and manual selection remains unresolved. Internal worth manipulation is a failure case within one assessment arrangement, not an architecture-wide absence of external judgment. |
| AERA / Nivel et al.: S1 experiments — ingest (snapshot required), §§1, 5, 6 | S1 contains the model-learning and execution machinery; developers supply the seed, drives, representation, and interaction setting. Human demonstrators and later interlocutors are outside S1. The research team also chooses the learning and assessment conditions. | Two nested dialogue experiments report learned behavior through video inspection and t-pattern analysis of recorded interactions. Human participants are developers. Experienced coders prepare records for analysis. These are behavioral assessments beyond a model's own reliability score, but independent replication or blinded assessment is not reported. | S1 receives streamed speech, motion, and interaction events and learns models from them. Observation continues until it accurately predicts major event types; it then interacts with the same humans for recorded assessment. No reserved task family or stopping rule independent of reaching the target is established. | The architecture continually updates model reliability and can phase out or delete poor models. The experiments show acquired models used in later interaction. They do not separately establish continuing deployment responsibility under new demands after the observation-and-demonstration sequence. | The protected motivation machinery does not exclude external assessment. The evidence supports model acquisition and subsequent behavior in the supplied setting, with researcher-shaped stopping and assessment. Robustness, independent assessment, and continuing responsibility beyond this sequence remain unknown. |
| Thórisson: seed-programmed general learning proposal — ingest (snapshot required), §§5–6, Fig. 5 | The proposed learner starts with supplied drives, observables, actions, ontology, and models, then constructs and revises causal-relational models. S0 and S1 are supporting examples, not a new controlled deployment of the general proposal. | The paper summarizes earlier S0 object-moving and S1 interview results and illustrates a human interaction. It does not supply a new independent evaluation of the full general-learning claim. Do not count these summaries as additional replications of the earlier AERA evidence. | Examples receive bounded multimodal observation streams; S0's goal is operationalized through human interaction and S1 learns after observation. The proposal describes continuing model comparison and revision, but this paper does not give enough assessment separation, sampling, or stopping detail to establish a new generality test. | Continuing knowledge revision is an explicit design intention. Earlier examples show acquisition and later use, not a new longitudinal test of the proposal's responsibility across novel demands or of improvement in its learning machinery. | A proposal with bounded supporting examples. Its broad cumulative-learning claim remains unestablished by an independently assessed deployment in this source. Missing protocol evidence is not evidence that the proposed architecture cannot support one; human-originating signals alone do not settle the question. |
| Meta-AQUA — ingest (snapshot required), §§4–5, Figs. 11–12 | Include story understanding, reasoning traces, failure diagnosis, learning-goal formation, planner, and repair algorithms. Tale-Spin supplies stories; the researcher supplies the experimental correctness judgment and partial-credit rule. | Six generated sequences compare goal-guided planned learning, randomly ordered repairs, and no learning. The researcher judges answer correctness; other points reward asking and answering questions. Goal-guided learning leads on cumulative question points. This is external performance assessment relative to the computational learner, not independent replication. | Every condition starts from the same initial state and processes each sequence. Learned background knowledge persists between stories and resets at the sequence end. Story failures guide repair; no reserved post-learning task family is specified. Runs end on resource exhaustion, with the interrupted story discarded. The separate “evaluate learning” phase is explicitly unimplemented. | Within each sequence, the system keeps using and repairing its background knowledge across later stories. That is bounded continuing responsibility for this knowledge, despite resets between experimental runs. The source does not establish that every repaired item meets the workshop's theory definition. | A finite horizon does not erase persistence, and the researcher-defined score does not erase external assessment. The evidence supports the bundled goal-and-planning treatment in this setting. Independent assessment, separate causal effects of goals and ordering, and warrant for individual repaired theories remain open. |
| GEPA: principal six-benchmark prompt-optimization study — ingest (snapshot required), §§3–4, Table 2 | Include prompt proposal, reflection model, candidate pool, and selection. The compound task system consumes retained prompts. Task metrics and feedback functions are supplied; model weights and the ordinary module decomposition are fixed. | Authors compare optimizer configurations through task evaluators on held-out tests and report cross-model transfer of prompts. These measurements assess task performance, not the truth of reflection text. A designer-supplied scoring rule can still assess outputs outside the prompt optimizer. | Full training text and labels support mutation; validation scores guide candidate selection while direct validation-content access is restricted. Final tests are held out from that optimization. Search ends at a rollout budget and returns the best validation candidate. Cross-model transfer is not a test of transfer to a new task domain. | Repeated prompt revision and consumption are observed inside the optimization run. Ongoing maintenance after return is not shown. Where retained prompts express revisable task commitments they may be theory-bearing; transient reflections and arbitrary prompts cannot be counted automatically. | Bounded external task assessment is supported; categorical “not persistent” is unnecessary. Continuing theory responsibility beyond the run, diagnosis faithfulness, and cross-domain transfer remain unestablished. The §5.1 kernel-search extension uses the target tasks for both training and selection and must not inherit this row's held-out interpretation. |
| ADAS: Meta Agent Search experiments — ingest (snapshot required), §§3–4.3 | The meta-agent, archive, proposal/reflection prompts, framework, and repair loop form the builder. Generated task agents are its products. People set domains, evaluation, budgets, and safety inspection; generated code does not revise that outer arrangement. | Authors report benchmark comparisons and transfer of MGSM-discovered agents to held-out math and non-math domains. ARC agents are also tested across models. Task evaluation metrics judge outputs. Transfer concerns discovered agent programs, not a learned improvement method. | Archive code and validation metrics condition later proposals; search runs for 25 or 30 iterations. Held-out tests are reported, but all discovered ARC agents are test-evaluated, and the top ARC agents for model transfer are selected by source-model test accuracy. Keep this selection distinct from untouched target-domain transfer. | The archive feeds later construction, so its retention has an operative role. The paper shows bounded design search, not a deployment maintaining a body of revisable theories across consumer demands. Continuing theory responsibility is therefore unestablished. | External outcome and bounded transfer evidence exist. Adaptive validation use is a protocol concern, not evidence that ADAS suffered a reported degradation. Do not infer general production-method transfer, continuing maintenance, or theory refinement from archive growth alone. |
| Current Commonplace note-review loop — review-system documentation and fix workflow | Include note authorship, LLM review, revision, criteria, and operator handling as roles in maintaining the KB. A reviewer in a fresh context is still an internal quality-control role when it judges the note. | Documented assays judge note text against criteria, with verdict or report results. This assessment does not audit a particular live review run or identify a corresponding independently judged consumer outcome. A pass or current freshness baseline is not a downstream result. | Review inputs are pinned note and criterion snapshots, with permitted linked evidence and criterion-directed snapshot access. Findings can guide revision and later review. This protocol tracks applicability of review evidence; it is not a reserved downstream task comparison. | Commonplace's maintainer roles retain responsibility for revising KB artifacts. A single review job only performs an assigned assessment; it is not by itself the continuing builder. The documented review and fix loop supports maintenance, not a measured downstream benefit. | Ordinary note-review acceptance lacks demonstrated independent assessment of the note's consequences in later work. This conclusion is confined to that loop. Another self-use episode could assess task outcomes under a declared boundary and protocol; it must be judged on its own evidence. |
| Proposed Commonplace downstream KB house — evidence protocol and illustrative episodes | Builder roles produce and maintain a versioned KB and supporting software. Consumer projects use the release on tasks; task suppliers and output judges act outside the builder while filling those roles. | No consuming-project run has been performed. Judges would assess task outputs under a predeclared outcome contract. The three episodes illustrate product repair, reflective machinery repair, and an unassessed broader claim; they are not results. | Proposed records connect tasks, consumed releases, budgets, outputs, and feedback. Development feedback may guide revision; reserved assessment follows candidate freezing. Project, models, budgets, sampling details, horizon, and reliability threshold must be fixed before scoring. | Continuing responsibility for the KB and software is part of the proposed commission. It becomes observed when successive work shows maintenance, later consumption, and assessment under the declared conditions. An installed release or retained archive alone would not show it. | Candidate for the main path, not an observed instance. Execution and comparison with the frozen seed and human-staffed builder remain required. A downstream gain would assess the combined arrangement; attribution to a note or to the methodology requires the additional comparisons specified in the protocol. |
Source identity and access record
For each available external source, this assessment derived
kb/sources/.snapshots/<ingest-name>.md, computed SHA-256 over the complete
file bytes, compared it with the ingest's snapshot_sha256, and checked
agreement of source URL, title, and author. All nine available paired files
matched. This verifies the retained pair, not the remote publisher's current
bytes or the fidelity of every extraction character. Claims above were
checked in the sections named in their rows. The Lenat–Brown retrospective
was also inspected as context; the EURISKO row relies on Lenat's primary
report for the concrete deployment details.
| Ingest name | Verified paired-snapshot SHA-256 / access result |
|---|---|
automated-refinement-first-order-horn-clause-domain-theories |
Paired snapshot absent. Expected 97e363997cf37e7fe662022c038b3236ac5155a5cb7e6fbe9988748015f5603e. Only the ingest's retained verbatim quotations provide source support here; broader ingest analysis was not substituted for the missing paper. |
goedel-machines-schmidhuber |
d70201102d8f00e7e65cf36a8a89733256d8f163b6b6439941662c47c51fc1f7 |
darwin-godel-machine-open-ended-evolution-self-improving-agents |
9ce2d857b213c66107e0b81be2f281bf691f6812d5146c89a83542e8041aa086 |
eurisko-learns-new-heuristics-and-domain-concepts |
13800f0870a95ac46213363b04cb93f4ec20e6014cccdcb2ab0fbb0117882e52 |
why-am-and-eurisko-appear-to-work |
66584ea51496ab71a0a63efd28ec0ec8729234ceca5b297703a56eec1c8c44b3 |
bounded-recursive-self-improvement |
3e63144551cc5337c5e971e82ff13ee83dd95360f39a2f72df2a680e53bc42d9 |
seed-programmed-autonomous-general-learning |
e7600f700c1c1df67f6bad08c94fefb811bb6b255c8fd00dfe3b4270b73b5c2b |
introspective-multistrategy-learning |
46416e98c244d3bf37bdad1659b341ca72f5d1f872b1e46c8e4ca95c5ddd141f |
gepa-reflective-prompt-evolution |
22af2ed5e7ce1a9faf680934bf4b6d7d8df474173be4595608f0a774cba8c3bb |
automated-design-of-agentic-systems |
ab419dcf4a54cbfa54564967d07ed82c1343405b96ad7c173393254b4dadd945 |
The Gödel pair's present match supersedes the older workshop inventory's claim that no existing file matches its hash. Its capture describes an extracted body with front matter, references, and some figure text omitted; do not silently treat it as a newly verified complete PDF. The identity warning is resolved for this local reading. The deployment classification remains open for substantive reasons, as its row states.
The Commonplace rows use the current local documentation and the proposed protocol, not source snapshots or audited deployment results. No source, ingest, or library artifact was changed for this assessment. Promotion of any source-side claim still requires the adopted claim and its retained support to meet the destination's grounding contract.