Meta-Harness synthesis
Type: agentic-system-analyses/types/agentic-system-synthesis.md
Bounded synthesis
Meta-Harness is a builder or improvement plane: a coding model proposes changes to executable task harnesses, software evaluates candidates, and retained outcomes become inputs to subsequent proposals. Its strongest supported contribution is this connection between editable solver organization and independently computed task feedback. The inspected code establishes opportunities for durable behavior change, not evidence that a search actually improved future capacity. The unresolved question is whether feedback caused a useful revision that later task execution depended on. RT-RTE-classification-search, RT-RTE-terminal-search, RT-RTE-pilot-search.
The evidence is SRC-1, the repository at commit 8123ccabe2b19fa1123090b2e5f1bacc20963ce8, with an analysis cutoff of 2026-10-14. The whole-system repository boundary includes both reference experiments, their task runtimes, onboarding and the experimental Harbor pilot. Implementation and shipped instructions support the account; the pilot README's probe scores are reported operation only. No target execution, inspectable search run or causal experiment was supplied. Claude Code, Harbor internals, model providers and remote environments are external dependencies. The paper, separate optimized artifact and downloaded task collections are not frozen evidence. See boundary and its Source register. This synthesis reads accepted members only and adds no source inspection.
Configure the experiment and propose a change
An operator selects the task, configuration and search budget. Onboarding first requests a domain specification describing mutable machinery, success criteria, baseline and evaluation budget, then stops pending a separate implementation request. Suggested defaults do not authorize spending. This is a shipped document handoff, not an executed adaptation service. RT-RTE-domain-onboarding.
The two reference drivers start fresh Claude Code sessions with run-specific paths and a shipped proposer skill. The model chooses reads and code edits; the Python driver owns benchmarking. History is not automatically injected in full. Instructions ask the proposer to use summaries, frontier scores and traces, but actual reading and interpretation are unobserved. Software-derived result/control fields and copied model-authored hypotheses remain different operative parts: a resume counter determines the next iteration, while a hypothesis offers advisory design rationale. Storing both in one JSONL row does not validate the explanation. RT-BAP-proposer-instructions, MEM-OBJ-summary-results, MEM-OBJ-summary-rationale.
The representative workflow is the Terminal-Bench reference search, selected because its procedure specifies analysis and implementation handoffs linked to task evaluation. The main proposer is instructed to ask an analysis child for a short state summary, a falsifiable mechanism hypothesis and predicted performance/efficiency effects. An implementation child receives the candidate name, class, selected changes and working directory, edits a new file, and returns import-check status. These are requested delegations, not observed child executions. Files remain shared effects; Claude Code owns actual context isolation. The main proposer writes a pending manifest, which—not its terminal prose—is the driver's candidate authority. The child's import self-check is distinct from the driver's veto. See workflow account and RT-OBJ-pending-eval.
Admit code, then measure task outcomes
Candidates gain executable authority before selection as winners. Reference admission imports Python; Terminal-Bench additionally checks Terminus2 inheritance and normally runs a one-task smoke trial. Smoke checks runtime compatibility, not positive task reward, and can be skipped. The experimental pilot instead scans literal forbidden strings and a minimal AST interface before staging source for child Harbor import. These gates assess different predicates. None establishes semantic safety, general usefulness or inaccessible benchmark data. The proposer has broad shell/edit tools, and model-facing prohibitions are not filesystem enforcement. RT-BAP-candidate-execution, RT-RTE-terminal-search, EPI-RTE-source-check.
Classification evaluates candidate memory systems against dataset targets. Its few-shot baseline retains labelled input/target pairs, without correctness filtering or retained prediction reasons. Offline batches expose ground truth; online batches include actual predictions and feedback, but this baseline discards those feedback fields. Prediction selects demonstrations through input-hash-seeded sampling/shuffling and a 30,000-character formatting cap, not semantic similarity. Saved state is reloaded for test prediction, but generic checkpoints can hold other candidate-specific strings; NoMemory demonstrates that successful reload need not affect prediction. MEM-RTE-classification-memory, MEM-OBJ-memory.
Classification's answer oracle is target agreement under dataset-specific comparator rules, not open-world truth. For example, substring extraction can mark not unsafe as agreement with unsafe. Aggregate accuracies therefore inherit target and encoding assumptions. Frontier selection uses required seed groups, per-dataset accuracy winners and aggregate accuracy/context-size Pareto eligibility. This is evidence-consuming operational selection for later search and testing, not acceptance of the hypothesis explaining a score. EPI-RTE-answer-check, EPI-RTE-frontier-selection.
Terminal-Bench uses external Harbor verifier rewards. The driver counts positive rewards among discovered trial directories and replaces frontier entries on strictly better scores. Expected trial counts produce warnings rather than completed denominators: an entirely undiscovered trial does not count as failure. The pilot's local verifiers instead check exact output equality against encoded expected objects, then aggregate child rewards. Infrastructure failure can become zero reward, so zero alone does not locate blame in the candidate. Pilot accepted means source admitted to scoring; an admitted harness can fail every task. EPI-RTE-terminal-scoring, EPI-RTE-suite-scoring, EPI-RTE-source-check.
Retain feedback, recover and deliver
Reference searches retain candidates, results, frontier identities, summaries and traces for another round. Requested short post-evaluation reports could preserve criticism, but no produced report or demonstrated criticism-mediated revision is supplied. Classification logs contain partial input previews and mode-dependent events, not necessarily full examples. Local completion caching replays a response on an exact request key; it is not a synthesized lesson and its key does not pin backend weights. MEM-OBJ-reports, MEM-OBJ-classification-log, MEM-RTE-completion-replay.
Context control also operates inside task execution. KIRA can supply a derived continuation after context pressure; inherited summarizer selection and fidelity are uninspected. Its fallback adds the original instruction and last 1,000 screen characters alongside surviving messages, not necessarily in place of all history. Classification removes prompt middles above a character threshold; pilot shell feedback keeps bounded stream tails; KIRA removes entire marker-bearing output lines. These filters can omit consequential evidence and do not themselves wire automatic reload of removed content. RT-RTE-kira-execution, its memory annotation, EPI-RTE-observation-filter.
Recovery is chiefly continuation, selective reuse and retention of earlier winners, not transactional rollback of arbitrary edits. Fresh-start cleanup leaves some artifacts reachable. Malformed terminal manifest JSON lacks the graceful recovery used for a missing manifest. Ordinary terminal delivery is candidate files, histories and selected identities, not deployment or frozen winner bytes. Its optional final pass repeats evaluation on the same dataset, can parse partial results after ignoring failure status, and is not an untouched holdout. Classification has separate test finalization and a completed-run evolution guard, but direct evaluation paths bypass that guard and publicly shipped test data is operationally separated rather than secret. RT-RTE-classification-finalize, RT-RTE-terminal-search, workflow account.
The pilot feeds projected reward/task/reason/budget feedback to its editor while retaining fuller history and source snapshots. Its normal evaluation tool enforces a finite budget over completed recorded calls, not all compute or standalone controller use. The final verifier checks report arithmetic and awards the maximum accepted historical reward. It neither authenticates that history nor restores/revalidates final code as the best evaluated source. Thus its delivered score and final edited harness are distinct outputs. MEM-OBJ-evaluation-history, RT-RTE-pilot-search, EPI-RTE-history-selection.
What the feedback loop establishes
For bounded task experimentation, the discriminating mechanism is executable search with heterogeneous admission and outcome-based retention. It changes code, prompts and state around external models rather than implementing weight updates. Exact served model identities remain unpinned, so the declared fixed-model comparison is architectural, not an authenticated constant-backend experiment. Outcome comparison can challenge a forecast, but without independently varied components it does not identify the causal mechanism or warrant transfer. RT-CLM-fixed-model, EPI-CLM-mechanism-tests.
The following properties remain independent, with statuses taken from the runtime search records:
- Theory-builder conditions 1–4: localized replaceable rules/hypotheses are afforded (condition 1), and candidate-rule execution is wired (condition 2). Reference instructions claim content-directed criticism and criticism-mediated iteration (conditions 3 and 4), without a supplied instance. In the pilot, conditions 3 and 4 are uninspected. Scores and retained outcomes do not establish either condition.
- Learning from criticism: no improved future-action capacity attributable to criticism of a consumed theory is established. Automatic retained continuation and feedback-to-code routes are supported contributions, but their write-route classification does not imply beneficial learning.
- Reflection: candidate representations and evaluation feedback are wired to editing opportunities. Exercised representation-mediated change remains unobserved; reflective theory-builder operation is not established.
- Autonomy: software/model roles handle candidate proposal, admission and scoring without per-change human approval. Autonomous theory-building remains unestablished because the internal criticism and iteration operations are not demonstrated.
- Self-improvement: at the enclosing builder-plus-task-harness boundary, over search rounds toward validation accuracy, pass rate or suite reward, the records support marked dispositional improvement-directed self-change. Candidate code can enter later solver organization and execution authority. Evidence causally determining the update and subsequent operation depending on it—causal links 1 and 4—remain unobserved. Neither exercised nor successful self-improvement follows. At the narrower proposer-only boundary, the harness may instead be an external output.
These assessments are supported separately by RT-RTE-classification-search, RT-RTE-terminal-search and RT-RTE-pilot-search. The epistemic bounded conclusion likewise distinguishes operational acceptance from accepted explanatory knowledge; it establishes no system-complete negative about opaque model processing or unavailable generated candidates.
A trace binding feedback, an explicit challenged claim and blame placement, the resulting revision, and later solver dependence would establish more of the currently unresolved progression. Authenticated code/model identities and complete evaluation denominators would strengthen comparison validity. Independent component interventions and separate outcome evidence would be needed for mechanism attribution and demonstrated improvement; held-out transfer would answer a different question. Restoring and revalidating a selected pilot snapshot would bind final delivery to historical reward. These changes would alter different conclusions rather than produce one system-wide grade.
Limitations
| Limitation | Affected SRC IDs and record citations | Inspected boundary | Conclusion prevented | Evidence that would resolve it |
|---|---|---|---|---|
| No inspectable execution or causal comparison | SRC-1; RT-RTE-classification-search, RT-RTE-terminal-search, RT-RTE-pilot-search | Frozen repository implementation, instructions and reported pilot probes | Exercised reliance, beneficial learning, exercised reflection/autonomous theory-building, successful self-improvement and component causality | Provenance-linked run traces, retained criticism/revision/dependence evidence and appropriately controlled outcome comparisons |
| External execution and evidence excluded | SRC-1; boundary, RT-BAP-proposer-instructions, RT-RTE-kira-execution | Shipped wrappers/adapters; external Harbor, Claude Code, providers, sandboxes, paper/artifact and downloaded collections excluded | Deployed isolation, provider/inherited-runtime correctness, paper-result reproduction and complete remote-task provenance | Separately frozen dependency/task/artifact evidence and inspectable deployed runs |
| Generated state and checkpoint initialization unresolved | SRC-1; MEM-OBJ-memory, MEM-RTE-classification-memory | Shipped baseline implementations, generic state interface and direct load paths | Universal memory format, provenance, immutability or behavioral activation from loading | Actual generated candidates and checkpoints with admission provenance and consumer traces |
| Inherited compaction and provider retention opaque | SRC-1; RT-RTE-kira-execution, MEM-OBJ-completion-cache, profile verification | Local handoff/cache call sites and visible fallback; external internals uninspected | Complete storage/form/curation/selector inventory, faithful summary or current-backend cache validity | Pinned inherited/provider contracts and implementation, actual inputs/outputs and reuse traces |
| Requested criticism and read-back not demonstrated | SRC-1; MEM-OBJ-reports, MEM-OBJ-summary-rationale, RT-RTE-terminal-search | Proposer skills, copied rationale and requested child handoffs | Produced criticism, verified explanation or criticism-mediated next-round learning | Reports and child returns linked to evidence, explicit revisions/changed reliance and later consumption |
| Ordinary validation/test JSON is not a per-answer archive | SRC-1; EPI-OBJ-predictions, MEM-OBJ-classification-log, report verification | Inner-loop returned predictions versus aggregate _build_output CLI serialization; partial training logs |
Durable per-answer validation/test evidence and replay of every comparator decision; the prediction record's storage wording overstates this path | CLI output retaining complete per-answer inputs, targets and comparison results, or independently supplied complete archives |
| Coarse cross-artifact and mixed-object inventory | SRC-1; MEM-OBJ-evaluation-history, EPI-OBJ-evaluation-results, EPI-OBJ-val-results, report verification | Accepted containers and their separately described check/consumer routes | Complete snapshot/flag/metric/winner-level comparisons; one uniform check, consumer, form or authority for every contained part; exact identity from overlap | Finer operative-part declarations with separate lineage, checks and consumer assessments |
| Inspect/validate child policy omitted from separate inventory | SRC-1; RT-RTE-pilot-search, epistemic coverage, report verification | Shipped pilot variant's requested local-validation and output-existence instructions | Complete comparison of task-level self-check policies across pilot variants; instruction is not an enforced or successful check | Separate policy/consumer inventory and execution evidence for actual local checks |
| Metric and holdout scope is narrower than explanatory warrant | SRC-1; EPI-RTE-answer-check, EPI-RTE-frontier-selection, EPI-RTE-terminal-scoring, EPI-CLM-mechanism-tests, RT-RTE-classification-finalize | Encoded labels, seed-group checks, discovered trials and same-dataset terminal final pass; public classification test data | Target truth, full candidate/dataset coverage, uncontaminated holdout, causal explanation or transfer from winner selection | Independently validated targets, complete task/trial ledgers, authenticated unchanged candidates/models, protected holdout protocol and component/transfer comparisons |
| Pilot historical score does not bind final code or enforce universal budget | SRC-1; EPI-RTE-history-selection, RT-RTE-pilot-search, MEM-OBJ-evaluation-history | Normal evaluation-tool charging, standalone/shell alternatives, report arithmetic and historical maximum | Universal compute cap, authenticated provenance or final-artifact performance equal to historical best | Enforced coverage of alternate evaluation paths, authenticated reports and restore-best/final-source revalidation |
| Lossy context and selective recovery | SRC-1; EPI-RTE-observation-filter, MEM-OBJ-classification-log, RT-RTE-terminal-search, workflow account | Character truncation, marker-line removal, partial logs, selective cleanup and terminal final parsing | Complete evidence delivery, automatic recovery of evicted content, transactional rollback/erasure or successful final evaluation from a completion message | Complete raw/delivered-context traces, explicit reload and rollback protocols, complete cleanup inventory and final-evaluation success checks |
The report verification declares no blockers and three local limits, carried above with their consequences. Profile verification declares no blockers or additional limits; it preserves partial coverage rather than complete memory-value claims. No unresolved cross-report conflict remains in the supplied reconciliation.