auto-harness synthesis
Type: agentic-system-analyses/types/agentic-system-synthesis.md
Bounded synthesis
Prepare an improvement plane, not an enclosing agent runtime
auto-harness supplies benchmark adapters, editable task-agent templates, experiment state and a program for an external coding agent. Its claimed work is repeated benchmark-driven improvement of agent/agent.py. This account covers the shipped improvement plane at Git revision 52360dbbeb8cea8886d90778e634a145c51da735, with cutoff 2026-10-15. SRC-1 supplies inspected implementation and declared procedure; there is no supplied execution trace or causal experiment. The whole-system boundary covers local drivers and BIRD wrappers, but not the external coding-agent runtime, benchmark frameworks, model providers or datasets. See the boundary and evidence and Source register. This synthesis uses the accepted members without additional source reads.
Preparation checks dependencies, installs a selected candidate and composes its improvement program. It initializes retained workspace state and records a baseline; first Terminal/BIRD setup generates reward-stratified 70/30 train/test membership with seed 42, excluding null outcomes. Existing workspace state is preserved, but candidate/program copying precedes the decision to skip an existing baseline. Re-preparation can therefore replace an optimized candidate while retaining its old score history. BIRD setup can also provision external code and services without pinning their downstream revisions (RT-RTE-prepare-baseline, MEM-OBJ-task-split).
Delegate task execution, then hand evidence back
The representative workflow is a connected Terminal-Bench improvement iteration. The external coding agent runs training tasks, diagnoses failures, proposes one focused candidate change, invokes the gate, and is instructed to commit or revert. The repository supplies the procedure and local handoffs, not the coding agent's scheduler or proof of its compliance. The workflow account establishes this progression from source, not a successfully delivered run.
Task execution has three materially different routes:
- Terminal: the runner launches Harbor with selected tasks and the candidate import path. The stock candidate starts a fresh system/task conversation, requests LiteLLM completions and dispatches bash through the supplied environment. It allows up to 80 model steps and a 120-second command timeout. Malformed arguments and unknown tools become tool messages; model exceptions end the loop. Text without tool calls declares completion, while Harbor's external verifier supplies reward (RT-RTE-terminal-task).
- Tau: a registered candidate factory returns assistant messages and mutable task history. External tau2 owns tool execution, simulation progression and evaluation; the local candidate prepends domain policy to its accumulated history rather than implementing a Terminal-style executor (RT-RTE-tau-task).
- BIRD: local code starts services and exposes HTTP session operations backed by ADK. Mode/task pairs select in-memory sessions; models choose SQL or clarification actions through externally supplied tools and callbacks. The external orchestrator/evaluator supplies rewards. Initialization locking does not establish safe concurrent turns, and reset changes the session reference without proving backend deletion. Services have cleanup paths, but durable restart sessions and callback budget enforcement are not established locally (RT-RTE-bird-session).
Task separation keeps the outer coding-agent conversation out of the stock Terminal model frame. The handback is rewards and selected traces, not the complete outer or inner reasoning process. Terminal filters long tool output to first/last halves around an omission marker at an 8000-character limit; the saved message trace already lacks the middle. There is no automatic recovery pointer, although bash can deliberately reread environment files. BIRD diagnostic previews instead shorten displayed events; they do not establish compaction of the actual ADK model window (EPI-RTE-output-filter, MEM-OBJ-traces).
Train trace copies support requested foreground diagnosis, not automatic historical-context injection. Latest files update only for encountered items, so stale files can remain. First-seen baseline directories suppress recopy and can preserve incomplete evidence. Non-train Terminal runs disable the stock candidate's trace saving, and BIRD copies train traces into the diagnosis directory. These controls reduce designated exposure; they do not restrict the coding agent's filesystem access, cover every external log, or survive arbitrary candidate changes as an information-flow guarantee (MEM-RTE-trace-retention, EPI-CLM-structural-cheating).
Revise behavior, check eligibility, retain different consequences
Candidate revision can change Python execution, tools, loop structure, context processing and embedded directions, not only prompts. Later imports give code executable authority; instruction constants reach task models as requested guidance. Cached BIRD runners and tau registration qualify when edits become live. The outer program requests diagnosis, three-failure hypothesis abandonment, rollback after rejection and learning-note updates, but does not implement these actions itself (RT-RTE-candidate-revision, RT-BAP-candidate-runtime).
The gate separates proposer from numerical decider. For a nonempty suite it reruns stored train IDs and checks the requested-task pass fraction against a retained threshold. It then evaluates the configured gate split, even after suite failure, and compares the returned mean with the highest recorded score; equality can pass. Only both successful checks permit subsequent promotion of formerly failing non-suite tasks that pass recheck with non-null reward at least 0.5. This changes future evaluation coverage, not the candidate's installation or commit (RT-RTE-gate-admission).
Three retained suite parts have different consequences. Membership selects future tasks and contributes the denominator; the threshold is carried-forward configuration, not learned from reruns; diagnostics are saved before final acceptance and may survive rejection. Historical diagnostics have no established separate decision consumer beyond merge/serialization. Score history supplies the next acceptance floor. Thus retention, operational authority and candidate acceptance cannot be treated as one event (MEM-OBJ-suite-tasks, MEM-OBJ-suite-threshold, MEM-OBJ-suite-last-results, MEM-OBJ-results).
The gate's checks are conditional, not universal safety guarantees. Its path guard is best effort, can be disabled and fails open when Git checking is unavailable; it does not intercept writes. Existing null rewards score zero, but absent task entries are not restored on every return path. An empty suite combined with no prior best or a zero best can admit an empty-result mean of zero. The CLI overwrites the file named train_results.json for arbitrary splits, while the promotion loader ignores split metadata. These source-level cases prevent treating gate success as complete evaluation or guaranteed train provenance (RT-RTE-gate-admission, RT-CLM-timeout-failure).
After gate success the coding agent is instructed to commit and record printed numbers. The recorder checks paths and appends caller-supplied scores with a commit label; it neither consumes a gate certificate nor binds evaluated bytes to that commit. Failed-gate recovery remains externally owned, and reverting the candidate alone need not remove another forbidden file. The intended iteration output is a candidate commit, score row and learnings; after five non-improving iterations the program requests an operator summary. Neither the transaction nor final delivery is independently enforced (RT-RTE-iteration-record, EPI-CLM-gate-everything, workflow account).
What the feedback supports
The strongest implemented experience-to-future-control contribution is reward-fed suite promotion and baseline split generation: software transforms execution feedback into durable selection IDs consumed by later gates and runners. This is a supported trace-fed write/read chain, not evidence of improved future capacity. Trace-responsive candidate revision supplies a further afforded pathway from diagnostic evidence to durable code/instruction changes and later imports. Actual diagnosis, edits and benefit remain unobserved. Requested learnings can retain hypotheses and reasons, including failed attempts, but the log has no wired reader; candidate imports do not consume its historical explanations (MEM-OBJ-suite-tasks, MEM-OBJ-task-split, MEM-OBJ-learnings, RT-RTE-candidate-revision).
Epistemically, the system imports task assertions, tool observations and rewards with external warrant unresolved. Generated answer assertions are not validated merely by generation or text-only completion; executable actions have operational force rather than truth warrant. Gate arithmetic derives conditional numerical conclusions, and acceptance licenses the prescribed benchmark commit, not a causal explanation of why a change works. No supplied instance establishes accepted and integrated ampliative explanatory knowledge. Suite adaptation remains a useful operational contribution without that stronger interpretation (EPI-OBJ-task-input, EPI-OBJ-tool-output, EPI-OBJ-assistant-content, EPI-OBJ-task-actions, EPI-OBJ-evaluation-state).
The independent property assessments of RT-RTE-candidate-revision remain separate:
| Property | Supported assessment at this boundary |
|---|---|
| Theory-builder condition 1: localized theory content | Afforded: whole instruction/rule units are localized and replaceable; no newly formulated hypothesis instance is supplied. |
| Condition 2: content-dependent consumption | Claimed: the program requests content-guided decisions, without an observed decision depending on theory content. |
| Condition 3: content-directed criticism | Claimed: diagnosis/revision is requested, without supplied criticism identifying the challenged claim, adverse result, blame and resulting revision. |
| Condition 4: criticism retained into another round | Claimed: repetition and learnings are requested, without demonstrated criticism shaping a later round. |
| Learning from criticism of consumed theory | Uninspected: improved future capacity and attribution are not established by retained scores or editable code. |
| Reflection | Afforded through representation-mediated trace-to-revision feedback, not demonstrated activation. |
| Reflective theory building | Uninspected: candidate tuning does not demonstrate criticism/revision of the improvement method against its own operation. |
| Autonomous theory building | Claimed: computation is assigned the internal procedure, but complete autonomous criticism and retained use are not demonstrated. |
| Self-improvement | Afforded within the improvement plane over later benchmark iterations, toward independently specifiable benchmark reward. Standing trace/update/import paths exist; subsequent operation causally depending on the update, link 4, and successful improvement remain unestablished. |
For the stock Terminal improvement scenario, the evidence supports a coordinated, externally executed optimization procedure with wired conditional checks and persistent evaluation-policy adaptation. It does not support reproduced performance gains, enforced isolation or universal non-regression. The README's 0.56-to-0.78 Tau3 comparison is attributed reported operation, not an inspectable run or component-level causal comparison (EPI-CLM-score-improvement). Linked pre/post revisions, complete task inventories, traces showing actual criticism and later consumption, and repeated comparable evaluations would strengthen exercised-learning conclusions; independently varied interventions would be needed for component attribution. Authenticated gate-to-commit records, enforced result provenance and complete-result accounting would strengthen admission guarantees. Consistent dataset forwarding during preparation and gating would remove the identified custom-Harbor population mismatch (RT-RTE-iteration-record, RT-RTE-gate-admission, report verification).
Limitations
| Limitation | Affected SRC IDs and record citations | Inspected boundary | Conclusion prevented | Evidence that would resolve it |
|---|---|---|---|---|
| Non-default Harbor baseline/gate population mismatch; carried report-verification limit | SRC-1; RT-RTE-prepare-baseline, RT-RTE-gate-admission, report verification | prepare.py omits configured dataset in both Terminal baseline constructors; later gate/CLI forward it and the runner defaults to terminal-bench@2.0 |
Same-population initial split/score floor for custom datasets, or the broader custom-Harbor “all work as-is” implication | Consistent dataset forwarding and invocation-linked baseline/split/gate task inventories for the configured dataset |
| No inspectable optimization run or causal comparison | SRC-1; RT-RTE-candidate-revision, EPI-CLM-score-improvement | Frozen implementation and programs; README endpoints without run artifacts | Exercised self-improvement, learning benefit, reproduced gain, causal link 4 or component attribution | Linked evidence, criticism, revisions and later behavior; comparable outcome trials and independently varied component interventions |
| External execution and oracle contracts are not frozen | SRC-1; RT-RTE-terminal-task, RT-RTE-tau-task, RT-RTE-bird-session, RT-CMP-candidate-model, RT-CMP-coding-model | Local adapters/wrappers, not framework internals, datasets/gold answers, providers or coding-host permissions | Independent verifier validity, full context assembly, callback budget enforcement, deployed isolation, exact immutable models or autonomous compliance | Frozen dependency/data/model identities, evaluator checks and deployment/session traces covering the external contracts |
| Designated traces can be filtered, stale or incomplete | SRC-1; MEM-OBJ-traces, MEM-RTE-trace-retention, EPI-RTE-output-filter | Terminal truncation and copy/prune paths; BIRD diagnostic previews/copies | Complete original transcript, invocation-matched latest snapshot or complete baseline diagnostic contrast | Full-output artifacts and invocation manifests with freshness/completeness checks and recoverable omitted content |
| Train-only diagnosis is policy plus partial trace controls | SRC-1; EPI-CLM-structural-cheating, RT-RTE-terminal-task, RT-RTE-bird-session | Stock candidate suppression/copying and outer-agent instructions; external logs and actual grants unresolved | Enforced test-data exclusion or structural impossibility of cheating | Enforced access boundaries and audited alternate candidate/framework logging and read paths |
| Gate, history and recording do not authenticate an evaluated commit or complete population | SRC-1; RT-RTE-gate-admission, RT-RTE-iteration-record, RT-CLM-timeout-failure, MEM-OBJ-train-results | Invoked arithmetic/path checks, loader provenance and caller-score append; explicit-ID and empty-result alternatives | Universal gate-before-commit, lifetime edit confinement, full-task timeout accounting or trustworthy score provenance | Authenticated evaluated-byte/task/result manifests, fail-closed guards, split checks and a gate-bound commit/record transaction |
| Learning-note reuse and semantic transformation remain unresolved | SRC-1; MEM-OBJ-learnings, RT-RTE-candidate-revision, EPI-OBJ-evaluation-state | Requested appends and diagnostic revision procedure; no supplied learned entries or reader | Actual reuse of explanatory reasons, accepted explanatory knowledge or complete classification of semantic curation and revised-code effects | Concrete entries linked to evidence, explicit later readers, content-dependent criticism/acceptance and resulting decisions |
| Caller/configuration origin and additional retained-content effects are unresolved | SRC-1; MEM-OBJ-results, MEM-OBJ-suite-threshold, MEM-OBJ-suite-last-results, MEM-OBJ-traces | Software serialization and known consumers; not provenance of supplied values, external payload construction or an additional historical-diagnostic reader | Complete admission/lineage/authority inventory beyond the supported local paths; global negatives about other trace-fed updates | Provenance and approval records for supplied content, inspectable payload transformations and concrete consequential consumers |
The accepted reconciliation reports no unresolved cross-report conflict. Profile verification declares no additional limit; its partial inventories preserve uncertainty rather than implying absence or observed benefit (profile verification).