Harness² synthesis
Type: agentic-system-analyses/types/agentic-system-synthesis.md
Bounded synthesis
Harness² is a code-coordinated improvement framework around external agent runtimes. It changes explicit instructions and executable harness components, runs candidates against a task, composes refinements, selects a produced output and optionally retains experience for future task improvers. The strongest supported contribution is an implemented execution-evidence-to-behavioral-artifact loop, including cross-task persistence. Whether that loop improves future capacity remains unestablished. Its evidence is source code, prompts and documentation in SRC-1, https://github.com/google-research/harness2, at commit 1cfe254e8d25b15f37048c3730134686abe60335, with an applicability cutoff of 2026-10-22. No execution traces, generated lessons or controlled experiments were supplied. The boundary covers the shipped improvement framework, adapters, prompts, bundled overlays and persistence callers, not complete external solver, provider or benchmark implementations. See the boundary and evidence and Source register.
From task stream to edited executions
An operator chooses a benchmark, OpenCode or Codex substrate, task stream, recursion budget k and scheduling mode. Code owns phase progression. Domains may run concurrently, but tasks within each domain proceed in order, with separate mode libraries restricted to preceding tasks. The proposer edits harnesses; the solver executes them; the selector chooses outputs and also supplies reflection; a separately configured judge supports benchmark measurement. These roles may share model configuration without sharing responsibilities. The framework does not update their weights, and model names do not pin exact provider parameters. See RT-RTE-task-recursion, RT-CMP-proposer, RT-CMP-selector, RT-CMP-solver and RT-CMP-judge.
The representative parallel workflow starts with k base executions, then 2k independent invariant/domain proposals and their executions, followed by k independent compositions and executions. Composers receive the proposal pool, notes and base evidence rather than inheriting one winning proposer's conversation. Sequential mode instead carries composed harnesses and history through k rounds. The 4k rollout accounting is not a count of all model calls or successful executions. The workflow account traces parallel fan-out, fan-in, recovery and delivery; RT-RTE-task-recursion covers both policies.
Each proposal gets a freshly reconstructed filesystem evidence frame: task/source material, current harness, rendered executions, candidate differences, decisions and eligible library files. The model chooses what to open. Only disk edits harvested from context/harness become the proposed tree; reply prose does not install changes. Temporary session workspaces are removed after return, while audits and evidence copies can survive. This separates supplied contexts but does not prove process isolation or reconstruct the external model's whole working window. Renderers clip observations and arguments and can cap the rendered prefix; full files being reachable does not mean their contents enter model attention. See MEM-RTE-evidence-frame, MEM-OBJ-evidence-tree and RT-RTE-edit-admission.
Code admits harvested trees using editable-surface, structural, adapter and grader-marker checks. Invalid proposals receive one repair attempt; exhausted rejection and some failures inherit a fallback harness. Thus a composed final need not contain any new change. Accepted new-tree constraints do not establish semantic correctness, safe code or superiority, and direct materialization APIs have weaker checks. Installed prose guides the external solver through interpretation; scripts, plugins and hooks can act through executable dispatch. Capability grants belong to the external runtime and deployment, not merely to the accepted edit surface. See RT-RTE-edit-admission, RT-RTE-solver-execution and RT-BAP-harness-consumption.
Evidence controls are route-specific. WorkBuddy code uses names-only artifact evidence for improvement and reflection; web and office use contents, and office can also supply raw assets. Trajectories, harnesses and task material remain available, and selection renders deliverables independently. JobBench can fall back from judge extraction to raw text. Path exclusions and suffix-limited marker scans therefore support bounded copied-context controls, not identical consumer renderings or universal answer-key isolation. Tool-capable sessions retain alternative acquisition opportunities; the records establish neither universal exclusion nor an observed exploit. See RT-CLM-answer-free and the forcing-route account.
From final outputs to retained experience
The shipped scheduler selects only composed finals, not bases or all intermediate candidates. At k=1 it delivers the single eligible final without comparative model selection. At larger k the selector sees anonymous candidate outputs and task/source evidence, with no supplied expected-answer oracle in this ordinary benchmark path. A recognized, persisted choice establishes operational delivery, not task correctness or harness improvement. A failure fallback can record a final label without successful selection; it does not authorize reflection. See RT-RTE-output-selection.
When selection succeeds and a library exists, reflection examines executions, candidate edits and decisions. The prompt requests component-level contrasts, caution about confounding and abstention with NO_ENTRY when no concrete reusable contrast exists. Even rejected candidates may inform a lesson. Code, however, admits experience by required fields and marker checks, not by verifying causal adequacy, confidence or lesson truth. Its linewise parser can drop an unkeyed qualifying continuation, so even a syntactically accepted lesson need not preserve the complete response's meaning. This is a reachable implementation case, not an observed damaged lesson. See RT-RTE-experience-admission, EPI-RTE-lesson-parse and EPI-CLM-evidence-lessons.
The library retains the adopted full tree plus an explanation containing cue, lesson, evidence and optional confidence/edit notes. An index supplies navigation. Later stages receive all eligible domain/mode/prefix files through coarse automatic copying, followed by model-requested browsing; there is no code-level cue-similarity ranker or top-k retrieval on this route. Library membership gives an improver reference material, not direct solver authority. A later improver must copy or adapt material into an admitted, installed harness before it can guide another solver. This trace-fed write and later-consumer chain supports wired trace learning in the memory comparison's sense, without requiring weight changes or demonstrating improved capacity. See MEM-OBJ-experience, MEM-OBJ-library-index, MEM-RTE-evidence-frame, RT-BAP-harness-consumption and profile verification.
Prefix withdrawal removes entries from subsequent library copies, not from every prior copy or descendant harness. Summary replay can reconstruct eligible entries; keyed proposer caches can return earlier replies and harvested workspaces. These are different recovery mechanisms, not semantic revalidation or global revocation. See MEM-RTE-library-withdrawal, RT-RTE-resume-reuse, MEM-OBJ-cached-response and MEM-OBJ-call-fingerprint.
Measurement and independent improvement properties
Separate judging acquires benchmark-dependent outcomes; reporting aggregates graded rows, not all attempted tasks. A previous graded row can survive a failed same-run re-judge. These results have reporting consumers but no inspected route that revalidates the task's lesson or revises its harness from those post-hoc grades. Evaluation-related selection is nevertheless not confined to that CLI: WorkBuddy fresh and batched reusable trials prefer parseable grade availability, then recency, not higher score values; single-rollout reuse instead takes the newest usable trial. The scripted demo is another distinct mode, with a source-derived signed-total substring check and fixed edits/experience. It is not evidence of ordinary-model oracle exposure or learning. See EPI-RTE-judge-result, EPI-RTE-score-cache, EPI-RTE-score-report, RT-RTE-solver-execution, RT-RTE-resume-reuse and EPI-RTE-demo-check.
The runtime distinguishes the following independent properties rather than assigning an overall grade:
- Theory-builder conditions 1–4: localized theory-bearing files are afforded (1); content-dependent solver decisions are afforded without demonstrated compliance (2); execution-motivated content criticism is requested, hence claimed rather than observed (3); sequential retention into another round is wired, conditional on retained results actually containing criticism (4). No exercised builder satisfying all four is established.
- Learning as improved future capacity: uninspected. No evidence shows that criticism of a consumed theory improved later capacity. The wired trace-fed artifact pathway remains a positive architectural result, not proof of this stronger property.
- Reflection: afforded. Harnesses and execution records can represent selected system aspects, and representation-mediated edits can affect later runs; accurate representation and exercised causal coupling are unobserved.
- Autonomous theory building: uninspected. Code/model proposal and admission are wired, but every internal theory operation is not evidenced. Code-owned scheduling alone does not establish this property.
- Self-improvement: evidence-responsive self-change is afforded dispositionally at the boundary of the shipped framework and its retained behavioral inputs, over later domain tasks, toward better requested-task execution. Exercised self-improvement remains uninspected: no supplied later operation closes the causal link from retained change through live authority to changed behavior. Successful improvement additionally requires outcome evidence.
These assessments follow RT-RTE-edit-admission and RT-RTE-experience-admission. They do not negate the implemented persistence loop or establish valid criticism by inference from it.
For repeated same-domain work, the source supports a mechanism for making earlier execution evidence and behavioral exemplars available to future improvers. For warranted lesson transfer, contamination resistance or demonstrated performance gains, that mechanism is insufficient evidence. Generated before/after harnesses, complete delivered reflection inputs and parsed lessons, a later-use trace, and controlled comparisons separating persistent experience from other changes would alter the assessment. Semantic preservation checks and evidence-consuming lesson acceptance could strengthen the admission claim, but their presence would still need distinction from successful operation. Reported benchmark gains remain attributed claims, not reproduced results. See EPI-CLM-benchmark-gains, EPI-OBJ-execution-evidence and report verification.
Limitations
| Limitation | Affected SRC IDs and record citations | Inspected boundary | Conclusion prevented | Evidence that would resolve it |
|---|---|---|---|---|
| No supplied executions, generated lessons or causal comparisons | SRC-1; RT-RTE-experience-admission, RT-RTE-edit-admission, EPI-CLM-benchmark-gains | Shipped source and attributed documentation at the reviewed commit | Actual criticism, rationale use, improved future capacity, exercised self-improvement, causal component benefit and reproduced gains | Linked candidate/evidence/lesson artifacts and later executions, with objective-specific controlled comparisons and component variation |
| Lesson warrant and semantic preservation are not established by admission | SRC-1; EPI-RTE-lesson-parse, EPI-OBJ-experience, EPI-CLM-evidence-lessons | Reflection request, line parser, field/marker checks and library formatting | Entry membership as acceptance of causal effect or transferable truth; guaranteed preservation of qualifiers | Complete responses and stored fields, evidence-consuming acceptance decisions with criterion/use/scope, and checked preservation of qualifying content |
| External runtime/provider memory and exact model state remain opaque | SRC-1; MEM-CLM-compaction-boundary, MEM-RTE-evidence-frame, RT-CMP-solver | Shipped integration/configuration, not native compactor, private delegated conversations or provider internals | Complete working-window or memory inventory, native trace-learning classifications, lossless compaction and exact weight identity | Pinned external implementations and payload/return/reload traces, with provider version evidence |
| Evidence projections and access controls are bounded by route | SRC-1; RT-CLM-answer-free, MEM-RTE-evidence-frame, EPI-OBJ-execution-evidence | Copied-context filters, render budgets, subset branches and tool-capable session launchers | Universal contamination isolation, lossless observation delivery or identical judge/improver evidence; no bypass is demonstrated either | Actual consumer payloads and deployment-wide access mediation evidence across tool, shell, plugin, network and provider routes |
| External grading validity and conditional reporting limit outcome interpretation | SRC-1; EPI-RTE-judge-result, EPI-RTE-score-cache, EPI-RTE-score-report | Adapter acquisition, score-cache merge and graded-only aggregation | Certified evaluator correctness, fresh confirmation of every cached grade, all-attempt performance or edit/lesson causal effect | Evaluator/rubric evidence, complete attempted-task outcomes, authenticated output/grade lineage and appropriate comparison design |
| Withdrawal and reuse do not establish global correction propagation | SRC-1; MEM-RTE-library-withdrawal, RT-RTE-resume-reuse | Library deletion/reindexing and saved-summary reconstruction | Revocation of copied or derivative harness guidance, irreversible erasure or semantic revalidation on restore | Dependency-aware withdrawal and replay policy plus traces covering descendants, surviving copies and restoration |