ACE synthesis

Type: agentic-system-analyses/types/agentic-system-synthesis.md

Bounded synthesis

ACE implements an improvement-directed context-change pathway: task-solving responses and outcome feedback feed model-generated reflection, which feeds retained playbook additions and utility counters delivered to later model calls. This is implemented trace-fed adaptation, not merely storage of logs. Whether the retained changes improve future capacity, or even change actual model decisions, remains unobserved (RT-RTE-sample-adaptation, MEM-RTE-counter-update, RT-BAP-playbook-guidance).

The evidence is source inspection of https://github.com/ace-agent/ace at commit 82709de050e1db6e6ef2f07bcb0393560b94992a, with an applicability cutoff of 2026-10-18. SRC-1 supplies implementation, design declarations and attributed outcome reports, but no inspected execution trace or causal experiment. The target is a memory/knowledge/context-engineering system covering sequential ACE, ACEBatch, their components and shipped Finance and two Mind2Web callers. Its boundary is complete artifact, partial loop: the AppWorld submodule contents are unavailable. No conclusion here establishes its environment-feedback loop or verifies the README's benchmark gains (boundary and evidence, source register, EPI-CLM-context-quality).

From configured task to retained guidance

Python schedules the work. Generator, reflector and curator are model-call roles, not independently scheduling agents. The operator configures datasets, processor, model clients, output paths and optional initial text. The representative workflow is sequential offline Finance with nonempty validation and test sets, positive scheduling intervals and the optional analyzer disabled. On normal completion it delivers saved playbooks, metrics and a results dictionary, rather than a human-approved financial decision (workflow account).

Each generator invocation receives the entire current playbook, question, context and current reflection in a formatted prompt. There is no question-dependent fixed retrieval stage before generation. The generator reports bullet IDs afterward; code uses these to select parsable playbook lines for the reflector. This narrows the reflector's view, not the preceding generator input, and does not prove which advice actually caused the answer. Calls receive values rather than access to a persistent shared model conversation (MEM-OBJ-playbook-content, RT-RTE-sample-adaptation).

A task processor compares the extracted answer with the dataset target. Incorrect answers trigger bounded reflection/regeneration rounds; initially correct answers still receive a reflection. Only the last reflection proceeds to sequential curation. At scheduled steps, the curator proposes additions; code checks outer structure and required keys, then applies supported ADD operations. No per-addition accuracy veto or human approval is required. Post-curator generation clears the reflection slot but retains installed playbook advice. The token budget is prompt policy, not a hard size bound (MEM-OBJ-reflection-content, EPI-RTE-curator-admission, RT-RTE-sample-adaptation).

The no_ground_truth option withholds the literal target in specified prompt inputs, not all supervision. Target comparisons still determine feedback, retry branches and stopping. Thus the inspected drivers do not establish fully label-free adaptation, regardless of the broader self-supervision claim (RT-CLM-context-improvement, RT-RTE-sample-adaptation).

ACEBatch changes the update process materially. Parallel samples work on a snapshot; selected reflections are grouped, optionally duplicated/shuffled, and sent to independent proposers. A reducer receives the base playbook and proposed operation groups, not complete original traces or proposer reasoning. Its final ADD operations are applied once. Batch curation runs per batch rather than through the sequential frequency gate (RT-RTE-batch-adaptation, MEM-OBJ-operation-groups).

What persists, what can change, and what checks license

Retained memory consists principally of natural-language advice and symbolic IDs/helpful/harmful counters in Python strings and text snapshots. Remote model parameters are not updated by inspected ACE code; exact provider weight identities and provider-side retention remain unresolved. Temporary analyzer embeddings do not establish a persistent vector store (MEM-OBJ-playbook-content, MEM-OBJ-bullet-metadata, RT-CMP-generator, RT-CMP-reflector, RT-CMP-curator, MEM-OBJ-analyzer-embeddings).

Counters encode reflector judgments, not independently measured causal utility. Within one update call the last tag for an ID wins; batch aggregation is not summed voting. Harmful counts do not automatically withdraw advice. Curator reasoning can survive in call logs, but installed bullets have no enforced separate rationale link, and no automatic historical-log consumer is established (MEM-RTE-counter-update, MEM-OBJ-call-logs).

ADD application preserves existing lines locally. Optional analyzer maintenance does not: embedding similarity selects groups for model-mediated replacement, and failed merging can fall back to the first bullet while removing the others. Similarity and regex admission establish neither equivalence nor meaning preservation. The public merge-disabled alternative also removes group members. Arbitrary initial-file loading is another independent replacement boundary (RT-RTE-bulletpoint-merge, RT-RTE-playbook-reload).

Answer checking has narrower authority than lesson validation. FINER field matching, formula value/string matching and Mind2Web action-description matching do not validate the generating explanation or transferable strategy; Mind2Web does not execute a browser action. FINER additionally routes model-derived prediction fragments to Python eval when preceding target evaluation succeeds. Extraction does not restrict those fragments to numeric literals. This is an implemented host-interpreter path, not an observed exploit or a claim that every answer executes (EPI-RTE-answer-match, RT-RTE-finer-scoring).

Scheduled offline validation selects a whole best_playbook snapshot on strict accuracy improvement from a zero tracking threshold. Training continues from current text rather than rolling back to best. Without a qualifying validation improvement the initial snapshot remains best. Errors can be skipped, so metrics need not cover all scheduled samples. Normal completion saves current and best text separately; final offline testing uses best. A later caller can explicitly request a saved file and load its whole text, but loading restores neither ID allocation history nor a complete run checkpoint. Online mode instead tests each labelled window before training on it and carries changes into later windows; eval-only omits adaptation (RT-RTE-evaluate-select, MEM-OBJ-best-playbook, RT-RTE-playbook-reload, RT-RTE-online-windows).

Scenario-relative assessment

For the inspected bounded, target-scored adaptation scenario, ACE wires experience into later behavior-capable inputs. Code enforces delivery; models interpret the advice without host-enforced obedience. Structural admission permits operational reliance before any established per-lesson evidence-consuming truth acceptance. No concrete generated lesson settles whether its content copies, derives, conjectures or only instructs. Whole-snapshot validation does not supply that missing acceptance history (RT-BAP-playbook-guidance, EPI-RTE-curator-admission, EPI-RTE-answer-match).

The independent property assessments remain distinct (RT-RTE-sample-adaptation):

  • Theory-builder condition 1, localized theory: uninspected operation; prompts request explanatory content, but no actual whole theory instance was supplied.
  • Condition 2, consumption: uninspected operation; delivery does not show decisions depended on what a theory said.
  • Condition 3, content-directed criticism: uninspected operation; requested diagnosis does not establish criticism naming a challenged claim, result and placement of blame.
  • Condition 4, iteration: wired pathway from reflection/update to another round; iteration of an actual criticized theory is unobserved.
  • Learning: no demonstrated improvement of future capacity through criticism of a consumed theory. This does not negate the positive trace-fed write-and-later-consumer wiring.
  • Reflection: a feedback channel represents prior reasoning and reaches later generation, but both causal directions of actual reflective behavior remain unobserved.
  • Autonomy: computation performs the bounded driver's internal proposal, decision and application roles. That does not establish an autonomous theory builder or an open-ended autonomous agent.
  • Self-improvement: a dispositional improvement-directed self-change pathway is wired, with answer accuracy specified independently of the update. Exercised self-improvement lacks evidence that subsequent operation causally depended on the retained change; successful improvement separately lacks outcome evidence.

Concrete generated artifacts linked to their traces could settle theory content, criticism and rationale delivery. Interventions on retained changes with later-task measurements could test behavioral dependence and capacity improvement; component attribution would require independent component variation. Implemented semantic admission checks or preservation checks would change the present gate assessment, while removing the FINER interpreter consumer would change its effect boundary. None of these changes or results is established by the present records.

Limitations

Limitation Affected SRC IDs and record citations Inspected boundary Conclusion prevented Evidence that would resolve it
AppWorld implementation unavailable SRC-1; source register, RT-CLM-context-improvement Main repository and unavailable Gitlink contents Full AppWorld feedback, admission, consumption and environment-effect account; generalization of target-supervision findings to that integration A separately authorized frozen AppWorld source and its complete producer/consumer routes
No inspected execution, generated theory instance or causal comparison SRC-1; RT-RTE-sample-adaptation, RT-RTE-batch-adaptation, EPI-CLM-context-quality Implementation, design and attributed reports only Actual activation, theory consumption/criticism, exercised reflection/self-improvement, learning gains and verification of reported performance Provenanced runs linking inputs, updates and later decisions; appropriately controlled outcome and component comparisons
Initial provenance and remote retained state unresolved SRC-1; MEM-OBJ-playbook-content, RT-RTE-playbook-reload, RT-CMP-generator, RT-CMP-reflector, RT-CMP-curator, profile verification Loader/API contracts and local retained objects Trustworthy initial authorship/approval, fixed remote weight identity, complete memory inventory or complete-value comparisons Initial-content provenance and admission records; pinned model identities and inspectable provider retention/consumer contracts
Candidate semantics and historical rationale consumption unresolved SRC-1; EPI-RTE-curator-admission, MEM-OBJ-operation-groups, MEM-OBJ-call-logs Proposal transformation, installed text and logging paths Produced accepted novel knowledge, guaranteed semantic consolidation, or later use of logged reasons Actual linked candidate/check/disposition artifacts and a demonstrated later rationale consumer
Dataset construction and deployed effects not audited SRC-1; EPI-RTE-answer-match, RT-RTE-finer-scoring Supplied standardized targets and inspected scoring consumers Benchmark validity, live-browser success, observed FINER execution or deployed containment Dataset provenance/preparation audit; scoped environment traces and containment evidence
Recovery and evaluation are not universal guarantees SRC-1; RT-RTE-batch-adaptation, RT-RTE-evaluate-select, RT-RTE-bulletpoint-merge Inspected error handlers, sentinel paths, scoring and analyzer fallback Failed-work exclusion, atomic rollback, complete scoring, guaranteed delivery or preservation of all prior advice Repaired and independently checked handlers/transactions, complete evaluation accounting and preservation checks, with scoped run evidence
Formal containment inventory incomplete; carried report-verification limit SRC-1; MEM-OBJ-operation-groups, RT-RTE-batch-adaptation, MEM-OBJ-analyzer-embeddings, RT-RTE-bulletpoint-merge, report-verification limit Supported single-route intermediate descriptions omit explicit Part of: relations Treating the emitted relation graph as a complete containment inventory; mechanism findings remain supported Declaring-member corrections adding the supported explicit containment relations and renewed verification