ACE report verification
Type: agentic-system-analyses/types/agentic-system-verification.md
Verification
The reports support a bounded implementation account of playbook adaptation, not a verified performance result. I read all pinned reports, their types, the record contract, reconciliation and structural findings. The four correction-answer files are empty; there are no declined objections to adjudicate. The supplied structural check reports no findings. I independently inspected source rather than treating the analysts' validation statements as semantic verification.
Boundary and evidence
The checkout HEAD equals the reviewed revision and its working tree is clean. The tracked tree agrees with the boundary, including the unavailable AppWorld Gitlink. All additional inspection used SRC-1 at that revision. No target code, tests, examples, model calls or scoring expressions were executed. Source inspection supplies implementation and doctrine/design evidence only. The reports correctly keep README outcome claims at the reported-operation layer and retain the missing AppWorld, provider, execution and causal evidence qualifications in the Source register.
Record and route judgments
I checked all 29 declarations, their supporting source anchors, evidence strength and downstream uses. Workflow adds annotations but no declarations. The following groups cover every declared record; linked records supply the retained load-bearing passages.
| Records checked | Independent source inspection and disposition |
|---|---|
| RT-CMP-generator, RT-CMP-reflector, RT-CMP-curator, RT-CMP-embedding-model | SRC-1, ace/core/generator.py, ace/core/reflector.py, ace/core/curator.py, ace/core/bulletpoint_analyzer.py, utils.py, llm.py and caller configuration. Local model names/client handles, external weights, name-based resolution and lack of inspected parameter training support the separate component assessments. This does not settle provider-internal learning or exact deployed weights. |
| RT-RTE-sample-adaptation, RT-RTE-batch-adaptation | SRC-1, ace/ace.py, ace/ace_batch.py, role implementations and prompts. Independently traced correctness branches, reflection replacement, counters, sequential modulo admission, batch snapshots, augmentation, grouping, proposals, reduction and post-curator generation. The target oracle remains active under no-ground-truth prompts. Batch curation does not use the sequential frequency gate. The descriptions of model roles, Python progression and partial trace delivery are supported. |
| RT-RTE-evaluate-select, RT-RTE-online-windows, RT-RTE-playbook-reload, MEM-OBJ-best-playbook | SRC-1, ace/ace.py, ace/ace_batch.py, utils.py and all three caller main/loader paths. Checked strict improvement from zero, current versus best consumers, test-before-training windows, file persistence and explicit reload. The initial snapshot can remain best without being the best measured baseline. Skipped errors, the all-failed return defect, normal-return-only restoration and reset identifier allocation are preserved, not converted into guaranteed recovery or delivery. |
| RT-RTE-bulletpoint-merge, MEM-OBJ-analyzer-embeddings | SRC-1, ace/core/bulletpoint_analyzer.py. Checked temporary embeddings, threshold grouping, direct merge transport, regex admission, failed-call/parse fallback, reconstruction and the public merge-disabled alternative. Even fallback can remove other group members. Similarity does not establish equivalence, preserved counters or preserved meaning. The reports correctly bound ADD-only preservation to another consumer. |
| RT-RTE-finer-scoring, EPI-RTE-answer-match, EPI-OBJ-answer | SRC-1, utils.py, eval/finance/data_processor.py, eval/finance/run.py, and both Mind2Web processors. Traced extraction through scoring and its correctness consumers. The FINER truncation deduction is correct; formula checks values rather than derivations; Mind2Web checks descriptions rather than live execution. FINER prediction evaluation requires successful target evaluation. Arbitrary extracted expression strings are possible, but neither every string nor every shipped target is a witness to interpreter execution. The records retain that condition and make no observed-exploit claim. |
| MEM-OBJ-playbook-content, MEM-OBJ-bullet-metadata, MEM-OBJ-reflection-content, MEM-OBJ-operation-groups, MEM-OBJ-call-logs, MEM-RTE-counter-update | SRC-1, playbook_utils.py, logger.py, llm.py, generator/reflector/curator implementations and both orchestration paths. Whole-store delivery precedes reported-ID filtering for reflection. Last-tag lookup is not summed voting. Operation groups omit proposer-level reasoning; installed text need not carry a separate rationale link. Log retention is not automatic historical-log consumption. Initial provenance remains unresolved separately from known later writes. |
| EPI-OBJ-reflection, EPI-OBJ-playbook, EPI-OBJ-bullet-tags, EPI-RTE-counter-update, EPI-RTE-curator-admission | SRC-1, ace/core/reflector.py, ace/prompts/reflector.py, ace/core/curator.py, ace/prompts/curator.py, playbook_utils.py. Checked candidate diagnosis versus demonstrated explanation, tag judgments versus utility measurement, parser checks versus semantic acceptance, and operational installation versus lifecycle integration. The single-line ADD reshaping deduction is appropriately restricted; arbitrary multiline content is not assigned the same preservation guarantee. |
| RT-CLM-context-improvement, EPI-CLM-context-quality, RT-BAP-playbook-guidance | SRC-1, README.md, ace/prompts/generator.py, curator prompts and call sites. The claimed benefits and label-free framing are distinguished from inspected mechanisms. Delivery is code-enforced, interpretation advisory. Wiring supports improvement-directed context change, not actual reliance, successful learning or exercised self-improvement. Theory-builder conditions remain separately assessed rather than inferred from role names or requested diagnosis. |
I also checked exception and alternate branches relevant to those conclusions. The unresolved is_api_error references in SRC-1, ace/ace_batch.py have no definition in that module or its project wildcard-import roots (utils.py, logger.py, playbook_utils.py). The handler qualification is warranted. llm.py can return an empty-response sentinel for training/testing prefixes without raising; this is not the batch API-error marker, and online prefixes differ. Curator parsing recovery also does not cover its preceding transport call. The reports do not infer universal failure exclusion from the intended rollback code.
The workflow account faithfully selects sequential offline Finance with the analyzer disabled. Its handoffs, context-only curator argument, local rejection, validation selection and normal-completion delivery agree with SRC-1, eval/finance/run.py, ace/ace.py and ace/core/curator.py. Measuring the post-curator answer does not veto retention and is not presented as a controlled experiment.
Reconciliation and relations
The three reconciliation amendments identify genuine duplicate referents: reflection response, playbook content and the counter updater. Neither source-dependent field nor epistemic assessment is strengthened by supersession. Keeping the combined incoming-tags/stored-counters object alongside the narrower stored-metadata object is defensible: neither finding becomes false when the parts are distinguished. The cross-route scorer and curator-admission declarations also remain distinct from their enclosing orchestration routes.
I checked the lack of emitted Part of: fields as well as the stated distinct-identity comparisons. The counter updater and scorer span several runtime routes. Two single-route intermediate objects have a local formal-containment omission recorded below; it does not invalidate their described producers, consumers or mechanisms. The integration dispositions preserve counter aggregation, rationale delivery, current/best separation and the shared evidence boundary. No contradictory findings requiring an unresolved-conflict paragraph remain. Empty structural findings require no routed structural blocker.
Consequential omission search and stopping decision
Search selection used the frozen tracked tree and producer/consumer dependencies, not only the reports' citation lists. I investigated whether another shipped caller, maintenance consumer or extension contract could overturn whole-playbook delivery, label-derived supervision, local-only ADD preservation, or the absence of established per-lesson semantic acceptance in the inspected routes.
Beyond the detailed paths above, I searched entry points, persistence and historical-log reads across SRC-1, ace/, utils.py and logger.py; searched both Mind2Web preparation scripts for target production and state consumers; and screened EXTENDING_ACE.md and tutorials/ExtendingDatasets.md, reading their processor and loader/invocation examples. The extension material delegates task comparison to a custom processor and reuses the same playbook loader/run interface; it does not justify replacing the reports' bounded scorer conclusions with a universal oracle claim:
- answer_is_correct() - Check if a prediction matches ground truth
- evaluate_accuracy() - Calculate overall accuracy ---
EXTENDING_ACE.md
The preparation searches identify upstream target construction, not an inspected alternate playbook-admission path. Their detailed sampling, provenance and benchmark validity remain unassessed, as the reports already say. The Finance sample configuration and one shipped FINER training row were additionally read to distinguish actual tag targets from the scorer's conditional interpreter branch; this does not establish branch activation on the shipped datasets or dataset-wide properties (SRC-1, eval/finance/data/sample_config.json, eval/finance/data/finer_train_batched_1000_samples.jsonl).
These searches leave the central conclusions supported with their existing scope. Further dataset/provenance auditing, provider inspection or exhaustive utility enumeration is not reasonably likely to change that bounded account without changing the question or evidence boundary. No identified material challenge is left undisposed. Missing observed operation, causal benefit, generated theory instances and AppWorld implementation are faithfully scoped uncertainties already retained by the reports, not additional defects or evidence of absence.
Blockers
none
Limits
- The formal containment inventory is incomplete for MEM-OBJ-operation-groups within RT-RTE-batch-adaptation and MEM-OBJ-analyzer-embeddings within RT-RTE-bulletpoint-merge. Their declarations explain the single-route intermediates in prose but omit the required explicit
Part of:relations. Their source-grounded identity, lineage and consumers remain supported by SRC-1,ace/ace_batch.py,ace/core/curator.pyandace/core/bulletpoint_analyzer.py, and by the passages in those records. Readers can retain the mechanism findings, but should not treat the emitted relation graph as a complete containment inventory. This local relation omission does not change a central runtime or lens conclusion or assert an unsupported system comparison value.