ExpeL report verification

Type: agentic-system-analyses/types/agentic-system-verification.md

Verification

The pinned reports support a bounded static account. Runtime's earlier unconditional extraction-holdout finding is repaired; no material blocker remains. The checkout HEAD equals the revision in the Source register, and its Git status is clean. The structural report-check says none. Runtime supplies one correction answer; the other answer inputs are empty. I judged the changed wording against source behavior rather than accepting the answer as evidence. No target code, tests, examples, providers or services were executed.

Independent inspection covered the complete three drivers, acting/reflection/ExpeL classes, provider and embedding dispatch, trajectory representation, benchmark adapters, maintenance templates and prompt dispatch. Persistence, answer normalization, reporting and fold-selection helper bodies were also read. These are implementation and doctrine/design evidence from SRC-1: train.py, insight_extraction.py, eval.py, agent/base.py, agent/react.py, agent/reflect.py, agent/expel.py, models/llm.py, memory/__init__.py, memory/episode.py, envs/__init__.py, envs/base.py, envs/hotpotqa/hotpotqa.py, envs/fever/fever.py, envs/fever/wikienv.py, envs/fever/wrappers.py, envs/alfworld/alfworld.py, envs/webshop/webshop.py, utils.py, tools/visualize_logs.py, prompts/templates/human.py, prompts/__init__.py, prompts/hotpotQA.py, configs/train.yaml, configs/agent/expel.yaml and README.md. Searches covered auxiliary callers, archive consumers and feedback hooks across agents, memory, environments, tools and the shipped drivers. None of these reads establishes observed operation or causal benefit.

Record support and evidence strength

I checked every declaration, its source anchors, evidence status and consequential qualifications. The two runtime components separate opaque model parameters from retained text and do not claim immutable weight identities or parameter training: RT-CMP-gpt-wrapper and RT-CMP-embedder. The four runtime routes distinguish driver scheduling, model proposal, adapter execution, retry control, rule updates and checkpoint replacement. The two authority paths distinguish interpretive rule guidance from retrieved demonstrations, not symbolic action vetoes. Training and evaluation dispatch differ; cache loading and restoration bypass critique. Pickle loading and attribute replacement do not certify provenance or semantic validity. Completed-step/token limits do not bound external retry time (SRC-1 agent/expel.py, agent/react.py, agent/reflect.py, models/llm.py, utils.py, train.py, eval.py).

All seven memory objects retain source-native identities, forms and storage, with consumers or expressly unresolved consumption. MEM-OBJ-trial-history is classified experience, not a generalized rule. MEM-OBJ-reflections separates active retry guidance from archives. Rule text, counters, retrieval documents and logs have different consumer meanings; counters are retention preferences, not correctness estimates. Selected retrieval keys need not identify the trajectory ultimately delivered: the selector uses the task's shortest trajectory. The two memory routes correctly separate auxiliary critique generation from working rule admission, and persisted-text reporting from checkpoint restoration. Annotations preserve per-unit rather than whole-prompt budgets, cache-copy survival after withdrawal, and the difference between model authorship, software admission and run-level human authorization (SRC-1 agent/expel.py, agent/reflect.py, memory/episode.py, utils.py, tools/visualize_logs.py).

The five epistemic objects, eight route declarations and three claims preserve their distinct targets and warrants. EPI-RTE-source-acquisition imports observations without endorsement. EPI-RTE-answer-check separates local normalized equality from imported external win/reward judgments; its result does not license rationale or transfer. EPI-RTE-outcome-disposition consumes that result for trajectory classification. Proposal, protocol checking and counter disposition are separate functions. Requested generality does not establish the semantic relation or truth of missing generated insights. No separate evidence-consuming per-rule truth/transfer acceptance or post-acceptance integration is established in the named extraction/parser/updater/cache paths. This scoped finding does not deny model-internal checking. Cleanup's implemented body is conditional on successful helper dispatch; it does not contradict memory's afforded invocation with missing default template keys. README efficacy remains attributed doctrine/design, not observed or causal evidence (SRC-1 agent/expel.py, prompts/templates/human.py, README.md, envs/fever/wrappers.py, envs/hotpotqa/hotpotqa.py, utils.py).

Runtime's separate theory-builder conditions, learning, reflection and self-improvement assessments distinguish editable units, requested criticism, feedback wiring, activation and benefit. The improvement-directed retained-rule pathway supports the marked dispositional self-change finding, not exercised or successful self-improvement. No evidence strength needs upgrading. No member declares an evidenced-absence record; prose negatives were checked against their named search boundary rather than read as complete system negatives.

The workflow account faithfully follows the selected non-testing HotpotQA command. I checked the testing override, Finish rejection, reflection-depth ownership, failed-retry retention, save ordering and resume. Foreground scheduling remains with the driver; model services do not acquire an independent delivery obligation. Delivery is logs and classified experience, including unsuccessful trials, not guaranteed correctness or demonstrated file delivery (SRC-1 README.md, configs/train.yaml, train.py, agent/reflect.py, agent/expel.py, envs/hotpotqa/hotpotqa.py, utils.py).

Correction, relations and conflicts

The runtime correction is supported. In RT-RTE-rule-admission, the account, selection predicate and evidence limits now distinguish driver subtraction of evaluation IDs, compare-branch idx2task lookup, success-group dictionary enumeration, and resumed reuse of saved groups. The evaluation account and RT-RTE-evaluation-context carry the corresponding qualification. Source inspection confirms all four distinctions. Retrieval independently uses the full success-history pool plus shipped examples and excludes the current task, not the entire fold. The reports do not infer actual leakage or delivered held-out experience (SRC-1 agent/expel.py, insight_extraction.py, eval.py). The earlier runtime blocker is discharged.

Every Part of: relation is substantively valid. Acquisition, answer checking and disposition are narrower training-loop functions, with evaluation overlap expressly retained. Insight proposal, syntax checking and counter disposition are narrower rule-admission functions. Containers retain additional scheduling, retention and recovery responsibilities and need not be superseded by those parts.

The three object supersessions are supported by identical rule-text, counter and auxiliary-critique referents. They preserve epistemic uncertainty rather than replacing it with stronger memory findings. Observation text is not the whole trial history; reporting is not restoration. Auxiliary generation and cleanup have distinct endpoints and conditional statuses; log inspection and log-derived summaries likewise overlap without being identical.

The remaining conflict is fully anchored: epistemic still calls runtime's holdout wording unconditional, although the pinned runtime is qualified. I dismiss this as a stale characterization of the peer report, not a current disagreement about the mechanism. Both retain the same enumeration condition, and reconciliation explicitly distinguishes the stale objection from current convergence. It changes neither the bounded extraction conclusion nor an emitted comparison finding. No source-unsettled disagreement remains. Memory's integration issues have supported dispositions: fold qualifications are carried, auxiliary default dispatch remains limited, archive retention does not establish reflection retrieval, and unobserved payloads/recovery/benefit remain scoped uncertainty.

Consequential omission questions and stopping decision

Questions were selected from the frozen tracked tree, shipped alternate entry points and producer/consumer dependencies: whether another caller bypasses admission, whether extraction holdout isolates retrieved evidence, whether auxiliary critiques or hooks add an acceptance route, whether reporting independently validates outcomes, and whether benchmark dispatch supports equivalent complete loops. Complete driver and ExpeL reads establish cache/restore bypasses and the broader retrieval pool. Searches found no shipped auxiliary-helper or prepare_new_eval caller and no direct past_reflections reader in the inspected roots. ReactAgent.step explicitly calls the empty BaseAgent.after_step; no additional operative evaluator is established there. These are scoped dispositions, not negatives about arbitrary custom callers (SRC-1 agent/expel.py, agent/react.py, agent/base.py, train.py, eval.py, insight_extraction.py, tools/visualize_logs.py).

Reporting and benchmark-dispatch alternatives yield the limits below. Further optional omission expansion stopped because the selected questions have supported dispositions and no identified material challenge remains unaddressed. Individual dataset records, exhaustive static examples and dependency internals are not needed for these mechanism conclusions. Their omission licenses no exact source-quality, uniform strategy-correctness, reproduced performance or causal-efficacy comparison.

Blockers

none

Limits

• RT-RTE-training-loop, RT-RTE-rule-admission and RT-RTE-evaluation-context must not be compared as uniformly functioning benchmark integrations. The reports already disclaim uniform branch support; direct dispatch inspection makes FEVER's restriction concrete. Its reflection inputs and compare critique instruction are None, and its success critique key is misspelled. The constructor and critique builder do not repair those values (SRC-1 prompts/__init__.py, agent/reflect.py, agent/expel.py). This leaves the selected HotpotQA workflow and separately inspected FEVER predicate in EPI-RTE-answer-check supported, but prevents extending them to successful standard FEVER training/extraction/evaluation wiring.

  fever=dict( compare_existing_rules=None, all_success_existing_riles=None ), --- prompts/__init__.py

• EPI-RTE-log-summary and MEM-RTE-log-inspection support reporting capability, not independent certification of benchmark success counts. I confirmed the score-marker predicate in recompute_stats against the adapter's reward-equals-one predicate (SRC-1 utils.py, envs/webshop/webshop.py); the epistemic record retains both passages. Withhold quantitative equivalence between plotted WebShop success and full-reward task success. Actual prevalence and aggregate error remain unobserved; the task-outcome oracle and retained-memory mechanisms remain supported.