MerchantBench report verification

Type: agentic-system-analyses/types/agentic-system-verification.md

Verification

The pinned reports support a bounded implementation account, with one local coverage limit below. I read all supplied members, types, the records contract and worker rules. The correction-answer files are empty; the supplied artifact check reports none. There is no structural failure to route. I independently inspected SRC-1 at the registered boundary, confirmed checkout HEAD equals the reviewed revision and found a clean checkout. No target code, tests, examples, providers or containers were executed. Implementation evidence does not establish actual agent behavior or benefit.

Runtime and enforcement

I traced request construction, provider invocation, tool-result accumulation, context selection and continuation in SRC-1, agent/baselines/react_160k_compact_30k.py, and observation/step-header transport in agent/sdk/merchantbench_tool_client.py. These support RT-RTE-react-loop, RT-CMP-openai-model and RT-OBJ-system-brief. The baseline sends rendered observation text and selected history, not all server state. Its endpoint/model string does not enforce an immutable weight revision or implement a local parameter update. Provider internals remain uninspected. RT-BAP-brief-guidance correctly distinguishes model-interpreted instructions from executable permissions.

For RT-OBJ-tool-registry, RT-RTE-act-gate, RT-BAP-handler-binding and RT-CLM-http-controls, I checked registry lookup/filtering, dispatch validation, HTTP normalization and locking, optional authorization, and successful-mutation caching. Sources: SRC-1, env/tools/registry.py, env/tools/dispatch.py, env/web/routes_agent.py, env/web/auth.py, env/web/app.py, env/core/simulator.py. Denial is enforced at dispatch, not merely by schema omission. Native and historical messages are recorded rather than executed. Step headers are optional. Cache keys distinguish agent, step and call ID; fingerprints detect conflicting reuse. These facts support conditional protocol controls, not universal sandboxing, transactional batch rollback or demonstrated crash atomicity. The reports retain those distinctions.

I checked hook-close persistence, trace finalization and snapshots in SRC-1, env/core/simulator.py, and rehydration in env/web/runner.py. Durable simulator recovery does not recreate the baseline's Python history. Source inspection supports the reported ordering, not successful recovery across every failure. RT-RTE-other-clients is supported by the deterministic client's SDK action calls and the human browser's HTTP calls; browser minimum-duration and reserved-turn checks are local controls, not server invariants (SRC-1, agent/baselines/auto_seed.py, agent/baselines/rule_based.py, env/web/templates/_human_playground_script.html).

RT-RTE-hermes-launch correctly stops at the MerchantBench-side profile and subprocess boundary. I checked configuration rewriting, initial skill copying, existing-home reuse, inherited environment and adapter launch in SRC-1, env/web/runner.py. They establish neither external skill acceptance nor actual summarization. RT-RTE-hosted-eval correctly describes a separate Docker/token envelope: local-image checks, bridge creation, scoped agent credentials, terminal polling, metric extraction and cleanup are present in eval/run_eval.py. Neither route establishes adversarial isolation or automatic improvement.

Memory and context

I traced the path functions, write-size check, overwrite-before-archive ordering, whole-body read and history-marker counting in SRC-1, env/tools/tools.py, against MEM-OBJ-memory-doc, MEM-OBJ-memory-history, MEM-RTE-document-write, MEM-RTE-document-read and MEM-BAP-document-guidance. The latest body and archive are distinct objects. HTTP schema admission and byte limits do not assess truth or usefulness. Filename sanitization is not injective for arbitrary IDs; the report properly avoids promoting the ordinary-ID assertions in tests/test_memory_doc_tools.py into universal isolation. I read those tests as source, not passing-test evidence. Human save/read bindings and textarea delivery are implemented in env/web/templates/_human_playground_script.html; an additional automatic human read path is the local limit below.

For RT-RTE-context-trim and MEM-OBJ-history, I followed the threshold branch through reminder or immediate trim, successful action handling, suffix selection and later request assembly. I also checked Forbidden retry redaction, malformed-argument wrapping, EOS-only retention and stale-step history slicing. Sources: SRC-1, agent/baselines/react_160k_compact_30k.py. The reports correctly characterize deletion and filtering rather than a generated summary, compulsory save or hard provider-token ceiling. A successful document read can still be removed before the next model request. These qualifications preserve the distinction between API delivery, model-visible delivery and behavioral reliance.

MEM-OBJ-step-traces and MEM-RTE-trace-inspection are supported by the per-step writer and scoped historical readers in SRC-1, env/storage/agent_log.py, env/web/routes_agent.py, and the browser trace loader in env/web/templates/_human_playground_script.html. The Hermes reader opens external-format SQLite data read-only and filters active messages unless compacted messages are requested; this is display selection, not evidence of external model context. MEM-CLM-telemetry-limits accurately treats reported compaction/evolution counters as accounting rather than accepted learning (SRC-1, env/web/leaderboard.py, baseline context reporting).

I checked MEM-CLM-state-boundary against SRC-1, env/storage/snapshot.py, env/storage/replay.py, and env/web/runner.py: checkpoint/delta reconstruction and SQLite rehydration have domain/recovery consumers, not an adopted-advice consumer. MEM-RTE-run-withdrawal is supported by deletion's stop/lease/connection/removal sequence and terminal runtime release, which preserves disk artifacts. The report's word unload describes that distinction; it does not establish an additional API named unload.

Epistemic and workflow findings

I checked all epistemic declarations against their named source transformations: EPI-OBJ-store-snapshot and EPI-RTE-observation-render map selected simulation fields into text in SRC-1, env/tools/observation.py; EPI-OBJ-daily-report and EPI-RTE-report-import import file content with computed dates and a persisted notice cursor in env/data/daily_reports.py and env/tools/tools.py. Date selection does not authenticate the unavailable private content. EPI-RTE-cash-pipeline derives known-order aggregates while explicitly denying exact settlement timing and guaranteed full proceeds in env/tools/tools.py. Its arithmetic classification is conditional on those database/cash premises, not a measured forecast guarantee.

EPI-OBJ-memory-doc and EPI-RTE-memory-document preserve operational byte admission, retention and requested recall without asserting evidence-consuming truth acceptance. EPI-OBJ-eval-metrics and EPI-RTE-wealth-metric correctly identify the last-series-point calculation, cash fallback and returned result consumer in SRC-1, eval/scoring.py, eval/run_eval.py. Their authority is bounded outcome reporting, not component attribution, theory criticism or automatic successor installation. EPI-CLM-horizon-evaluation accurately attributes the README's long-horizon rhetoric while withholding claims that wealth directly proves coherence, learning or causal benefit. The independent qualifications on learning, reflection and self-improvement are not an unsupported system-wide absence.

The workflow's WF-RTE-batch-runs is supported by direct tracing of SRC-1, scripts/run_batch.py, registry bootstrap configuration and the baseline's failure branches. Creation/status errors report abandoned identities without stopping those runs; stopped/error states precede finished-phase success; programmatic results are ordered tuples, while the CLI reports aggregate exit status rather than writing scored JSON. Inner EOS is not outer delivery. I checked these distinctions against both runtime and workflow annotations rather than treating the workflow's supplied runtime citations as independent verification.

All declared route statuses remain source wiring with operation uninspected. No evidenced-absence records are declared. I checked every declared Part of: relation: local history and observation rendering belong to the ReAct loop; memory write/read and the combined epistemic memory chain belong to the action gate; wealth extraction belongs to hosted evaluation. The combined memory-chain record remains a valid container alongside the separately described write/read parts. Trace storage spans more than one route and is not incorrectly forced into a single parent.

The reconciliation supersession of the epistemic scratchpad object by the memory scratchpad object is justified by identical file identity. Both declarations remain, and the archive is not conflated with the latest body. Batch delivery, hosted scoring and context maintenance are not duplicates. The epistemic trim row links the scratchpad as its candidate association, but expressly confines its transformation assessment to code deletion; its surrounding account and reconciliation do not assert that trimming edits the file. Read together, this does not warrant a second memory-curation mechanism or require a material correction. I found no unresolved cross-report contradiction requiring an Unresolved conflict: paragraph. Default-scenario memory availability and Hermes-overlay denial are different configurations, not conflicting findings (SRC-1, env/scenarios/default.yaml, env/scenarios/agents/hermes.yaml).

Omission questions came from the frozen tracked tree and producer/consumer dependencies, not just the submitted anchors:

  • Could another shipped consumer automatically restore archived documents or traces to ReAct? Repository-wide document/history references in env, agent and tests, followed through handlers, baseline and trace readers, supported only latest-body recall, archive numbering and human inspection. This supports the reports' bounded search claim, not absence of arbitrary external consumers.
  • Could alternate clients change the assertion that document delivery always waits for an individual human request? Following the human tool list to its automatic refresh caller found the local omission below. Separately, the deterministic daily-report client parses report text into sourcing queries; the reports already cover that client coarsely and explicitly leave its detailed epistemic policy unassessed (SRC-1, agent/baselines/auto_seed.py). This does not contradict the scoped ReAct account or establish semantic acceptance.
  • Could dashboard artifacts or experiment metadata justify a broader complete epistemic inventory or automatic improvement conclusion? Tree/search inspection and entry-point reads identified dashboard catalog diagnostics and experiment-group metadata (SRC-1, env/web/catalog_diagnostics.py, env/web/experiment_groups.py, env/web/routes_dashboard.py). Those equations and downstream analytical comparisons were not fully verified. They do not justify a system-complete negative, which the reports explicitly withhold; the accepted positive loop and scoped scoring-consumer conclusions remain supported.

Further omission expansion stopped because the selected challenges have either source-supported dispositions or the explicit local limit below. No identified material contradiction or unsupported broad comparison remains undisposed. Uninspected diagnostic equations, private artifacts, external agents and provider behavior remain unassessed; this judgment does not certify a complete inventory of every human-facing diagnostic or every possible consumer.

Blockers

none

Limits

  • MEM-RTE-document-read describes model-requested and human-button reads but does not inventory the human browser's automatic refresh alternative. In SRC-1, env/web/templates/_human_playground_script.html, AUTO_TOOLS includes read_memory_doc; observeLoop calls runAutoRefresh() on a new step, and that function calls the available tools unless automatic refresh is skipped for low quota. The existing result handler delivers the document into browser memory state/textarea, subject to preserving dirty edits. Thus readers must withhold a complete comparison of document-selection triggers across human and model clients, or any inference that every human document delivery requires a separate read-button action. ReAct's explicit-read architecture, whole-document addressing, storage admission and unobserved model reliance remain valid; the report's no-automatic-injection statement is explicitly scoped to the baseline/observation route. This is an omitted alternate consumer trigger in a partial inventory, not a false system-wide absence or an unsupported emitted comparison value. The source's automatic tool list is:

const AUTO_TOOLS = [ ["get_store_snapshot", {}], ["query_order_updates", {include_ordered: true, page: 1, page_size: 100}], ["query_open_orders", {page: 1, page_size: 50}], ["query_supply_chain_anomalies", {mode: "new"}], ["query_supply_chain_anomalies", {mode: "now"}], ["read_memory_doc", {}], ]; --- env/web/templates/_human_playground_script.html