MerchantBench synthesis
Type: agentic-system-analyses/types/agentic-system-synthesis.md
Bounded synthesis
From configured experiment to repeated store decisions
MerchantBench is a shipped benchmark workflow for operating a simulated store, not a general-purpose agent operating system. This account covers its public simulator, agent clients, tools, persistence, launchers and evaluators at commit d9e5bd7519ab100193c21778aecfb59e30f24651, with an applicability cutoff of 2026-10-15. SRC-1 supplies inspected implementation and attributed design documentation only. No target execution, passing tests, provider behavior, observed run or causal experiment is established. Private datasets, arbitrary submitted agents and the separately installed Hermes runtime are not frozen evidence (boundary, Source register).
The public claim is evaluation of sustained and revised merchant policy over a 365-day operation. The implementation supports repeated decisions under delayed simulated consequences and terminal outcome measurement. Despite the source's contrast with a “bounded task,” the scenario, seed and horizon still bound the experiment. Wealth is an outcome proxy, not a direct measure of coherent reasoning, learning or the truth of a policy explanation (EPI-CLM-horizon-evaluation).
A representative shipped workflow starts with an operator's queued ReAct job. The batch driver supplies scenario, seed, model and bootstrap configuration, creates a run with automatic progression, and retains its run ID for polling. The registry launches the baseline; the worker owns simulated time and opens decision hooks after transitions. This is software-managed process delegation, not model-created sub-agents (WF-RTE-batch-runs).
At each hook, the baseline sends a scenario-derived brief, rendered observation text, selected process-local history and tool schemas to an OpenAI-compatible model. It does not send the full server environment. The model proposes operations; the SDK transports them; server Python handlers execute admitted calls and return results for another model hop. Store changes affect subsequent simulation and observations. The provider is a distributed-parametric component with no baseline-side parameter update or required immutable model revision; its internal behavior remains uninspected (RT-RTE-react-loop, RT-OBJ-system-brief, RT-CMP-openai-model).
What feedback and admission license
Observation rendering reshapes selected simulated state into model-visible text. The cash-pipeline diagnostic derives aggregates from known cash and orders; it explicitly does not guarantee exact settlement timing or full sale proceeds. These are scoped simulation facts and deductions conditional on database premises, not validated claims about real commerce. Daily opportunity reports follow a different route: date-selected private Markdown is imported verbatim, and computed date labels do not authenticate its contents or warrant its market claims (EPI-RTE-observation-render, EPI-RTE-cash-pipeline, EPI-RTE-report-import).
Server admission checks liveness, hook state, quota, a step header when supplied, tool grants and argument validity; handlers can reject business-invalid operations. The model proposes, software can veto, and no human must approve each action. This licenses execution, not strategic correctness or truth. Dispatch enforces denial even for a fabricated tool name, but its authority covers HTTP environment calls, not external agents' native tools or direct host Python invocation. Authorization is optional and configuration-dependent. Mutation replay is scoped to agent, step, call ID and argument fingerprint; neither exactly-once execution across arbitrary failures nor whole-batch rollback follows (RT-RTE-act-gate, RT-CLM-http-controls).
Keeping experience available
The strongest supported memory contribution is an optional experience-to-note-to-later-context path. The foreground model can author a durable run-local Markdown document from accumulated conversational and tool material, and explicitly read it later. Software admits the body without per-content human approval; a human editor provides a separate manual write path. This affords trace-fed durable guidance, but does not establish an exercised update, improved future capacity or a particular semantic transformation such as consolidation (MEM-RTE-document-write, MEM-RTE-document-read, MEM-BAP-document-guidance, profile verification).
The mechanism is one whole latest document per run and sanitized agent filename, plus an adjacent archive of full submitted versions. Admission checks protocol/type and a 256 KiB limit, not evidential adequacy. Overwrite precedes archive append, so the two writes are not one transaction. The normal read returns only the latest body; the archive's established later consumer counts version markers, not restores old advice. Delivered notes supply advisory context through a tool result, not executable authority or standing system-message instructions (MEM-OBJ-memory-doc, MEM-OBJ-memory-history, MEM-RTE-document-write).
ReAct's “compaction” removes older turns rather than generating a summary. Defaults trigger around 160,000 tokens and target a 30,000-token suffix, using reported or estimated size rather than a hard provider-token bound. When writing is available, the same foreground model receives a save reminder; successful saving is not a prerequisite for eviction. Later requests automatically reuse surviving history by recency, role boundaries and budget. A requested document read can itself disappear before the next model call. Removed material remains available only through separately retained files or inspection routes, not pointers in the model context (RT-RTE-context-trim, MEM-OBJ-history, MEM-RTE-document-read).
The default scenario permits document tools despite contrary agent documentation; the Hermes overlay denies them. Hermes profile setup, copied skills, external-session display and evolution counters do not establish external compression, skill learning or actual model delivery. ReAct requires explicit document reads, but human delivery has an omitted alternative: the browser can automatically refresh the document on a new step, subject to availability, quota and dirty-edit handling. The report verification retains this local inventory limit; neither every human read nor every client trigger should be inferred from the selected ReAct path (boundary, RT-RTE-hermes-launch, MEM-CLM-telemetry-limits, report-verification Limits).
Recovery, delivery and what would change the assessment
Provider failure, missing tool calls or exhausted hop budgets lead the baseline to attempt end-of-step; worker timeout independently prevents indefinite waiting. A stale-step response discards newly accumulated local history, not earlier accepted effects. Simulator rehydration restores domain/runtime state, not the baseline's Python history. These are source-supported recovery mechanisms, not demonstrated crash-consistency or completed-work guarantees (RT-RTE-react-loop, RT-RTE-act-gate, MEM-CLM-state-boundary).
Closing a hook is not delivering the full task. Batch completion uses finished lifecycle status after checking stopped/error states. Its programmatic output is ordered result tuples; the CLI reports aggregate success or failure rather than scored JSON. Creation/status failures can leave reported abandoned runs active. Hosted evaluation is separate: it launches containers with a token boundary, polls terminal status and writes outcome metrics. Its wealth score uses the last net-assets point or a cash fallback, so terminal status, survival and metric count matter. Neither consumer automatically installs an improved successor policy (WF-RTE-batch-runs, RT-RTE-hosted-eval, EPI-RTE-wealth-metric).
For the bounded seeded-store scenario, the architecture therefore supports repeated model-selected actions, delayed evidence, optional retained advice and outcome reporting. Actual adaptation and benefit remain unresolved. The independent properties must not be collapsed into a grade:
- Theory-builder condition 1, localized theory: no concrete saved theory is established by the supplied records; a writable document alone does not establish its content.
- Condition 2, consumption of what the theory says: delivery routes are wired, but theory-dependent decisions are uninspected.
- Condition 3, content-directed criticism: challenged claims, blame placement and resulting revision or changed reliance are not established by validity checks or wealth measurement.
- Condition 4, iteration of criticism: retained criticism shaping a next round is not established by repeated hooks or version archives.
- Learning from criticism: the optional retention path is a supported contribution, but no comparison establishes that criticism of a consumed theory improved future capacity.
- Reflection: arbitrary scratchpad self-description does not establish a self-representation with both causal directions.
- Autonomy: action selection without per-action human approval is wired in ReAct; autonomy of every internal theory-building operation is not established.
- Self-improvement: optional notes could influence later operation, but evidence-responsive determination of an actual update and subsequent behavior causally depending on it are unestablished. Exercised self-improvement and successful improvement therefore remain unsupported at this run boundary.
These are limits on the inspected pathways, not evidenced system-wide absences or claims about opaque model processing (RT-RTE-react-loop, RT-RTE-context-trim, EPI-RTE-memory-document, EPI-RTE-wealth-metric, epistemic bounded conclusion).
Linked input/output traces showing a save, its surviving recall and a consequential later decision would establish exercised continuity more strongly. Retained claim-specific criticism, resulting changed reliance and its next-round consumer would change the theory assessment. Matched outcome comparisons would support performance claims; independent variation of the memory or criticism mechanism would be needed for component attribution. Inspectable Hermes update/consumer routes would resolve that separate branch. None of those additions is supplied here. Source-read log: none; this synthesis adds no source evidence.
Limitations
| Limitation | Affected SRC IDs and record citations | Inspected boundary | Conclusion prevented | Evidence that would resolve it |
|---|---|---|---|---|
| No execution or intervention evidence | SRC-1; RT-RTE-react-loop, EPI-RTE-wealth-metric | Public implementation/design only; no target execution | Actual adaptation, reliability, performance benefit or component effect | Provenanced run traces and outcome comparisons; independent component variation for attribution |
| Omitted automatic human document-refresh trigger, carried from report verification | SRC-1; MEM-RTE-document-read, report-verification Limits | Human browser new-step refresh in env/web/templates/_human_playground_script.html; ReAct explicit read remains supported |
Complete comparison of document-selection triggers, or every human delivery requiring a read-button action | Accepted records inventorying refresh selection, quota/availability and dirty-edit conditions |
| External Hermes memory and provider processing remain opaque | SRC-1; RT-RTE-hermes-launch, MEM-RTE-trace-inspection, MEM-CLM-telemetry-limits, RT-CMP-openai-model | MerchantBench setup/readers and remote endpoint configuration only | Complete memory inventory, external skill/summary acceptance, actual external context or parameter-change findings | Frozen external implementation and provenance-linked update/consumer evidence; immutable provider identity where relevant |
| Save/recall semantics and reliance are unobserved | SRC-1; RT-RTE-context-trim, MEM-RTE-document-write, MEM-RTE-document-read, EPI-RTE-memory-document | Optional foreground save, arbitrary-body admission and request trimming | Successful preservation, consolidation, warranted knowledge, theory criticism or exercised learning/self-improvement | Actual inputs, saved versions, criticisms, surviving consumer context and behavior dependence |
| Private report content and research artifacts unavailable | SRC-1; EPI-RTE-report-import, boundary | Public reader and synthetic-data workflow; no private datasets or local historical runs | Warrant of imported market claims or reproduction of private-data performance | Frozen report/data artifacts, provenance and corresponding runs |
| Diagnostic coverage is selective | SRC-1; EPI-RTE-observation-render, EPI-RTE-cash-pipeline, report verification | Selected transformations inspected; other tool equations, catalog diagnostics and experiment-group comparisons not fully verified | Certification of every diagnostic, complete epistemic inventory or system-wide information-leakage negative | Focused inspection of remaining material calculations and downstream consumers |
| Protocol and persistence controls have conditional scope | SRC-1; RT-CLM-http-controls, RT-RTE-act-gate, RT-RTE-hosted-eval, MEM-RTE-document-write | HTTP gate, scoped cache, separate filesystem/database operations and hosted container setup | Universal sandboxing, stale-step rejection without headers, arbitrary-crash exactly-once behavior or transactional rollback | Explicitly broader enforcement plus failure/isolation evidence covering alternate paths |
| Sanitized document names and archive consumers are bounded | SRC-1; MEM-OBJ-memory-doc, MEM-OBJ-memory-history | Ordinary-ID test source and searched document/history references; no executed tests | Universal distinct-agent filename isolation or automatic restoration of older bodies | Injective naming contract and coverage; an implemented, traced history-restore consumer |
No unresolved cross-report conflict was reported in the supplied reconciliation. Profile verification declares no additional limits; its partial external-boundary assessments do not upgrade local affordances into observed behavior (profile verification).