MerchantBench boundary

Type: agentic-system-analyses/types/agentic-system-boundary.md

Boundary and evidence

MerchantBench is in scope as a model-dependent benchmark workflow. The target is the complete shipped public artifact, not just its HTTP SDK or simulator. Its intended use is long-horizon evaluation of agents operating a simulated store. The artifact includes its own ReAct agent loop, simulator, launchers, persistence, tools, human interface and evaluator. Deterministic baselines are included as alternate consumers of the same protocol, not classified as model-calling agents themselves (SRC-1, README.md, agent/submission_template/my_agent.py, agent/baselines/react_160k_compact_30k.py).

This repository contains the simulator, agent SDK and reference baselines, evaluation harness, batch runner, and test suite. Credentials, local run records, private source-system connectors, and non-redistributable datasets are intentionally excluded. --- README.md

The whole-system boundary follows the shipped loop: env/run.py builds the Flask application; env/web/app.py installs agent and dashboard routes around the run registry; env/web/run_worker.py advances the simulation; observations and scenario-derived briefs reach the agent through env/web/routes_agent.py and the SDK. The ReAct baseline builds a model request from its history and environment tool schemas, then sends returned actions through the SDK. The agent route dispatches tools, records messages and usage, and releases the simulation hook. These are in-repository callers and consumers, not excluded applications (SRC-1, the named paths and agent/baselines/react_160k_compact_30k.py).

The model-calling path is executable source, rather than an SDK-only proposal:

return self.openai.chat.completions.create(**request) --- agent/baselines/react_160k_compact_30k.py

State coverage includes simulator listings/orders/cash, SQLite-backed rehydration, delta snapshots and checkpoints, agent traces/costs, the run-local Markdown memory tools and their write history, and the ReAct history-compaction branch. Scenario configuration and composed prompts are part of the boundary because they determine what the model receives and which tools it can call. The evaluator and batch launcher are included because they create runs and consume terminal state (SRC-1, env/core/simulator.py, env/web/runner.py, env/tools/tools.py, env/tools/observation.py, env/scenarios/default.yaml, eval/run_eval.py, eval/scoring.py, scripts/run_batch.py).

Documentation and executable defaults must be distinguished. Although agent/README.md says the default scenario denies memory tools, the frozen env/scenarios/default.yaml comments out those denylist entries. Thus this boundary includes memory tools and their configured availability; it does not adopt the documentation's default-denial assertion (SRC-1, those two paths).

tool_denylist: # null = 全部工具可用；例: [search_products] → 禁用 search_products - market_brief - hot_search_terms # - read_memory_doc # - write_memory_doc --- env/scenarios/default.yaml

Frozen evidence and limits

The sole frozen evidence source is SRC-1 at full commit d9e5bd7519ab100193c21778aecfb59e30f24651; its checkout HEAD was verified against the acquisition pin. The applicability cutoff is 2026-10-15, the supplied run date, not a claim about later repository state. Overall evidence is code-grounded, supplemented by source-native design documentation. No target, tests, examples, model endpoints or container evaluation were executed. Test source establishes intended assertions, not passing tests; no observed run or causal experiment is registered.

External dependencies include Python libraries, an OpenAI-compatible model provider for ReAct, Docker for hosted evaluation, and optionally a separately installed Hermes adapter/runtime. MerchantBench's Hermes discovery, profile preparation and subprocess-launch code are included; the external Hermes runtime and its skills/session implementation are not frozen here. This prevents conclusions about Hermes-internal memory, skill evolution or actual isolation behavior beyond the MerchantBench-side setup (SRC-1, env/web/runner.py, README.md).

Private business catalogs, opportunity-report datasets, credentials, historical local runs, arbitrary submitter images and source-system connectors are not available in this public source set. Their absence prevents reproduction claims about private-data research results or submitted agents' actual performance. The synthetic-data path and private-data loader interfaces remain inside the artifact boundary. The paper, project website, news links and external Hermes repository were not acquired and cannot supply evidence. Their exclusion prevents verification of their claims, not inspection of the shipped public workflow (SRC-1, README.md, env/README.md).

Top-level tracked coverage

Included means within this analysis target, not that every file has already been inspected. Initial inspection traced the material loop above. Remaining source files at this commit may be inspected by later analysts; they are not access gaps.

Source register