Agent Workflow Memory synthesis
Type: agentic-system-analyses/types/agentic-system-synthesis.md
Bounded synthesis
Agent Workflow Memory (AWM) implements a reusable-guidance loop: collect task examples or agent traces, derive or select procedures, replace a local text file, and supply that file to a model making later decisions. Its workflows are model-interpreted instructions and demonstrations, not executable subroutine graphs. This analysis covers both shipped benchmark implementations and their producers, consumers, prompts, evaluators and alternatives at commit 8c0ff8cd11d648c8fceb99e4e42f37e3b75381b1, with an applicability cutoff of 2026-10-12. The evidence is source inspection, design instructions and attributed performance reports under SRC-1. There are no registered observed runs or causal experiments. BrowserGym execution, provider internals and downloaded benchmark inputs remain outside the frozen implementation evidence (boundary, Source register).
From task request to recorded experience
The representative progression is the WebArena pipeline because a shipped driver connects task execution, evaluation, induction and subsequent file consumption. An operator chooses a website and task range. The parent launches inference, evaluation and induction subprocesses sequentially; it passes task identities and paths, not a conversational subagent handoff. BrowserGym owns observation/action scheduling, browser effects and experiment persistence. AWM constructs the legacy agent and experiment, then renames the result directory. Each action request builds observation/history context and rereads the workflow file into the system message. The chat model selects an action; local parsing and conversion can reject malformed answers and return feedback to the same model. Parse exhaustion yields a null action with diagnostics. Rate-limit exhaustion can propagate instead. Neither recovery path establishes task success, and conversion is not authorization or isolation (WF-RTE-pipeline-handoff, RT-RTE-browser-action; SRC-1, webarena/pipeline.py, webarena/run.py, webarena/agents/legacy/agent.py).
Mind2Web has a different effect boundary. It predicts textual actions against annotated observations, compares predictions to reference actions, and writes conversations and metrics. Later observations and previous-action context come from the benchmark rather than execution of the model's predictions. Moreover, cleaned observation construction uses the first positive annotated candidate and ranked negatives, subject to the consumer's rank-based positive-presence gate. Its scores therefore describe conditional annotated replay agreement, not annotation-blind retrieval or live navigation (RT-RTE-benchmark-action, EPI-RTE-annotation-score, report verification limits; SRC-1, mind2web/memory.py, mind2web/utils/env.py).
From experience to replacement guidance
AWM's strongest supported learning-related contribution is an implemented automatic trace-to-durable-guidance path with later consumers. This is an architectural contribution, not demonstrated improvement in future capacity. Online Mind2Web saves prediction messages, projects each record to its last input and output, and prompts a model to induce workflow text. Offline induction instead uses imported annotated training trajectories. Both filter generated text and overwrite the configured workflow file without per-content human approval. Online scoring metrics do not gate admission, and an unsorted file/sample zip leaves task association unresolved (RT-RTE-mind-admission, MEM-OBJ-conversation, MEM-OBJ-mind-workflows; SRC-1, mind2web/online_induction.py, mind2web/offline_induction.py).
WebArena selects ostensibly successful source trajectories using a model-produced Boolean or external cumulative reward. The model evaluator checks apparent task completion from a restricted evidence view. Text mode receives the last thought, response and action history; vision receives the last screenshot and action history. Vision omits a separately inserted response field, but preserved send_msg_to_user(...) actions can carry answer content into that history. Actual answer presence and assessment accuracy are unobserved. Software enforces source eligibility; it does not certify the workflow generated afterward (EPI-RTE-trajectory-check, EPI-RTE-success-selection; SRC-1, webarena/autoeval/evaluator.py, webarena/autoeval/prompts.py, webarena/induce_prompt.py).
After selection, neural induction removes some actions, including scroll/noop, groups examples by task template, samples them and requests recurrent abstract procedures. The model receives query/thought/action projections, not full observations, screenshots or evaluator reasons. The writer appends fixed action reminders and replaces the whole file. The prior workflow is not an explicit proposal input. Thus replacement is regeneration, not evidenced content-directed revision of the old procedure. A rule alternative retains concrete examples and deduplicates abstract action keys; exact human y controls item admission unless automatic mode bypasses it. Human approval does not establish a named correctness or transfer criterion (EPI-RTE-trajectory-cleanup, RT-RTE-neural-admission, RT-RTE-rule-admission; SRC-1, webarena/induce_prompt.py, webarena/induce_rule.py).
These checks have different licenses. Annotation scoring derives agreement under imported references. Outcome selection admits source examples. Literal filtering and deduplication reduce content without establishing procedure equivalence. Neural instructions request generalization, but no source-linked generated output establishes whether the actual transformation only reshapes existing claims or introduces new ones. No inspected generated-workflow check establishes acceptance of a generalized claim against a named intended-use criterion. Available shopping examples contain local factual assertions, but do not establish their production or acceptance history (EPI-RTE-workflow-filter, EPI-OBJ-workflow-text; SRC-1, mind2web/utils/data.py, webarena/workflow/shopping.txt).
From retained file to later decision
WebArena appends the entire configured file to the system message on each action request. It supplies both procedural directions and factual reference context; neither is guaranteed to be followed. Mind2Web loads workflows as user-message exemplars, followed by tag-filtered random concrete examples. It admits exemplars sequentially against a token budget and stops at the first overflow, so an oversized workflow can also prevent later concrete examples from being delivered. WebArena fits its main prompt before adding workflow text, and its fitter can return an oversized prompt: there is no whole-request token invariant. These default routes request configured files rather than performing task-targeted workflow search or unsolicited memory delivery (RT-BAP-workflow-system, RT-BAP-workflow-exemplar, RT-CLM-context-budget, profile verification; SRC-1, webarena/agents/legacy/agent.py, mind2web/memory.py).
Optional retrieval is separate preprocessing, not default-pipeline integration. Random workflow selection can write a subset file. The semantic workflow branch hands metadata to a method expecting a FAISS store; concrete-example ablation constructs its store directly and is not covered by that defect. Both later consumer handoffs remain optional, and neither has observed retrieval-quality evidence. An optional in-process scratchpad can return notes through enabled history, but the shipped WebArena caller disables its display; durable cross-episode note use is unresolved (MEM-RTE-workflow-selection, MEM-RTE-exemplar-ablation, MEM-RTE-episode-memory; SRC-1, mind2web/workflow/retrieve.py, webarena/run.py).
Assessment at the declared boundary
For repeated WebArena tasks, AWM supplies a route by which selected experience can alter later standing model guidance. However, the default evaluator and inducer use different model-named filenames; old matching results can be selected while the newly evaluated task is missed. Induction scans accumulated results without a website predicate and can generate replacement text with zero selected examples. An evaluator exception also writes a dictionary where induction expects a list. The parent waits without checking child success or accepting artifacts. Consequently, source-supported coordination is not evidence of successful default evidence-fed updating. Mind2Web additionally has model/token-counter and missing exemplar-filter configuration obstacles. These are static findings, not executed failures (WF-RTE-pipeline-handoff, RT-RTE-neural-admission, runtime account; SRC-1, webarena/pipeline.py, mind2web/pipeline.py).
The independent learning and theory questions remain bounded:
- Theory-builder condition 1, localized content: retained text exposes whole claims, scope conditions and procedures. This supports inspectability of available content, not a demonstrated explanatory theory produced by induction.
- Condition 2, content-dependent consumption: delivery is wired; decisions depending on particular retained claims are unobserved.
- Condition 3, content-directed criticism: the inspected writers establish fixed-guidance derivation, eligibility gates and filtering, not stated criticism assigning blame to a consumed theory and changing it or reliance on it. Opaque model processing is not settled by this finding.
- Condition 4, iteration of criticism: repeated file replacement and retained outcomes do not establish that criticism's result shapes the next round.
- Improved future capacity through criticism: not established. The supported trace-fed artifact path remains distinct from learning as improved capacity; no registered comparison isolates its effect.
- Reflection: activity is recorded and can feed later guidance, but this does not establish a reflective theory builder or a represented method criticized against its own operation.
- Autonomy: automatic induction and automatic rule admission are computational routes; the manual rule variant requires internal human content approval. No autonomous theory-builder loop is established.
- Self-improvement: a conditional pathway can change AWM's own behavior-shaping guidance rather than only an external product. Exercised evidence-responsive self-change is not established: default evidence handoff is uncertain and no subsequent operation demonstrates causal dependence on the update. Successful improvement requires additional outcome evidence.
These distinctions are supported by the admission routes and epistemic bounded conclusion, not a system-wide grade. Inference clients implement no parameter updates; exact service weights and provider-internal changes remain uninspected (RT-CMP-chat-model, RT-CMP-openai-client; SRC-1, webarena/agents/legacy/utils/chat_api.py, mind2web/utils/llm.py). README benchmark benefit remains an attributed report, not a reproduced result or causal contribution established here (EPI-CLM-benchmark-benefit; SRC-1, README.md).
A source-linked run spanning selection, generated file, later message delivery and content-dependent action would establish activation more strongly. Controlled later-task comparisons would address beneficial capacity and attribution at the level independently varied. Generated-procedure tests would address applicability rather than merely source success. Identity-based trace joins, aligned evaluator filenames and result shapes, website filtering and stage/artifact checks would remove specific static handoff obstacles, but implementation changes alone would still not establish beneficial operation.
Limitations
| limitation | affected SRC IDs and record citations | inspected boundary | conclusion prevented | evidence that would resolve it |
|---|---|---|---|---|
| No registered execution or causal comparison | SRC-1; RT-RTE-browser-action, RT-RTE-benchmark-action, EPI-CLM-benchmark-benefit | Both shipped loops and README claims at the frozen commit | Successful end-to-end operation, workflow-dependent behavior, reproduced performance, improved capacity and causal component benefit | Source-linked execution artifacts and controlled comparisons over later tasks |
| External execution and reference inputs | SRC-1; boundary, MEM-OBJ-arena-traces, EPI-RTE-annotation-score | AWM callers/readers; BrowserGym internals and downloaded Mind2Web inputs are not frozen | Host containment, environment/reward fidelity, complete benchmark-input quality or reproduced scoring | Pinned host implementation, reward specification, datasets and ranking inputs with run provenance |
| Annotated observation construction omitted from the route account | SRC-1; RT-RTE-benchmark-action, EPI-RTE-annotation-score, report verification limits | mind2web/utils/env.py positive/negative construction and consumer rank gate |
Comparisons treating cleaned observations as independently retrieved, annotation-blind browser inputs | Complete observation-selection account and an independently constructed observation comparison |
| Alternate backend constructor obstacle omitted from the component account | SRC-1; RT-CMP-chat-model, report verification limits | webarena/agents/legacy/utils/chat_api.py requires eai_token, but the factory omits it |
Working Hugging Face backend availability or recovery through that factory | Compatible constructor/factory wiring and source-linked operation of that branch |
| Default stage handoff and input setup obstacles | SRC-1; WF-RTE-pipeline-handoff, RT-RTE-neural-admission, MEM-OBJ-autoeval-signal, RT-RTE-benchmark-action | Evaluator filenames/error shapes, unchecked child exits, Mind2Web token-counter and exemplar-filter setup | Reliable default progression or a newly evaluated task necessarily feeding the next update | Aligned setup/contracts, explicit artifact checks and a trace of successful stage transitions |
| Task association and same-site lineage | SRC-1; MEM-OBJ-conversation, RT-RTE-neural-admission | Unsorted Mind2Web file/sample zip; WebArena accumulated-result scan | Correct task attribution and exclusively same-site input to website-named guidance | Identity joins, explicit site predicates and retained input/output provenance |
| Initial retained content history and bounded corpus inspection | SRC-1; MEM-OBJ-exemplar-content, MEM-OBJ-exemplar-specifier, MEM-OBJ-arena-workflows, EPI-OBJ-workflow-text | Sampled static files and inspected shopping assertions; scoped helper reads | Complete corpus correctness, author/admission history or attribution of shipped files to induction routes | Source-linked corpus provenance, approval records and appropriately scoped content/helper audits |
| Generated workflow semantics and acceptance | SRC-1; RT-RTE-mind-admission, RT-RTE-neural-admission, EPI-OBJ-workflow-text | Fixed prompts, filters, lossy trace projections and file writers | Faithful abstraction, invariant-ID transfer, accepted generalized knowledge or a content-directed theory criticism loop | Input-linked generated artifacts, procedure-level checks with intended-use criteria, and retained criticisms shaping later rounds |
| Model evaluator license and image classification | SRC-1; EPI-RTE-trajectory-check, MEM-OBJ-autoeval-thoughts, profile verification | Last-thought/text or last-screenshot/vision evidence; Boolean consumption instead of reasons | Calibrated factual correctness, answer presence in every vision input, rationale-informed induction or complete controlled form classification of the image | Actual evaluator inputs/results, independent judgments, documented image classification and a recorded rationale consumer |
| Optional retrieval and scratchpad alternatives | SRC-1; MEM-RTE-workflow-selection, MEM-RTE-exemplar-ablation, MEM-RTE-episode-memory | Standalone retrieval with branch-specific defects; in-process notes disabled by ordinary caller | Successful semantic workflow retrieval, retrieval-quality gains or durable scratchpad trace learning | Repaired workflow index handoff, configured later consumers, retrieval comparisons and note persistence/reload evidence |
| Context and local effect boundaries | SRC-1; RT-CLM-context-budget, RT-RTE-browser-action, RT-RTE-neural-admission | Main-prompt fitting, separate workflow addition, action conversion and executable log parsing | Guaranteed workflow delivery, whole-request token bounds, universal authorization/isolation or transactional rollback | Whole-request budget enforcement and separately evidenced execution, log-consumption and write-recovery controls |
| Exact provider identity and internal mutability | SRC-1; RT-CMP-chat-model, RT-CMP-openai-client, RT-CMP-openai-embeddings | Configurable inference/embedding aliases and external endpoints | Exact-weight reproducibility or conclusions about provider-internal learning and embedding provenance | Immutable model identities and inspectable provider contracts or provenance |
No source files were read for synthesis; this account uses the supplied accepted members. The report verification's two declared limits are carried above. Profile verification declares no additional limits. No current unresolved reconciliation conflict requires a two-position limitation.