Agent S synthesis

Type: agentic-system-analyses/types/agentic-system-synthesis.md

Bounded synthesis

From request to desktop action

Agent S is a computer-use runtime that turns natural-language tasks and screenshots into desktop actions. This account covers the shipped repository at commit 15776a8dc659658b1963816ea006f7af1978e595, with an applicability cutoff of 2026-10-16. The whole-system boundary includes drivers, historical generations, retained-state consumers and benchmark selection scripts—not just the prediction API. Evidence is source implementation, shipped instructions and attributed result reports from SRC-1. No desktop session, provider call, test execution or causal experiment was supplied. Hosted Simular products and external host implementations are excluded. See the boundary and evidence and Source register.

The installed command selects S3, a single-worker architecture. A human admits the task and configures generation and grounding services. The driver obtains a screenshot; the worker combines task instructions, recent observations, notes, optional trajectory reflection and any pending coding return. A generation model chooses an action under natural-language procedural guidance. Grounding translates a referring expression into coordinates; the driver executes returned Python and acquires the next screenshot. The CLI allows 15 outer prediction iterations, whereas SDK and benchmark callers have different bounds. This is an enclosing action/observation loop, not merely an answering model. See RT-RTE-desktop-loop and RT-OBJ-procedural-memory.

Action-format checking is not a safety or task-correctness gate. It counts calls by regex and trial-evaluates code against the live action interface. The worker evaluates the selected response again. Consequently an immediately effectful method can run during validation and again during conversion; retries can add further effects. Exhausted formatting retries return the last response rather than enforcing rejection. The outer CLI executes ordinary returned code directly despite a nearby permission comment. Host permissions constrain deployed effects, but no captured configuration establishes isolation, per-action approval or rollback. These are implementation findings, not executed exploit observations: RT-RTE-action-check.

Screenshots supply evidence about rendered state, not endorsement of displayed assertions. Grounding's digit parsing validates an output shape, not correspondence to the intended element. A separate OCR/text-span path can remove numeric tokens during cleanup and default to element zero when the model returns no digits; its remaining screenshot input does not restore the omitted table text. Thus correct coordinate arithmetic is conditional on recognition and target selection. See EPI-OBJ-screen-evidence and EPI-RTE-text-grounding.

Coding handoff, recovery and delivery

The representative connected workflow is S3 desktop execution with local coding explicitly enabled. Coding is off by default in the installed CLI; OSWorld supplies a controller through a different caller. When selected, coding delegation is synchronous: the action interface supplies the original task or an explicit non-None override, a screenshot and a controller to a fresh coding-model context. The override is not semantically checked for task fidelity. The coding model chooses Python/Bash, consumes controller feedback and can revise its scripts for up to its round budget. Local Bash receives a timeout; the Python call has no corresponding timeout argument. Capability opt-in is not approval of individual scripts. See RT-RTE-code-delegation and the workflow account.

Normal coding FAIL and budget exhaustion return control to the worker, allowing possible GUI recovery rather than automatically failing the outer task. They do not reverse earlier file effects. The coding frame keeps detailed controller feedback out of the worker dialogue. Its return includes a summary and action history, but the summary is generated from actions/thoughts rather than direct controller-result messages. The next worker invocation receives that reduced return and then clears the cache. Separate contexts therefore isolate dialogue, not shared desktop/file effects: MEM-OBJ-code-result, RT-BAP-code-return.

Worker instructions require GUI verification, including reopening applications after file edits. This is a meaningful requested check, but DONE does not inspect verification history. DONE and FAIL both stop the CLI through the same completion-dialog branch; iteration exhaustion also stops work without certifying success. Delivery is changed desktop/files, streamed plans and driver termination, not an independently validated result object. In the OpenClaw bridge, success means subprocess exit zero, not task satisfaction; its step argument configures history length rather than the CLI's fixed outer bound. See RT-CLM-gui-verification, RT-RTE-desktop-loop and RT-RTE-openclaw-bridge.

For an open desktop request, the implementation supports autonomous model-owned action selection within caller-granted capabilities, with local feedback and possible recovery. It does not establish reliable completion or enforced verification. A trace connecting a specific request, coding effects, reopened GUI evidence and the terminal decision would establish exercised behavior; an enforced verification postcondition would change the architectural guarantee.

What persists and affects later work

The strongest supported learning-related contribution is wired automatic transformation of recorded work into retained material with later consumers. Historical S1/S2 persist task and subtask summaries for later planning; S3 retains a coding continuation summary for the next worker invocation; offline captioning retains descriptions for later comparison. These are distinct trace-fed pathways, not evidence that all generations share a continual-learning architecture. Their wiring does not establish improved capacity for future action. The profile verification confirms these distinctions without treating trace learning as demonstrated improvement.

Historical S1/S2 interpose manager/worker planning. Their CLI trajectories feed model summaries admitted into JSON on missing keys, without individual human approval or a factual-success gate. Task summaries can retain failure reasons; episodic extraction requests correct necessary recipes and need not retain discarded steps or their reasons. Existing keys suppress replacement, so subsequent experience does not automatically repair an earlier summary. Terminal CLI ordering also prevents assuming complete trace capture or a final pending-episode flush. See RT-RTE-historical-experience, MEM-RTE-s1-experience, MEM-OBJ-narrative-memory and MEM-OBJ-episodic-memory.

Later manager/worker retrieval requests one summary by similarity over embedded keys. The vectors rank candidates; the selected prose supplies optional advice, not direct action replay or certified knowledge. Exact-top skipping can fail for a singleton identical key, and S2 modifies retrieved recipes before delivery. Separate exact-instruction caches retain generated queries and search/model responses; changed observations do not invalidate a cache hit. Their fusion can supply planning guidance, but the epistemic overlay does not separately assess this whole chain. An exposed alternative save/finalize API has no established shipped lifecycle caller and cannot repair CLI finalization by inference. See MEM-RTE-experience-retrieval, MEM-OBJ-embeddings, MEM-RTE-query-cache and MEM-RTE-knowledge-save.

Current S3 notes instead occupy an action-interface RAM list and are pushed in full into worker messages. Worker reset can reuse that interface, so it need not clear notes; this is not a demonstrated cross-process knowledge bank. Working-context controls are another mechanism: S3 removes older images for selected engine types and older turns for others, without producing a replacement summary or reloading removed images. Text can continue growing in the image-only branch. Notes, coding returns, current reflection and historical disk experience therefore have different lifetimes and consumers. See MEM-RTE-notes-delivery and the memory working-context annotations.

Offline comparison is a separate continuation

The OSWorld driver records actions, screenshots and externally evaluated scores. A separate offline pipeline turns screenshot/action pairs into captions, then asks a comparative model which completed rollout better satisfies the task. It receives caption answers and available initial/final image pairs, not all intermediate evidence or narrator thoughts. Model task-fit judgment supplies substantive evaluation; integer/range handling admits a selected directory for scoring/reporting. Saved task keys suppress rejudging. Scores identify a variance subset and support retrospective scoring; they are not expected answers supplied inside judge messages. See RT-RTE-osworld-driver, RT-RTE-rollout-selection, MEM-OBJ-fact-captions and MEM-OBJ-judge-results.

This is an implemented evidence-consuming comparative selection procedure, not merely unchecked logging. Its intended criterion and operational scope are bounded to supplied completed rollouts. It neither certifies each caption nor installs a live successor policy. The prompt's assertion that facts are accurate exceeds the demonstrated caption checks, although before/after evidence instructions and endpoint images provide genuine checking opportunities. Public Best-of-N gains remain attributed reports; no frozen trials identify selection accuracy, magnitude of benefit or a component's causal contribution. See EPI-CLM-fact-accuracy and EPI-CLM-bbon-performance. Actual rollout artifacts, evaluator definitions and controlled comparisons would change that assessment.

Independent property conclusions

Reflection has an inspectable architectural contribution: changes in the visible trajectory feed a model self-assessment, and that assessment returns as advice capable of changing subsequent action. Accurate criticism, actual changed behavior and benefit remain unobserved. This reflective feedback path is separate from the autonomous action-choice roles above: EPI-RTE-trajectory-reflection.

For explanatory theory building, each condition remains independently unestablished in the inspected account: (1) no inspected instance establishes a localized explanatory theory; (2) no supplied behavior establishes decisions depending on such theory content; (3) no supplied criticism identifies a challenged explanatory claim, blame placement and resulting changed reliance or revision; (4) no supplied iteration retains that criticism's result to shape the next theory round. Task-progress reflection and rollout ranking do not establish these conditions. This is not an exhaustive absence claim about unassessed historical routes or opaque model processing. Nor is there evidence that criticism of a consumed theory improved future action capacity. See EPI-RTE-trajectory-reflection and the epistemic bounded conclusion.

For self-improvement, historical retained guidance can enter behavior-determining organization through later prompts. That is a positive potential contribution, not merely an external work product. Across later historical tasks, however, the record does not establish an objective-linked, evidence-responsive update followed by subsequent operation causally dependent on that update. Improved capacity and successful improvement remain unobserved. S3 reflection, coding returns and offline selection must not substitute for those missing links: changing a work file or selecting a finished rollout is not by itself changing the live agent's organization. A provenance-linked update/use trace and objective-linked comparisons would support stronger claims. See RT-RTE-historical-experience, MEM-RTE-experience-retrieval and the epistemic bounded conclusion.

Limitations

limitation affected SRC IDs and record citations inspected boundary conclusion prevented evidence that would resolve it
Historical query/search/fusion is not separately assessed in the epistemic overlay; this is the report verifier's retained limit. SRC-1; MEM-RTE-query-cache, MEM-OBJ-formulate-query, MEM-OBJ-rag-knowledge; report-verification Limits Recorded S1/S2 cache and planning consumers; selected epistemic route families Complete historical epistemic inventory or route-by-route warrant comparison. The omission does not show absent truth-apt content, acceptance or other epistemic functions. Accepted records separately assessing query generation, cached search, fusion, checks and authority transitions.
Implementation is not observed operation or benefit. SRC-1; RT-RTE-desktop-loop, RT-RTE-code-delegation, RT-RTE-historical-experience, RT-RTE-rollout-selection Shipped drivers, prompts and retained-consumer wiring; no supplied sessions Realized completion, exercised GUI verification, useful recall, improved capacity or exercised beneficial self-change Provenance-linked execution and later-use artifacts; appropriate objective-linked comparisons for benefit claims.
Initial archive/local content and vector provenance are unresolved. SRC-1; RT-RTE-kb-bootstrap, MEM-OBJ-narrative-memory, MEM-OBJ-episodic-memory, MEM-OBJ-embeddings Downloader and local transformations, not frozen release payloads Initial entry truth, original authorship/review, trace mixture, vector-version consistency or complete initial-memory classification Frozen archives/local stores with per-entry and per-vector provenance and admission history.
Provider and deployed effect envelopes are external. SRC-1; RT-CMP-lmm-agent, RT-CMP-grounding-model, RT-CMP-embedding-engine, RT-RTE-code-delegation Inspected clients, controller calls and configurable endpoints Immutable model identity, hidden retention/training, model accuracy or deployed isolation/permission guarantees Fixed service/model identities, provider-state evidence and captured deployment controls.
Reduced consumers lack complete result or rationale evidence. SRC-1; MEM-OBJ-code-result, MEM-OBJ-episodic-memory, MEM-OBJ-fact-captions, MEM-OBJ-judge-results Summary construction, recipe extraction, answer-only loading and selected-directory scoring General reason preservation, faithful semantic compression, accepted caption truth or content-directed theory criticism Generated inputs/outputs with lineage, retained criticism and actual rationale consumers; direct outcome evidence at verification.
Benchmark evidence is reported or externally evaluated, not a frozen causal comparison. SRC-1; RT-RTE-osworld-driver, RT-RTE-rollout-selection, EPI-CLM-bbon-performance Evaluation call sites and offline selection; external task/evaluator definitions and actual rollout directories unavailable Score correctness, reported gain magnitude, human comparison, transfer or component-level causal attribution Frozen task/evaluator definitions, raw trials and candidate sets, allocation details and independently varied components.
Historical epistemic coverage and opaque alternatives remain partial. SRC-1; epistemic coverage, EPI-RTE-trajectory-reflection; profile verification Selected historical transformations and known local memory pathways, not every platform transformation or service-state mechanism Exhaustive epistemic negatives, complete memory-value statistics or a universal no-learning/theory-building verdict Accepted route-specific records for unresolved historical operations and evidence resolving included opaque state alternatives.