AgentFly synthesis
Type: agentic-system-analyses/types/agentic-system-synthesis.md
Bounded synthesis
AgentFly implements a planner–executor runtime with two experience-reuse drivers and a separate numerical retriever trainer. The frozen repository calls the project Memento; this analysis retains AgentFly as the target identity. The evidence is SRC-1, the repository at commit 42fbbcac63dd58ed6856c0761357345a58e4f032, with an applicability cutoff of 2026-10-12. The whole-system boundary includes shipped clients, prompts, memory producers and consumers, training, MCP tools and deployment integration. External model, service and dependency implementations are excluded. Source code establishes wiring; README claims and committed data do not establish successful execution. No target execution trace or causal experiment was supplied or produced (boundary and evidence, Source register). This synthesis uses the accepted members without additional source reads.
From request to tool work and delivery
An operator supplies a question, optional file path, model names and server scripts. Python launches the selected MCP servers, discovers tools and rejects duplicate tool names. It owns progression: the planner proposes JSON tasks, Python executes them serially, and an executor model selects tool calls or returns task text. The planner receives no tool schemas; the executor does. This is delegation between model invocations controlled by one coroutine, not a distributed worker scheduler (RT-RTE-query-execution, workflow account).
Information narrows at each handoff. Servers receive call arguments rather than the whole conversation. Raw tool results enter the task-local executor conversation. Only the executor's final task text reaches shared planner history; the planner runs again after the entire plan, not after each task. These summaries can support replanning but are not independently verified evidence. Current-query conversation is process state rather than a durable experience bank (RT-OBJ-shared-history, RT-BAP-task-guidance).
The ordinary executor treats nonempty text as completion before checking tool calls. The local Qwen3 streaming alternative instead suppresses text when calls exist and reconstructs calls through name and argument adaptation. Its fallback search schema does not guarantee that a matching tool is registered (RT-RTE-local-execution). Three planner cycles bound only outer replanning. The executor loop is unbounded, and empty replies or nonresponsive dependencies can prevent progress. Executor trimming is approximate and omits tool-schema and call-argument costs; planner history has no equivalent trimming. Normal return can be an answer, parsing error or exhausted plan, all printed under a final-answer label. Provider retries and cleanup are not semantic verification or rollback (RT-RTE-query-execution, workflow account).
The effect boundary depends on selected server scripts and the deployment. The exposed terminal tool uses Bash through a subprocess interpreter with confirmation disabled by default, inherits the host environment, and uses a caller-selected workspace. Neither MCP transport nor a working directory establishes isolation. String-encoded execution errors can receive success wording. Workspace files can later supply executor context or executable input; text reads may truncate content, while binary reads return metadata (RT-RTE-terminal-execution, MEM-OBJ-workspace-files, MEM-RTE-workspace-read).
What counts as evidence and acceptance
Open interactive delivery has no reference-answer gate. Search supplies imported snippets with links but unknown source warrant. Query-aware crawling returns model-selected text rather than the whole page; grounding instructions are not an independent quotation check. Media tools return candidate interpretations without an intervening fidelity check. Arithmetic offers bounded Python derivation for add/subtract and awaited rounding, not a universal numeric oracle: multiply/divide call the later async round without awaiting it (EPI-RTE-search-import, RT-RTE-crawl-extraction, EPI-RTE-media-answer, EPI-RTE-math-result). These distinct routes cannot be assigned one epistemic grade.
The benchmark drivers introduce a narrower acceptance decision. A model judge compares the final prediction with dataset reference answers for semantic agreement. Python consumes its normalized correct token for benchmark scoring and labels. The criterion is answer/reference agreement for that question, not source truth, plan validity or the marginal usefulness of a retrieved case. Judge failure becomes incorrect, not abstention (EPI-RTE-answer-check, EPI-RTE-outcome-label).
From completed work to later guidance
The strongest supported improvement-directed contribution is a wired trace-fed memory loop: completed benchmark work automatically produces durable labelled examples that later queries can receive. Both successes and failures are admitted. The nonparametric bank stores question/latest plan/reward; the parametric bank stores case/latest plan/positive-or-negative label. Neither is necessarily the full procedure that produced the outcome (RT-RTE-case-admission, MEM-OBJ-dummy-memo, MEM-OBJ-memory).
At the next question, the nonparametric selector ranks question embeddings by similarity. The parametric selector ranks query/case-plan pairs by classifier probability. Labels partition examples only after ranking. Unknown parametric labels can be scored and written as training pairs while receiving no example section. Selected examples enter user-role history, with instructions to imitate positives and avoid negative patterns. Positive plan formatting can omit fields outside IDs and descriptions; judge rationale and tool evidence accompany neither sign. Count caps are not a dedicated delivered-token guarantee (MEM-RTE-embedding-examples, MEM-RTE-classifier-examples, EPI-RTE-case-format, RT-BAP-case-guidance). Thus retention and delivery support future contextual guidance, not demonstrated dependence, accepted plan knowledge or successful learning.
Rich result files retain more traces and judge reasons, but the inspected restart reader consumes question identity to skip completed work. It does not send those reasons back to planning. Results precede later training/case writes; a partial failure can therefore suppress a retry even though memory admission failed. Physical deletion of the parametric bank does not necessarily withdraw an already loaded pool. Exceptional parametric continuation can also carry prior history when failure precedes normal-return clearing; ordinary reset semantics cannot be generalized to that path (MEM-OBJ-query-record, MEM-RTE-classifier-examples, memory annotations).
From outcome pairs to numerical ranking
A second wired contribution is outcome-fed retriever training. The parametric driver copies the current answer's correctness label to every retrieved case. A separately launched trainer tokenizes those pairs, updates classifier parameters, saves final weights and optionally selects best weights by validation AUC, falling back to F1. Later initialization can load the configured checkpoint into ranking. This changes the retriever, not planner/executor LLM parameters; the no-LLM-weight-update claim must not be read as no numerical model update anywhere (MEM-OBJ-training-data, MEM-OBJ-checkpoint, RT-RTE-retriever-training, EPI-CLM-weight-updates).
The unresolved question is whether either contribution improves future task capacity. Validation measures pair-label prediction, not independently identified case utility or agent improvement. Pair-row splitting need not keep all pairs for one question together. Training defaults omit plans unless enabled; the README recipe explicitly enables them. The driver reloads cases, not weights, and does not schedule training or deployment. Automatic collection plus an operator-started trainer therefore falls short of establishing the claimed fully automated retraining cycle (memory annotations, RT-RTE-retriever-training, EPI-CLM-experience-learning, EPI-CLM-automated-retraining).
Independent property assessments and scenarios
Automatic trace-fed case guidance and numerical training are supported at wired strength, as distinct from improved capacity for future action. For the inspected case/training pathways, theory-builder condition 1 (a formulated theory) is not established by latest plans or weights; condition 2 (decisions depending on what that theory says) is not established by delivery; condition 3 (content-directed criticism with blame and changed reliance) is not established by outcome labels; and condition 4 (retained criticism shaping another round) is not established by retaining input/outcome records. These are independent unresolved findings, not system-wide absences. Hidden model processing supplies no inspectable substitute. Improvement caused by criticism of a consumed theory remains unestablished (RT-RTE-case-admission, RT-RTE-retriever-training, epistemic bounded conclusion).
Model-proposed plans and computational dispatch are implemented, but a theory-builder autonomy conclusion is not established. A reflective-system conclusion is likewise not established merely by storing records of prior work: the accepted account does not establish the required self-representation and both causal directions. Neither uncertainty defeats the supported memory/update mechanisms. At the benchmark-query and later-process-start horizons, case guidance and trained ranking support a dispositional, improvement-directed self-change pathway toward future task performance. Exercised self-improvement is unestablished: there is no trace showing subsequent operation causally depended on the changed cases or parameters. Successful improvement requires additional outcome evidence (RT-BAP-case-guidance, RT-RTE-retriever-training, epistemic bounded conclusion).
For open tool work, the implemented strength is flexible serial planning with returned summaries; correctness, termination and containment remain separate questions. For repeated reference-labelled benchmark questions, the strength is immediate case reuse and data collection; the label proxy and omitted reasons limit causal or explanatory interpretation. For numerical adaptation, the strength is a separately runnable update/load path; checkpoint availability, configuration alignment and external scheduling determine whether it reaches later ranking. Document/media side files add filesystem persistence and requested reuse, not established automatic case-bank ingestion; their inventory is partial (MEM-OBJ-tool-files, report-verification limits).
Assessment would change with linked traces of case production, selection and later decisions; controlled comparisons independently varying memory or retrieval policy; authenticated checkpoints and training/deployment provenance; or an implemented criticism route that retains challenged claims, diagnoses and changed reliance. A scheduled trainer/deployment consumer would strengthen the automation account, but would not alone prove benefit. The independent checks accept the bounded reports and profile without blockers; the report check retains the side-file inventory limit below (report verification, profile verification).
Limitations
| limitation | affected SRC IDs and record citations | inspected boundary | conclusion prevented | evidence that would resolve it |
|---|---|---|---|---|
| No supplied execution or causal comparison | SRC-1; RT-RTE-query-execution, RT-RTE-case-admission, RT-RTE-retriever-training | Frozen shipped clients and training; code and attributed claims only | Successful end-to-end operation, exercised memory dependence, exercised self-improvement, learning gains or reproduced benchmark results | Linked run artifacts and interventions at the claimed horizon; independently vary the component for component attribution |
| Outcome labels do not identify plan validity or case contribution | SRC-1; EPI-RTE-answer-check, EPI-RTE-outcome-label, MEM-OBJ-training-data | Judge sees question/reference/prediction; pair writer copies one outcome to all cases | Treating positive cases as validated rules or pair metrics as causal retrieval value | Plan/evidence checks and case-level intervention comparisons; reference and judge reliability evidence |
| Initial genealogy and exact numerical state unresolved | SRC-1; MEM-OBJ-dummy-memo, MEM-OBJ-memory, MEM-OBJ-training-data, MEM-OBJ-checkpoint, RT-CMP-case-retriever | Sampled committed corpora, loader and trainer; no tracked trained checkpoint | Corpus-wide quality, verified artifact generation, exact deployed ranking policy or its provenance | Corpus-wide provenance/quality audit and immutable assets/checkpoints tied to generation and deployment records |
| External dependencies and live grants excluded | SRC-1; RT-CMP-openai-backend, RT-CMP-tool-judge-models, RT-RTE-terminal-execution | Integration code, not provider internals or deployed OS/service policy | Exact provider state, deployed reliability, current grant inventory or isolation guarantees | Versioned deployment/configuration evidence, provider contracts and runtime permission/containment inspection |
| Acquisition alternatives and opaque payloads incompletely assessed | SRC-1; EPI-RTE-media-answer, RT-RTE-crawl-extraction, MEM-OBJ-tool-files, MEM-OBJ-workspace-files; epistemic coverage, profile verification | Selected extraction/return paths; other converters/page/spreadsheet transformations and binary contents remain unresolved | Complete source-fidelity, payload-form/authority or all-route knowledge-production comparisons; absence of hidden learning | Inspect consequential transformations, actual payloads and later consumers, with fidelity/lineage evidence |
| Side-file locations and inventory are too coarse | SRC-1; MEM-OBJ-tool-files, report-verification limits | Verified document/spreadsheet paths: DOCX/Chunkr basename outputs use working directory; PPTX images use source stem; unzip uses cache; XLS conversion writes a sibling XLSX omitted from inventory | Complete artifact-location, withdrawal or cleanup comparisons; not local persistence or requested reuse | Corrected path-by-path artifact inventory and cleanup/withdrawal consumers; a new analysis for substantive retained-record correction |
| Missing scheduling and deployment evidence | SRC-1; RT-RTE-retriever-training, EPI-CLM-automated-retraining; boundary | Standalone trainer and initialization loader; documented retraining script absent | Fully automated retraining/deployment or live checkpoint refresh through case reload | Shipped scheduling/deployment route, installed checkpoint identity and linked later initialization evidence |
| No inspectable formulated-theory criticism chain established | SRC-1; RT-RTE-case-admission, RT-RTE-retriever-training, epistemic bounded conclusion | Latest-plan/outcome cases and numerical update routes; model internals uninspected | Theory-builder conditions, criticism-mediated improved capacity, reflective or autonomous theory-builder characterization from these mechanisms alone | Formulated content, decision-dependence evidence, criticism with blame/reliance changes and retained next-round effects; separately evidence reflection/autonomy |