AI Agents in Depth synthesis
Type: agentic-system-analyses/types/agentic-system-synthesis.md
Bounded synthesis
Separate experiments, not one operating agent
AI Agents in Depth is a book and companion agentic workflow and teaching suite: separately launched model-dependent experiments with shared packaging and operating guidance. This account covers selected context-agent, user-memory and retry-policy workflows in https://github.com/bojieli/ai-agent-book at commit dbc046eb896ac4e39aa19c7774c8bf49583b89a6, with applicability cutoff 2026-10-12. The boundary is a complete artifact, partial loop. External reproduction repositories, model services, weights and agent hosts are not frozen evidence. SRC-1 supplies inspected implementation and attributed design; no target execution, retained run or causal experiment was adopted. Consequently, the account establishes wiring and its conditions, not successful reproduction or measured benefit. The Source register identifies that evidence.
The selected workflows expose three different progressions: a model proposes tools while Python controls task execution; conversational traces become retained facts supplied to later models; and failure evidence feeds source proposals that software evaluates without installing a successor. These are distinct state and authority paths, not stages of one shared runtime.
From a task to tool observations and a returned answer
A human launches the context CLI or calls its Python API. The local process assembles messages, calls a configured OpenAI-compatible model, dispatches proposed calls through four fixed tool names, and returns terminal text or an error/result dictionary. The model chooses proposed actions; Python owns progression and dispatch. The single-task driver can serialize results, whereas interactive reuse retains history across tasks until reset or agent replacement. No inspected reload consumes the single-task result file. See RT-RTE-context-task and RT-OBJ-conversation-history.
What the model sees depends on request construction, not everything stored locally. The legacy ablation suite starts a fresh agent per arm. NO_HISTORY withholds earlier history; NO_REASONING removes exposed reasoning text without disabling hidden provider reasoning; NO_TOOL_CALLS omits request-side tools without adding a separate response-side execution prohibition; NO_TOOL_RESULTS can execute tools but substitute their delivered results. These are experimental visibility controls, not security boundaries. See RT-RTE-context-ablation.
Tool guarantees remain local. Calculator AST checks constrain accepted expression syntax, not resource use or the whole agent. The adjacent interpreter executes Python in the host process with reduced builtins; PDF file/HTTP access also lies outside the calculator check. An iteration ceiling cannot interrupt a blocking host execution. The exception handler's undefined openai reference further prevents a universal structured-recovery guarantee. See RT-RTE-calculation-path, RT-RTE-code-execution and RT-RTE-context-task.
Content warrant is equally route-specific. PDF text retains URL/page association without author authentication or extraction verification. Currency results follow a fixed rate table, despite the tool's current-rate wording; a calculation timestamp does not date the rate premises. Nonempty terminal text means completion, not correctness. See EPI-RTE-pdf-import, EPI-RTE-currency-conversion, EPI-CLM-current-rates and EPI-OBJ-final-answer.
The legacy numerical assessor compares extracted large quantities against task and delivered tool text. Observation presence makes it abstain after the no-answer branch; without observations, a valid arithmetic result can be unmatched. Its consumer warns and labels reports, but neither vetoes the answer nor initiates repair. It licenses a numerical-match predicate, not derivation provenance or task truth. This is not every context evaluator: independent verification identifies an omitted alternate campaign with a fixed numerical reference predicate and canonical evidence promotion, without establishing its execution or rubric validity. See EPI-RTE-grounding-check, EPI-RTE-grounding-report and report-verification limits.
From conversation traces to later context
The user-memory workflow implements automatic trace-fed retained-context production: it records user/assistant dialogue, selects recent unprocessed turns, asks an analysis model to invoke add/update/delete tools, and reloads resulting files for later answering. This is a supported structural contribution even though extraction fidelity and improved future capacity are unobserved. Model/software admission is automatic; manually starting processing is not approval of each fact. Parsing and manager checks can reject mutations, but no separate factual evaluator approves them. Assistant mistakes can enter the same extraction input as user assertions. See MEM-RTE-trace-maintenance and MEM-RTE-conversation-reload.
Notes and hierarchical cards store model-interpreted content alongside symbolic addresses and metadata. JSON does not make factual values executable policy. Advanced cards can retain a supplied backstory and display it while omitting internal metadata; neither mandatory rationale retention nor successful entity disambiguation follows. Notes display selected fields, not their stored creation time. See MEM-OBJ-memory-note, MEM-OBJ-memory-cards, MEM-OBJ-categories and report-verification limits.
Each chat automatically supplies all rendered facts and identity-filtered current-session history, without item relevance targeting or an enforced memory-input token cap. The output token limit does not bound that input. Disk deletion changes a fresh context block but cannot retract consequences already present in live exchanges. Persistence also has weaker semantics than success labels suggest: save failures are swallowed, processed IDs are marked before successful analysis, and stale manager instances can overwrite intervening edits. See MEM-RTE-conversation-reload and MEM-RTE-trace-maintenance.
Explicit CLI editing and query provide a different, request-driven path. Deterministic consolidation merges normalized equal notes and selects newest versions within tag groups; this is not semantic contradiction adjudication and can discard distinct facts. Optional local/Dify history search remains an API capability without an inspected shipped model consumer. Semantic-search settings alone establish neither provider storage nor user isolation or deletion propagation. See MEM-RTE-offline-maintenance and MEM-RTE-history-search.
A separate sequential campaign replaces arbitrary parsed JSON memory after each fixture session. The next writer gets only latest memory plus the next session; the final answer gets memory and question; the judge gets original histories and rubric. Checkpoints preserve previous states and receipts, but select latest state for continued work. Grading changes reported pass/reward, not prior memory admission or rollback. Resume checks omit fixture, prompt and budget identity, so cached results do not guarantee fresh evidence. See MEM-RTE-sequential-campaign and MEM-BAP-campaign-state.
The profile verification supports partial memory classifications, all at wired strength. Automatic dialogue-to-context writes and retained retry-policy candidates qualify structurally as trace learning, meaning automatic trace-fed durable artifact production with a later consumer. That classification does not establish improved capacity, learned weights or live self-improvement. All ten comparison axes remain partial because included alternatives, initial provenance and opaque remote state are unresolved.
From failure evidence to a checked candidate and audit package
The representative multi-step workflow is Experiment 9-6's retry-policy campaign because one driver connects diagnosis, negative-control rejection, coding-model handoff, software verification and delivery. The objective is fewer redundant permanent-error calls while preserving temporary recovery and public interfaces. Python groups supplied repeated non-retryable failures, generates an intentionally over-broad control, evaluates it, and passes its failed-check summary alongside diagnosis and stable source to one coding request. This control is software-authored, not a naturally failed model predecessor. The model proposes replacement source, not release authority; rejecting its proposal does not trigger another coding round. See RT-RTE-policy-candidate and the workflow handoff.
Host checks reject oversized, uncompilable or deny-listed source before a Docker evaluator checks signatures and bounded retry/circuit expectations. The requested container limits depend on trusted Docker, kernel and image contracts. Candidate and evaluator share a process, so host containment does not establish tamper-proof verdicts. The manifest accepts a changed candidate with a nonempty all-true supplied check set; direct callers can substitute that set. The campaign adds a protected-file hash check that the demo lacks. See RT-RTE-candidate-validation and RT-RTE-policy-candidate.
Normal delivery is an audit package: candidate source, manifests, diagnosis, comparisons, metrics, receipts and hashes, followed by a terminal summary and gate-dependent exit. Both rejected and eligible candidates are written. Sandbox failures have host continuation paths, but provider, parsing and filesystem errors can interrupt delivery. Writes are not transactional; an out-of-root output directory can fail summary construction after artifacts exist. release_to_canary means shadow eligibility with stable unchanged, not live traffic, installation or exercised rollback. See the workflow account and RT-CLM-canary-decision.
Independent theory and improvement conclusions
The strongest supported change-plane contribution is localized, content-directed retry-policy criticism and candidate evaluation, with rejected-check feedback delivered to a later proposal request. At the retry-policy campaign boundary, RT-RTE-policy-candidate separately supports:
- Theory-builder condition 1 — wired: a whole formal policy module is localized and replaceable.
- Condition 2 — wired: evaluator decisions consume what its functions say through execution.
- Condition 3 — wired: deterministic diagnosis targets specific retry/circuit omissions and generation changes those regions.
- Condition 4 — afforded: rejected criticism reaches a subsequent model request, but content-dependent model revision and a recurrent successor loop are unestablished.
Criticism improving capacity for future action remains uninspected. Persistence, tests and prompt delivery are not evidence of that benefit. Reflection has a wired partial mechanism—represented failures can generate a candidate for the represented policy—but the return direction into live stable behavior is uninspected, so a complete reflective loop is not established. Computational diagnosis, proposal and veto are wired within the bounded experiment; autonomous fulfillment of all four theory-builder conditions is not established. Revision of the diagnostic/validation method against its own operation is also uninspected.
For self-improvement, the assessment horizon is the launched campaign and subsequent operational use of stable retry policy, under the independent objective above. Evidence shapes candidate determination, but an installed organizational change, a live later authority path and subsequent causally dependent operation are not established. Evaluation consumes candidate code only. Therefore neither exercised self-improvement nor successful improvement is attributed; this does not negate candidate generation, narrow admission or structural trace learning. These are independent findings, not a ladder or a suite-wide verdict.
For a bounded teaching scenario, the inspected code supports tracing context visibility, memory maintenance and proposal/check authority separation. It does not support a scenario requiring verified answers, reliable multi-writer memory, hostile-code isolation across tools or an operationally improving successor. The assessment would change with provenance-linked execution evidence showing actual deliveries and decisions; freshness-complete checkpoint identity and durable write/error handling; authenticated evaluator/deployment contracts; and, for improvement, an installed candidate traced into later behavior with an outcome comparison preserving temporary recovery. Component-level causal attribution would additionally require independently varying the component being credited.
Limitations
| Limitation | Affected SRC IDs and record citations | Inspected boundary | Conclusion prevented | Evidence that would resolve it |
|---|---|---|---|---|
| Partial suite coverage and external reproduction loops | SRC-1; boundary, MEM-RTE-trace-maintenance, MEM-RTE-sequential-campaign | Selected context, user-memory and retry-policy paths; other backends, retrieval variants, skills, training and collaboration unresolved | Suite-complete mechanism/value inventories, suite-wide negatives and end-to-end reproduction claims | Material-route inspection across included alternatives and separately frozen external dependencies |
| No adopted execution or causal outcome evidence | SRC-1; RT-RTE-context-task, MEM-RTE-sequential-campaign, RT-RTE-policy-candidate | Source and attributed design only; retained results unassessed | Successful operation, extraction quality, measured benefit and causal component effects | Provenance-linked traces and bounded intervention/comparison evidence |
| Omitted alternate context evaluator; carried report-verification limit | SRC-1; RT-RTE-context-ablation, EPI-RTE-grounding-check, EPI-RTE-grounding-report, report-verification limits | Legacy suite records; verifier inspected alternate chapter1/context/run_experiment_1_1.py reference predicate and evidence promotion |
Uniform absence of context reference answers; complete evaluator/admission inventory; alternate execution or rubric validity | Recorded alternate evaluator/promotion route, rubric assessment and provenance-linked runs |
| Note metadata visibility overstatement; carried report-verification limit | SRC-1; MEM-OBJ-memory-note, report-verification limits | Notes context renders ID, session, content, tags and update time, not creation time | Full-field or creation-time visibility comparisons at the model consumer | Corrected field-projection record supported by inspected renderer and delivered context |
| Model identity and internals opaque | SRC-1; RT-CMP-context-model, RT-CMP-coding-model, MEM-CMP-writer-client, MEM-CMP-judge-client | Configurable clients, routing and external endpoints; no immutable weights | Immutable answering-model identity, provider parameter adaptation and authenticated judge independence | Version-pinned provider artifacts and relevant internal/service evidence |
| Tool and report warrant stays local | SRC-1; EPI-RTE-pdf-import, EPI-RTE-currency-conversion, EPI-RTE-grounding-check | PDF association, fixed-rate arithmetic and numerical matching | Source truth, current financial accuracy, actual derivation provenance or task correctness from these checks | Source/fidelity checks, dated authoritative rate premises and task-specific derivation/reference evaluation |
| Persistence, withdrawal and checkpoint freshness are incomplete | SRC-1; MEM-RTE-trace-maintenance, MEM-RTE-conversation-reload, MEM-RTE-offline-maintenance, MEM-RTE-sequential-campaign | Separate managers, swallowed saves, premarked IDs, live history and incomplete resume signature | Durable admission from success labels, universal withdrawal, multi-writer consistency and fresh cached evaluation | Error-propagating synchronized persistence, consumer-aware withdrawal, complete input signatures and traced failure/recovery |
| Initial provenance and optional remote search unresolved | SRC-1; MEM-OBJ-memory-note, MEM-OBJ-memory-cards, MEM-OBJ-categories, MEM-RTE-history-search | Generic direct inputs/preexisting files and Dify request capability; no shipped model-search consumer established | Complete initial lineage/admission classification, remote storage classification, model read-back, user isolation or deletion propagation | Initial write provenance, identified consumer wiring and inspected service/index lifecycle contracts |
| Isolation and recovery guarantees do not generalize | SRC-1; RT-RTE-calculation-path, RT-RTE-code-execution, RT-RTE-context-task, RT-RTE-candidate-validation, workflow account | Calculator AST, host interpreter, requested Docker envelope and nontransactional campaign delivery | Whole-agent isolation/resource bounds, adversarial evaluator integrity, universal structured recovery or artifact presence proving delivery | Enforced effect/resource boundaries, verified image/oracle separation, repaired exception paths and explicit delivery/recovery evidence |
| Candidate admission does not close iteration or improvement | SRC-1; RT-RTE-policy-candidate, RT-BAP-candidate-policy, workflow handoff | Synthetic failures, one coding request, caller-supplied checks and evaluation-only candidate authority; stable unchanged | Reliable content-dependent iteration, complete reflection/autonomous theory building, improved future capacity or exercised/successful self-improvement | Content-linked revision evidence, installed successor with live authority and later causally dependent operation; outcome comparison under the stated objective |