AI Agents in Depth report verification

Type: agentic-system-analyses/types/agentic-system-verification.md

Verification

The reports support a bounded implementation account, with the local inventory and display limits below. The teaching-suite classification and partial-loop boundary remain supported; no evidence requires changing the frozen target. I read all supplied members, types and contracts, the empty correction-answer inputs and the structural report-check (none). There are no structural failures to route and no correction arguments to adjudicate. I independently inspected source rather than treating the analysts' validation statements as semantic verification. SRC-1 checkout HEAD matched the reviewed commit and its working tree was clean. No target code, tests, examples, providers or containers were executed. All added evidence is implementation at the boundary's Source register, not observed operation or causal support.

Runtime, tool content and evidence strength

Direct inspection of complete SRC-1 chapter1/context/agent.py, chapter1/context/calc_sandbox.py, and the test/report and single-task bodies of chapter1/context/main.py supports RT-RTE-context-task, RT-RTE-context-ablation, RT-RTE-calculation-path and RT-RTE-code-execution. The loop owns dispatch; the model proposes actions. Stored history and actual delivered context differ by mode. Terminal text is completion, not task correctness. The calculator's AST restriction does not cover adjacent host exec, file access or HTTP access. The undefined openai exception-handler reference is present and prevents a universal structured-recovery claim. The report does not turn this source defect into an observed failure.

RT-OBJ-conversation-history and RT-BAP-system-prompt correctly distinguish a mixed live container from interpretive instruction authority. The declared parts EPI-OBJ-pdf-result, EPI-OBJ-currency-result and EPI-OBJ-final-answer have distinct content and checks without invalidating that container. EPI-RTE-pdf-import and EPI-RTE-currency-conversion are genuine subroutes of the task loop. The source preserves PDF URL/page association without authentication; conversion uses constants and a calculation timestamp, not a market feed. The mismatch in EPI-CLM-current-rates is supported.

Complete direct reading of SRC-1 chapter1/context/grounding.py and its consumers in chapter1/context/main.py supports EPI-RTE-grounding-check, EPI-RTE-grounding-report and EPI-CLM-grounding-source. I traced quantity extraction, tolerance, branch ordering, sent-message selection, warnings and summary labels. Observation presence causes abstention after the no-answer branch. With no observations, the reported arithmetic counterexample follows from the implementation: a valid sum can be unmatched. Numerical availability is neither derivation provenance nor task truth. Reporting is advisory, not a veto or revision loop. RT-CLM-context-purpose remains attributed teaching intent, not a demonstrated context effect. The ledger's conditional arithmetic and indeterminate model/code-content assessments are appropriately scoped.

Complete SRC-1 agentbook/providers/resolution.py, context configuration imports, and the coding-model client support RT-CMP-context-model and RT-CMP-coding-model. Requested names can be mapped or rerouted; no immutable weights or provider-internal adaptation are established.

Memory producers, transformations and later consumers

Direct source inspection covered complete SRC-1 chapter3/user-memory/memory_manager.py, conversational_agent.py, conversation_history.py and memory_cli.py; analysis-client construction, prompts, context, dispatch and task-loop bodies in agent.py; processor construction, analysis, direct updates and recent-processing bodies in background_memory_processor.py; interactive and evaluation preparation/answer bodies in main.py; and prompts, grading, client construction and complete run_one in run_evaluation.py. I also inspected ChatRecorder.create in chapter3/experiment_utils.py to check the checkpoint receipt producer.

The storage and transformation findings in MEM-OBJ-memory-note, MEM-OBJ-memory-cards, MEM-OBJ-categories and MEM-RTE-offline-maintenance are supported except the small note-display overstatement retained below. Basic string tool updates wrap the string as a value; advanced updates replace cards, preserving creation metadata rather than merging all facts. Consolidation chooses by normalized text, first tag and update time, not semantic adjudication. Save errors are swallowed; atomic replacement does not establish multi-writer consistency. MEM-CLM-backstory-confusion is prompt doctrine, not measured effectiveness.

MEM-OBJ-conversation-turn, MEM-RTE-trace-maintenance, MEM-RTE-conversation-reload and MEM-BAP-memory-context correctly separate raw dialogue, model-directed mutation, persistence and later prompt delivery. Processed IDs are marked before analysis, analysis returns an empty list after tool mutation, and answering reloads disk while the analysis manager can remain stale. Withdrawal cannot erase prior live conversation consequences. The background processor explicitly constructs its analysis client with conversation-history recording disabled; therefore the generic analysis client's history-writing branch does not establish recursive insertion of analysis tasks into this ordinary processor's persisted history. No additional defect is inferred from that branch.

MEM-RTE-history-search is a capability, not a shipped model retrieval consumer in the inspected user-memory project. A project-wide call-site search found the history method, its Dify forwarding and tests, not a model-consumption call. The mutation dispatcher exposes add/update/delete only. Dify indexing omits a user identifier, and search does not transmit its user argument; dataset isolation and deletion remain external uncertainties, not proven exposure.

MEM-OBJ-memory-states, MEM-RTE-sequential-campaign and MEM-BAP-campaign-state preserve the difference between derived memory, raw receipts and symbolic resume controls. Only the latest state reaches the next writer and final answer; the judge gets original histories and rubric. Parseability is not fact acceptance. Failed grading affects reported pass/reward without rolling memory back. Signature equality omits fixture/prompt/budget identity, so resume is not a freshness guarantee. The recorder producer retains actual request/response objects or error receipts; that implementation inspection does not authenticate any retained run.

The four client records MEM-CMP-analysis-model, MEM-CMP-conversational-model, MEM-CMP-writer-client and MEM-CMP-judge-client identify distinct roles and configurations, not necessarily distinct underlying weights. Their unpinned identities and uninspected provider parameters remain appropriate.

Candidate admission, workflow delivery and reconciliation

Complete direct reads of SRC-1 chapter9/self-modifying-agent/evolution.py, candidate_sandbox.py, sandbox_runner.py, llm_generator.py, demo.py, run_experiment_9_6.py and stable/retry_policy.py support RT-OBJ-retry-policy, RT-RTE-policy-candidate, RT-RTE-candidate-validation, RT-BAP-candidate-policy and RT-CLM-canary-decision. I traced diagnosis threshold, exact-region patches, negative-control feedback, the coding request, prefilter, Docker request, evaluator functions, check consumption, manifest decision and final filesystem/terminal effects.

The workflow account and both annotations retain consequential ordering and authority: the negative control is evaluated before the coding handoff; only its summary reaches the coding model; host checks decide eligibility; candidate files can be written despite rejection. Aggregate campaign gates differ from candidate manifest gates. Out-of-root paths can fail terminal summary construction after artifact writes. Container requests depend on Docker/kernel/image contracts, and the candidate shares a process with its evaluator. Neither isolation flags nor check labels prove adversarial verdict integrity. Shadow eligibility is not an installed successor or exercised rollback.

The theory account's localized formal policy, content-dependent evaluator and exact-region deterministic criticism are supported. Its afforded model-mediated iteration is not promoted to reliable content-dependent revision. Learning, full reflection, autonomy and exercised self-improvement remain separately unestablished; evaluator execution is not live stable adoption. No correction is needed merely because the source names the campaign self-modification.

I compared every declaration, source anchor and evidence status across the reports, all seven epistemic Part of: relations, all annotations and the reconciliation. The relations denote actual container parts or subroutes, not replaceable candidates. No supersession is proposed or needed. Distinct memory schemas, clients, claims and authority horizons are not duplicates. Additional epistemic/workflow inspection does not contradict runtime's report-local inspection limits. Source-claim mismatches are not inter-report conflicts. No unresolved conflict requiring both findings and prevented conclusion was omitted. The inventory limitations below do not invalidate those identity dispositions.

Omission searches and stopping decision

The frozen top-level and scoped tracked trees supplied the search basis, rather than only report anchors. I selected challenges that could change the central findings: another shipped ablation evaluator with a reference answer; a consumer installing accepted retry policy; a dispatched history-search consumer; and intervening checkpoint serialization or processor history-writing that might invalidate the stated memory chain.

The alternate context campaign does have a numerical rubric and evidence promotion; its omission is a local limit below, not a reason to erase the supported legacy-suite assessment. Retry-policy project search plus complete demo/campaign tracing found evaluation and artifact consumers, not live installation in those drivers. User-memory call-site and construction checks support the bounded search-capability and producer findings. The recorder dependency confirms the checkpoint receipt chain without adding observed evidence. No broader absence follows from these searches.

Further optional expansion stopped because each selected challenge now has a supported disposition or a stated local limit. Other chapter mechanisms, external-host skills, retrieval backends and retained campaign artifacts remain available but unassessed, not absent or inaccessible. Their inspection could enable broader comparisons or observed-run findings, but is not needed for the reports' selected-route conclusions. Suite-complete epistemic, memory, reliability and improvement comparisons remain unsupported; no emitted system-wide negative requires that breadth.

Blockers

none

Limits

• RT-RTE-context-ablation, EPI-RTE-grounding-check and EPI-RTE-grounding-report describe the legacy AblationTestSuite, not every shipped context comparison. The alternate SRC-1 chapter1/context/run_experiment_1_1.py has a fixed numerical reference predicate, separate completion/correctness fields and canonical evidence promotion. Direct reads of its task/constants, summarizer and launch/delivery bodies establish that omitted mechanism, not its execution or rubric validity. The selected records remain valid, but readers must withhold a claim that context experiments uniformly lack a reference answer, or that the retained inventory covers all evaluator/admission paths.

  return bool(final_answer) and all(number in normalized for number in EXPECTED_NUMBERS) --- chapter1/context/run_experiment_1_1.py promoted = canonical_run and evidence["analysis"]["experiment_execution_accepted"] --- chapter1/context/run_experiment_1_1.py

• MEM-OBJ-memory-note says all fields enter formatted model context, but SRC-1 chapter3/user-memory/memory_manager.py renders note ID, session, content, tags and update time, not the stored creation time. This is a local metadata-projection imprecision: the retained-content, symbolic-addressing, lineage and interpretive-authority conclusions still hold, and no extracted comparison value here depends on creation-time delivery. Readers must withhold full-field or creation-time visibility comparisons for the notes consumer.

  context += f"Note {i} (ID: {note.note_id}, Session: {note.session_id}):\n" context += f" Content: {note.content}\n" if note.tags: context += f" Tags: {', '.join(note.tags)}\n" context += f" Updated: {note.updated_at}\n\n" --- chapter3/user-memory/memory_manager.py