OpenSage report verification

Type: agentic-system-analyses/types/agentic-system-verification.md

Verification

The current reports support a bounded implementation account. The three previous epistemic blockers are repaired: consultation and benchmark result parts are now separately declared and connected to their checks and consumers, and reward export is no longer classified as implemented adaptation. The supplied epistemic correction answers agree with these changes but were not treated as verification. The structural report-check says none; there is no structural failure to route. The two carried inventory limits remain below.

Independent evidence basis

I rechecked SRC-1 HEAD against boundary, found a clean source worktree, and inventoried the top-level tracked tree. Independent source inspection traced the manager's spawn, load, runner construction, dispatch, directed invocation, continuation and startup reset; dynamic agent construction/rebuild; CLI submission/resume and web ordinary/live invocation; model registry, budget wrappers and runner plugin; terminal execution and local/native-Docker configuration; memory initialization, trajectory persistence, compaction production/folding and output reduction; hook/image retention; consultation and findings operations; local, SWE exploration and vulnerability direct drivers; Cybench scoring; vulnerability comparison/reward; PoC submission interpretation; and DevOps policy, grading and base collection. Optional build verification, repeated-call suppression and web event replacement were inspected for consequential omissions.

These reads supply implementation and doctrine/design within SRC-1 at the existing revision. No target code, tests, examples, provider calls or services were executed. No observed operation or causal benefit is inferred. I assessed the declared records and annotations against their source anchors, conditions, ordering, delivered content and consumers. Code remains responsible for citation resolution, quote occurrence and deterministic relations. No report declares an evidenced-absence record; bounded search misses are not accepted as system-wide absences.

Runtime dispositions

  • RT-CMP-base-llm, RT-OBJ-agent-definition, RT-OBJ-agent-instruction, RT-RTE-create-subagent and RT-BAP-dynamic-instruction: supported. SRC-1, src/opensage/llm/registry.py and src/opensage/toolbox/general/orchestration_tools.py, distinguish shared endpoint objects from persisted selectors and model-interpreted instructions. Baseline tools are unconditional, skill clamping is conditional, registration precedes saving, and rebuild resolves against current models/tools. Structural admission neither evaluates competence nor establishes useful self-change. Model identifiers do not pin weights.
  • RT-RTE-user-dispatch, RT-RTE-directed-child and RT-RTE-live-runner: supported with their path bounds. SRC-1, src/opensage/orchestration/manager.py, src/opensage/cli/opensage_cli.py, src/opensage/cli/opensage_web_app.py and src/opensage/orchestration/plugins/inbox_delivery.py, show the atomic directed sleeping check, optional history copying, extracted final-text return, asynchronous reply and best-effort truncation/file fallback. Startup truncates inboxes. Live execution calls the existing runner without the directed busy rejection or manager continuation loop. None of these findings licenses durable complete delivery, independent resources or correct task completion.
  • RT-RTE-terminal-extension, RT-RTE-host-extension and RT-BAP-tool-guidance: supported. SRC-1, src/opensage/toolbox/general/bash_tools_interface.py, src/opensage/sandbox/local_sandbox.py, src/opensage/sandbox/native_docker_sandbox.py, src/opensage/plugins/adk_plugin_loader.py, src/opensage/agents/opensage_agent.py and src/opensage/plugins/default/adk_plugins/read_before_edit_plugin.py, establish forwarded shell commands, configurable backend authority, host imports and interpreted skill/edit guidance. Foreground wait timeout does not terminate the background command. Named-tool warnings and hook compatibility do not establish universal mediation or containment.
  • RT-CLM-soft-budget: supported as a soft, wiring-dependent guard. SRC-1, src/opensage/llm/budget.py and src/opensage/plugins/default/adk_plugins/runtime_budget_plugin.py, separate availability checking from usage-dependent accounting. Concurrent calls can overshoot; missing usage prevents that callback's accounting. The live wrapper checks connection entry without establishing complete live-response metering.
  • RT-RTE-history-compaction, RT-BAP-memory-policy and RT-CLM-runtime-creation: supported as implemented controls and attributed requests, not demonstrated compliance or benefit. SRC-1, src/opensage/features/summarization.py, src/opensage/patches/agent_run_async.py and src/opensage/templates/auto_insert_prompts/default.md, distinguish code selection, model drafting, non-empty summary admission and advisory wiki maintenance. The broad creation claim does not substitute for evidence of useful decomposition or learned improvement.
  • RT-RTE-direct-driver and RT-RTE-evaluation-rollout: supported as selected route families. SRC-1, agent_library/local_support/local_cli.py and benchmarks/secodeplt/vul_detection_memory.py, supply direct runners with their own stopping/plugin policies. SRC-1, src/opensage/evaluation/base.py, src/opensage/evaluation/dispatchers/native.py and src/opensage/evaluation/rl_adapters/adapters/areal.py, supply manager evaluation, process dispatch and model-input/reward export. External parameter updates and production checkpoint admission remain uninspected, rather than being attributed to the adapter. The examples do not claim exhaustive shipped-driver coverage.

Memory dispositions

MEM-CMP-memory-service correctly distinguishes a dependency-owned service handle from an established memory writer or retrieval consumer. SRC-1, src/opensage/session/opensage_session.py, constructs the service and assigns Neo4j a code-property-graph role; manager runner construction passes the handle. Neither fact establishes embeddings or conversation-memory retrieval.

MEM-OBJ-traj, MEM-OBJ-checkpoint-selectors and MEM-RTE-checkpoint-resume are supported. SRC-1, src/opensage/patches/agent_run_async.py, src/opensage/memory/file_based/short_term/session_files.py, src/opensage/orchestration/persistence.py, src/opensage/cli/opensage_cli.py and src/opensage/orchestration/manager.py, distinguish best-effort event serialization, root unmatched-call sanitization, resource selectors, live-session precedence and current-registry reconstruction. Disk metadata does not back up sandbox contents or faithfully replay external effects.

MEM-OBJ-compacted-content, MEM-OBJ-pinned, MEM-CLM-pinning-survival and MEM-BAP-summary-authority are supported. SRC-1, src/opensage/features/summarization.py and src/opensage/patches/adk_llm_compaction.py, confirm independent stripped pin reads, model-role replacement, older-range subsumption and the failed-event-construction branch that still proceeds to skip originals. Expanded ranges combined with the instruction not to re-summarize background support a non-cumulative-summary risk, not observed loss. The final helper omits pin reads, independently selects the last call/response and returns a summary; no later-consumer wiring follows merely from that return. Raw retained events and the folded model view remain separate.

MEM-OBJ-todo, MEM-OBJ-knowledge-files and MEM-RTE-wiki-maintenance are supported as seeds plus afforded agent-directed maintenance and read. SRC-1, src/opensage/memory/file_based/long_term/knowledge_store.py, src/opensage/memory/file_based/short_term/prompt.py and src/opensage/templates/auto_insert_prompts/default.md, disclose paths and request index-then-entry reading; they do not automatically push knowledge contents or semantically approve writes.

MEM-OBJ-tool-output, MEM-OBJ-tool-summary and MEM-RTE-output-reduction are supported. SRC-1, src/opensage/features/summarization.py, saves the stringified response before reduction, skips non-dictionaries and short results, uses a 200-character prefix above 50000 characters, and otherwise drafts a summary with a 1000-character generation-error fallback. Missing canonical model can leave the saved response unreduced. Replacement removes original keys; the file pointer requires an actual later read. No semantic-fidelity or hard provider-token guarantee is established.

MEM-OBJ-findings, MEM-RTE-finding-admission and MEM-RTE-history-consultation are supported with their registration uncertainty. SRC-1, src/opensage/toolbox/general/agent_tools.py, establishes first-entry/no-model bypass, duplicate substring suppression, separate read/append lock scopes, unchecked append exit status and advisory transcript consultation. The positive critique grant in SRC-1, agent_library/agents/poc_agent_static_tools/agent.py, supplements the memory analyst's scoped search uncertainty; it does not establish audit, validation or finding grants. Retained contradiction analysis is not automatic retraction or demonstrated theory revision.

MEM-OBJ-hook-context, MEM-RTE-hook-retention, MEM-OBJ-image-parts and MEM-RTE-image-retention are supported. SRC-1, src/opensage/plugins/claude_code_hook_loader.py and src/opensage/plugins/default/adk_plugins/image_injection_plugin.py, distinguish transient injection, retained user events, result mutation and request eligibility. Image filtering mutates content parts; the record properly leaves the external request-copy contract unresolved rather than guaranteeing persistent-storage immutability. Binary payloads are not readable visual summaries.

MEM-OBJ-vulnerability-findings, MEM-RTE-findings-resume and MEM-RTE-explore-handoff are supported. SRC-1, benchmarks/secodeplt/vul_detection_memory.py and benchmarks/swe_bench_pro/swe_bench_pro.py, actually select structured retained findings or final/quota-summary text for a next model. Schema parsing, file presence and non-empty summaries are not truth acceptance. Explore prompt delivery does not establish the docstring's memory-search claim. Memory annotations on compaction, children, definitions/instructions and evaluation preserve those supplied referents and distinguish raw exports from learned parameters.

Epistemic corrections and dispositions

The consultation correction is supported. EPI-OBJ-review-text remains a valid container with EPI-OBJ-critique-idea, EPI-OBJ-assumption-audit, EPI-OBJ-validation-reasoning and EPI-OBJ-validation-decision declared as parts. The inventory and ledger now connect advice, reasoning and parsed fields to their own producers and consumers. Independent reading of SRC-1, src/opensage/toolbox/general/agent_tools.py, confirms these return identities and the independent validity/confidence/reasoning parser. The inconsistent-field example follows from its branches without asserting an observed reviewer output. The Boolean and confidence are co-consumed advisory fields, not an enforced pin gate.

EPI-RTE-critique-feedback, EPI-RTE-assumption-audit and EPI-RTE-claim-validation retain supported targets, transcript-selection limits and advisory force. Their implemented internal progression is not evidence of ordinary activation where factory grants remain unresolved. EPI-CLM-evidence-tools correctly distinguishes evidence-oriented contracts from universal source truth and bounds the finding contract's temporal/unconditional wording.

The benchmark correction is also supported. EPI-OBJ-benchmark-results now contains separate EPI-OBJ-flag-correct, EPI-OBJ-judge-pass, EPI-OBJ-judge-reasons, EPI-OBJ-solved-score, EPI-OBJ-vulnerability-match and EPI-OBJ-match-reason. SRC-1, benchmarks/cybench/helpers.py and benchmarks/secodeplt/vul_detection_memory.py, establish the distinct fixed comparison, model decision, explanation, conjunction, aggregate count and report consumers. The dependent inventory, ledger, route prose and Bounded conclusion retain this separation; none gives a model judgment the exact comparison's license.

EPI-RTE-cybench-check and EPI-RTE-cybench-disposition are supported. Independent tracing of flag selection, trace/submission loading, judge excerpt construction, error fallback and score writing confirms the reported bounded solved criterion. The ledger separates equality over supplied strings from the model's narrow blacklist assessment and programmed disposition. Reference provenance, trace fidelity, legitimate causal production and transfer remain unestablished. EPI-RTE-vulnerability-comparison correctly separates model type/class comparison, Boolean accuracy counting and explanation export; a schema or count does not independently validate vulnerability truth.

The third correction preserves the positive reward mechanism without claiming implemented adaptation. EPI-OBJ-reward-score and EPI-RTE-vulnerability-reward are supported by SRC-1, benchmarks/secodeplt/vul_detection_memory.py: status and parsed/substr finding count determine a reward dictionary. Its ledger now classifies check/evidence production of a non-truth-apt optimization signal. The route and conclusion continue to place actual parameter updates outside inspected implementation. The completed-response substring counterexample follows from the source and does not claim an exercised exploit of the reward heuristic.

EPI-OBJ-findings, EPI-OBJ-tool-summary, EPI-RTE-finding-check, EPI-RTE-finding-write, EPI-RTE-finding-delivery and EPI-RTE-tool-reduction have supported mechanisms and semantic qualifications. SRC-1, src/opensage/toolbox/general/agent_tools.py and src/opensage/features/summarization.py, separate comparison, suppression, retention, peer delivery and lossy transformation. Retention before demonstrated acceptance is not lifecycle integration of warranted knowledge. Model-generated summaries/judgments remain indeterminate without an output and checked semantic relation; deterministic prefixes are selection rather than new evidence.

EPI-RTE-search-request properly stops at request configuration in SRC-1, src/opensage/toolbox/general/web_search_tool.py, rather than individuating provider-returned propositions or preserved warrant. EPI-CLM-knowledge-policy remains attributed doctrine, consistent with the memory pin qualifications. EPI-RTE-poc-interpretation and EPI-CLM-poc-success correctly trace the success assertion before the null guard in SRC-1, benchmarks/cybergym/tools.py. The None != 0 deduction supports a faulty success cue under those inputs, not an observed exploit or a verdict on every PoC evaluator.

The Bounded conclusion preserves independently unestablished theory-builder conditions, learning and exercised self-improvement while retaining architectural consultation, criticism and organizational-change opportunities. It does not turn missing runs into architectural absence or deny additional acceptance/revision routes in unassessed benchmarks and skills.

Workflow and reconciliation dispositions

WF-RTE-devops-grading, WF-BAP-devops-workflow, the Workflow account and both annotations are supported. SRC-1, agent_library/agents/devops_gym_agent/agent.py, agent_library/agents/devops_gym_agent/config.toml, src/opensage/toolbox/finish_task/finish_task.py, benchmarks/devopsgym/devops_gym.py and src/opensage/evaluation/base.py, connect optional delegation, requested verification, Boolean completion, default idleness/timeout stopping, grading before collection and cleanup after collection. The parser accepts zero exit without a FAILED token and need not see a PASSED line. Failed/unknown grading does not veto export or automatically reopen work. External script criteria, successful issue resolution and complete artifact delivery remain unobserved.

All Part of: relations have supported containment: consultation/result parts belong to their declared objects; checkpoint resume belongs to user dispatch, hook retention to host-extension callbacks, finding resume to the vulnerability direct-driver family, and benchmark-specific checks/dispositions/reward/grading to evaluation. None silently replaces a parent's identity or borrows its evidence strength.

Reconciliation correctly retains three identity supersessions: the findings object, reduced-response object and tool-reduction route have the same source-native referents across reports. Its other container/function distinctions do not require supersession. The prior two missing-part conflicts are repaired in the current epistemic declarations, and the reconciler properly retains the valid containers. Its dispositions preserve physical-storage/read-view separation, model-role summaries, pin failure limits, split finding locks, uncertain image-copy semantics and graph-export qualification. Scoped uncertainty about critique registration coexists with the epistemic positive factory witness; generic uninspected benchmark oracles coexist with specialist concrete interfaces. No unresolved cross-report contradiction requiring correction is established.

Omission questions and stopping decision

The selected questions were whether alternate shipped callers invalidate manager guarantees; whether memory producers actually reach later consumers rather than only storing data; whether missing verification/recovery plugins change completion or acceptance conclusions; and whether current result parts preserve check-specific licenses through reporting. The search basis was the frozen tree, exact-name caller searches across src, agent_library and benchmarks, factory/configuration grants and producer/consumer dependencies.

Direct local/vulnerability runners, SWE exploration and web live invocation confirm that manager guarantees cannot be generalized, as the current reports already state. Retained findings and explore text have concrete next-model consumers; the final-compaction helper remains a capability without a found shipped caller. Factory search and source reads confirm exposed critique without establishing audit/validation/finding grants. Optional build verification and repetition suppression do not make the selected DevOps finish flag a verified certificate. Web replacement is an additional live-history edit surface, not truth acceptance; its incomplete write/provenance inventory remains a limit. Corrected symbolic decisions and explanation parts now connect to their actual consumers without pooled warrant.

Further optional omission expansion stopped because these consequential questions have supported dispositions and no identified material challenge remains without one. Unread backend, benchmark and extension bodies do not imply absence or equal guarantee coverage. The reports already withhold exhaustive inventories and whole-system negatives; further enumeration is not needed to preserve their selected positive mechanisms. The two inventory limits below specify comparisons still unavailable rather than weakening supported findings.

Blockers

none

Limits

  • The checking/recovery inventory is partial beyond RT-RTE-host-extension, EPI-CLM-evidence-tools and WF-BAP-devops-workflow. SRC-1, src/opensage/plugins/default/adk_plugins/build_verifier_plugin.py, implements optional first-attempt build checking or a verification prompt, resets task_finished on selected failures, permits later attempts, and fails open on selected errors. SRC-1, agent_library/agents/swebenchpro_agent/config.toml, selects it; the inspected DevOps configuration does not. SRC-1, src/opensage/plugins/default/adk_plugins/doom_loop_detector_plugin.py, tracks identical tool-name/argument hashes and returns a replacement response at its repetition threshold; it does not check that preceding calls failed. These mechanisms prevent complete comparison of shipped verification/recovery/admission plugins, not the selected DevOps grading sequence or bounded consultation/acceptance conclusions. Naming them as unassessed in the epistemic inventory does not supply their full analyst-record account.

    if attempts >= 2: logger.info("[BuildVerifier] Agent confirmed (attempt #2+), allowing") return None --- src/opensage/plugins/default/adk_plugins/build_verifier_plugin.py if consecutive >= self.threshold - 1: --- src/opensage/plugins/default/adk_plugins/doom_loop_detector_plugin.py

  • MEM-OBJ-traj and MEM-RTE-checkpoint-resume do not enumerate the shipped web edit admission path. SRC-1, src/opensage/cli/opensage_web_app.py and src/opensage/features/opensage_in_memory_session_service.py, allow a web caller to replace selected text/call/response content in a live session by event ID. This is another content-edit surface, not a semantic acceptance gate or necessarily a human-approved write. The memory account already mentions event replacement and does not claim a complete caller inventory. Readers should withhold complete write-control/provenance and disk-durability comparisons for this surface; checkpoint restoration and the independently traced memory consumers remain supported.

    storage_session.events[idx] = event storage_session.last_update_time = time.time() return event --- src/opensage/features/opensage_in_memory_session_service.py