nao report verification

Type: agentic-system-analyses/types/agentic-system-verification.md

Verification

The supplied reports support a bounded implementation account. The three previous blockers are repaired, and no new material blocker was established. The supplied structural report-check says none. Runtime's two correction answers and epistemic's one answer are supported by the revised passages and independent source inspection; memory and workflow supply no correction arguments.

I read all supplied reports, declarations, annotations, correction answers and reconciliation, and rechecked the checkout HEAD against the frozen boundary. The top-level tracked tree agrees with the boundary's coverage table. I assessed every record's meaning, source-anchor support, evidence strength, containment and connections; quotation occurrence and deterministic cross-member form remain the content check's responsibility. All source checks below concern SRC-1 implementation or doctrine/design. No target code, tests, examples, model calls or services were executed. Wiring does not establish activation, successful operation or benefit.

Correction dispositions

Applied status: both corrected interpretations are supported. Independent reads of SRC-1 apps/backend/src/services/context-recommendations.service.ts and apps/backend/src/services/context-pr.service.ts confirm two assignments: reconciliation resolution writes applied without opening a PR, and automatic PR creation separately writes applied after its successful call. RT-RTE-context-recommendation, the runtime alternate-callers discussion, the epistemic recommendation-retention row in the Authority-route ledger, EPI-RTE-finding-reconciliation and the bounded conclusion now preserve both producers. None requires inferring a PR or deployment from status alone. The previous runtime and epistemic blockers are dismissed. Memory's older objection in Integration issues is not the current runtime finding; reconciliation explicitly disposes that stale cross-report characterization.

Compaction exposure: the corrected distinction is supported. SRC-1 apps/backend/src/services/compaction.ts slices and replaces the eligible prefix. apps/backend/src/agents/compaction/compaction-llm.ts then strips images and selects summarizer input under its own budget, reserving output tokens and prompt overhead. apps/backend/src/utils/ai.ts walks backward and stops at a nonfitting message or assistant/tool pair. The runtime Working-context control account, RT-CMP-compaction-model and RT-RTE-web-chat now distinguish replaced material from delivered summarizer input, including possible loss before summarization. The previous runtime blocker is dismissed. Agreement with MEM-RTE-chat-compaction does not establish summary faithfulness or successful context fit.

Record and connection dispositions

  • Model roles and fixity. RT-CMP-agent-model, RT-CMP-compaction-model, RT-CMP-verification-model, MEM-CMP-memory-extractor and EPI-CMP-memory-extractor distinguish call roles from exact weight identity. SRC-1 apps/backend/src/utils/llm.ts supplies configurable model IDs and endpoints; the agent, compaction, test and memory service paths supply the separate calls, not provider-side training or weight digests. The extractor supersession in reconciliation is justified by the same resolved MemoryExtractorLLM role and selection call. It preserves the declarations and does not strengthen evidence about provider fixity.
  • Runtime ownership and guidance. RT-OBJ-tool-context, RT-OBJ-system-prompt, RT-RTE-web-chat and RT-BAP-project-guidance support the separation of executable grants, model-interpreted instructions, model-selected tools and code-owned persistence. SRC-1 apps/backend/src/routes/agent.ts and apps/backend/src/handlers/agent.ts locate web role, budget and owner checks; these are not universal caller policies. Direct reads of apps/backend/src/services/agent.ts confirm prompt construction and override ordering, the preparation chain, compaction callbacks, streaming message upsert and extraction scheduling after dispatch rather than after answer verification. Standard guidance can be bypassed by an override, while restrictions are appended separately. The route's implementation/operation separation is supported.
  • Alternate invocations and effects. RT-RTE-explore-task, RT-RTE-mcp-chat and RT-RTE-automation-run are different caller/return contracts, not one delegation mechanism. SRC-1 apps/backend/src/agents/subagents/run.ts constructs a fresh system-plus-task message array while sharing ToolContext; its report is the return rather than the child transcript. apps/backend/src/mcp/tools/sub-agent.ts starts a retained background chat and races completion against a response budget. apps/backend/src/handlers/automation.handler.ts uses the automation owner, persists streamed progress and checks expected integration-call presence. These do not establish an answer oracle, distributed locking, exactly-once effects or cross-channel authorization. apps/backend/src/agents/tools/index.ts preserves extra tools while filtering built-ins. SQL checks in apps/backend/src/agents/tools/execute-sql.ts, Monty setup in apps/backend/src/agents/tools/execute-python.ts and Boxlite network/jailer settings in apps/backend/src/agents/tools/execute-sandboxed-code.ts support scoped adapter controls, not a universal safety invariant.
  • User-memory content and admission. MEM-OBJ-memories, EPI-OBJ-personal-fact, EPI-OBJ-global-rule, MEM-RTE-memory-extraction, EPI-RTE-memory-extraction, EPI-RTE-memory-admission, MEM-RTE-memory-maintenance and MEM-BAP-user-memory preserve the distinctions between candidates, structural admission, withdrawal and advisory use. Direct reads of SRC-1 apps/backend/src/services/memory.ts, apps/backend/src/agents/memory/memory-extractor-llm.ts, apps/backend/src/queries/memory.ts, apps/backend/src/trpc/memory.routes.ts, both DB schema memory ranges and apps/backend/src/components/ai/system-prompt.tsx confirm recent text-only sampling, normalization, existing-ID filtering, transaction, user-scoped active selection, current-chat exclusion, category/content projection and token-budget selection. Project enablement is not project-local memory storage. Reasons are not retained in the memory schema or projected to the default consumer. Extraction and admission overlap but have different endpoints; the category-specific content objects do not duplicate the assembled prompt or invalidate the retained-row container.
  • Withdrawal claim. MEM-CLM-memory-replacement correctly distinguishes the extraction prompt's deletion wording from soft supersession. SRC-1 apps/backend/src/queries/memory.ts changes supersededBy and filters active rows; explicit deletion is a different query. Neither deleting a successor nor deleting its source chat implements restoration of the old row. The schema's chat deletion action can remove lineage without withdrawing derived content. This source mismatch is not a remaining report conflict.
  • Checkpoints and work products. MEM-OBJ-compaction and MEM-RTE-chat-compaction correctly distinguish marker content, model summarization, in-process replacement and later reconstruction. SRC-1 apps/backend/src/services/agent.ts uses an optional stream writer for marker persistence; generate does not acquire the same checkpoint channel merely because prepareStep compacts. MEM-OBJ-user-files and MEM-RTE-saved-files distinguish persisted payloads, metadata, readable conversion and explicit access. apps/backend/src/services/storage/user-files.ts supplies scoped byte/text storage and conversion; apps/backend/src/agents/tools/read.ts and apps/backend/src/agents/tools/write.ts provide the model-facing route. Their limited inventory does not assert complete conversion fidelity, cleanup or provider durability.
  • Query evidence and lineage. EPI-OBJ-query-results and EPI-RTE-query-results correctly distinguish full retained rows, selective presentation and replacement under a query ID. SRC-1 apps/backend/src/agents/tools/execute-sql.ts, apps/backend/src/components/tool-outputs/execute-sql.tsx, apps/backend/src/agents/tools/read-query-result.ts and apps/backend/src/mcp/tools/context-layer.ts support cache mutation, default 40-row rendering, paging and full direct-MCP serialization. Semantic-query edit rejection and stale-output suppression preserve lineage, not business truth. The conditional derivation and indeterminate model-text classifications are appropriate without a concrete warranted query or answer instance.
  • Evaluation and delivered workflow. RT-RTE-test-answer, EPI-RTE-test-comparison and the workflow account describe one connected analytics → extraction SQL → execution/repair → reference comparison → report chain. Direct reads of SRC-1 apps/backend/src/routes/test.ts, apps/backend/src/services/test-agent.service.ts, apps/backend/src/services/test-agent-verification.ts, the query-result executor in apps/backend/src/services/duckdb.service.ts and comparison/delivery ranges in cli/nao_core/commands/test/runner.py confirm ordering and ownership. The extractor prompt receives table descriptions and expected columns, not an independent expected-row oracle. One execution repair precedes comparison; a data mismatch does not rerun analytics. Both-empty, assertions-only and no-verification branches qualify pass semantics. Saving a report extends runtime's inspection coverage without contradicting nonpersisted test-chat generation or establishing later model consumption. The workflow declares no duplicate records.
  • Recommendations and live admission. RT-RTE-context-recommendation, MEM-OBJ-context-recommendation, EPI-RTE-finding-reconciliation and RT-RTE-context-delivery support the separation of model diagnoses, identifier repair, fingerprint/impact/dismissal disposition, retained reasons, optional PR creation and live byte admission. Direct reads of SRC-1 apps/backend/src/services/context-recommendations.reconcile.ts and apps/backend/src/services/context-recommendations.service.ts confirm administrative selection rather than root-cause acceptance. PR creation in apps/backend/src/services/context-pr.service.ts is not live-context mutation. apps/backend/src/services/context-explorer-git/live-context.service.ts, cli/nao_core/context/git.py, docker/entrypoint.sh and cli/nao_core/commands/sync/__init__.py supply distinct fast-forward, reset and provider/template delivery policies. No successful learning, observed theory instance or exercised self-improvement follows from these paths alone.
  • Claims, containment and reconciliation. RT-CLM-context-readonly, EPI-CLM-analytics-answers and EPI-CLM-reliability-visibility are scoped tool or attributed design claims, not observations of benefit. All declared Part of relations are supported: checkpoint and recommendation objects belong to enclosing routes; user-memory delivery is within project guidance; query values are within ToolContext; comparison and finding reconciliation are downstream parts of test and diagnostic routes. Different parent/part kinds are explained. Valid containers remain alongside their parts. No evidenced-absence records are declared, and uninspected alternatives are not treated as absent. Reconciliation's supersession, independent-convergence account and dispositions of all memory integration issues preserve the current findings without selecting stronger statuses. No unresolved cross-report conflict remains.

Consequential omission questions and stopping decision

The search basis was the frozen top-level tree, tool registration, alternate shipped callers and retained-state producer/consumer dependencies. I selected questions that could change the principal runtime and lens conclusions: whether direct MCP changes query delivery; whether another output editor bypasses the ordinary context restriction; whether fork producers consume checkpoints across chats; whether recommendation postprocessing changes the retained population; and whether shipped maintenance instructions add a reference-governance and criticism loop.

Direct MCP exposes full adapter output and a separate cache, already covered by the query route. SRC-1 apps/backend/src/agents/tools/str-replace.ts rejects project-context edits under the same story/storage distinction as write, so it does not overturn the scoped read-only claim. A direct read of apps/backend/src/agents/tools/load-skill.ts confirms capability-dependent instruction delivery, not automatically learned guidance. Fork seeding, feedback coverage and maintenance skills establish the additional inventory limits below. They do not contradict ordinary continuation, primary diagnostic admission or the selected reference-SQL workflow.

Optional omission expansion stopped because the selected challenges have supported dispositions and no identified material contradiction remains undisposed. Further inventory of providers, channel policies, story lifecycle or deployment would not establish the missing run or causal evidence. The current wording preserves the positive mechanisms without asserting complete memory-route enumeration, an all-routes epistemic negative or a comprehensive security judgment. Those stronger conclusions are withheld, not inferred from unsearched behavior.

Blockers

none

Limits

  • MEM-RTE-chat-compaction, MEM-OBJ-compaction and RT-RTE-web-chat do not inventory fork initialization fully. SRC-1 apps/backend/src/trpc/chat-fork.routes.ts calls useLastCompaction before seeding shared-chat and selected-story forks; full-story paths can seed query data separately. Retained summaries/data therefore have additional new-chat consumers. The reports already exclude a complete fork account, so ordinary continuation and compaction findings remain valid, but cross-chat replay inventories and checkpoint-copy comparisons remain incomplete. The shared-chat seed is:

const rawMessages = await chatQueries.getChatMessages(share.chatId); const seededMessages = compactionService.useLastCompaction(rawMessages); --- apps/backend/src/trpc/chat-fork.routes.ts - MEM-OBJ-context-recommendation and EPI-RTE-finding-reconciliation describe a supported primary admission chain, not the complete recommendation population. SRC-1 apps/backend/src/services/context-recommendations.feedback-coverage.ts adds existing-link attachment, a separate model clustering call and fallback handling after primary reconciliation. Both reports explicitly leave this postprocessor unassessed. Their primary structural-admission and non-acceptance findings remain supported; withhold complete producer, lineage and model-role inventories, and do not infer that every recommendation passed the primary impact floor. The alternate dispatch is:

const clusters = await clusterFeedbacks(projectId, model, remaining); if (clusters.length > 0) { await applyFeedbackClusters(projectId, runId, clusters, { dismissedSet, existingByFingerprint }); } else { await applyFallbackRecos(projectId, runId, remaining, { dismissedSet, existingByFingerprint }); } --- apps/backend/src/services/context-recommendations.feedback-coverage.ts - EPI-RTE-test-comparison, EPI-CLM-reliability-visibility and RT-RTE-context-delivery are not a complete account of shipped maintenance guidance. SRC-1 skills/audit-context/SKILL.md directs failure classification and one-change-at-a-time retesting; skills/create-context-tests/SKILL.md asks for user validation of references and later diagnosis of saved outputs. This is an additional doctrine-level editor/test loop, not observed execution or a nao-owned external editing harness. The reports explicitly leave it unassessed; retain the positive runtime/test findings, but withhold an all-routes judgment about reference governance, criticism, saved-report consumption or theory building. An explicit admission instruction is:

  1. Have the user validate — confirm prompts match their team's phrasing and SQL matches their definition of truth. --- skills/create-context-tests/SKILL.md