nao synthesis
Type: agentic-system-analyses/types/agentic-system-synthesis.md
Bounded synthesis
From project context to an analytics conversation
nao is an enclosing analytics runtime, not only a memory extractor or a tool exposed to another agent. Its CLI builds project context; its backend constructs model instructions, runs a model-and-tool loop, streams answers and retains conversations. The assessed whole-system boundary includes context maintenance, alternate callers and evaluators. Evidence is inspected implementation and attributed design in SRC-1, https://github.com/getnao/nao, at commit 0085625a8fd22e4f404b6a3971d153f6b6699479, with an analysis cutoff of 2026-10-14. No target code, tests, models or services were executed. Whole-system scope does not mean exhaustive inspection. External providers, warehouse contents, repository governance and deployed infrastructure are not frozen. See the boundary and Source register.
An ordinary web request passes authentication, role, budget and chat-ownership checks before nao reloads history and constructs an agent. Code owns admission, grants, context preparation and persistence. The selected remote model chooses exposed tools and composes the answer; nao dispatches calls to database, storage, MCP or execution adapters. These responsibilities are separated in RT-RTE-web-chat and RT-OBJ-tool-context. There is no reference-answer oracle in ordinary chat.
Default instructions combine project rules, active user memories, context inventories and available skills. A complete prompt override can replace that composition, while user-group restrictions are appended separately. Natural-language instructions direct the model; they are not executable access controls. The ordinary write tool permits storage/story outputs and rejects project-context edits, but separate synchronization and deployment routes can replace context. Thus read-only describes a tool boundary, not immutable project guidance. See RT-OBJ-system-prompt, RT-BAP-project-guidance and RT-CLM-context-readonly.
SQL tools supply reusable evidence rather than verified interpretations. Full rows enter a cache while ordinary model rendering defaults to 40 visible rows; explicit paging can recover more. Reruns replace results under a query ID and suppress stale bodies. These mechanisms preserve selected values and lineage, not source-data truth or business-metric fidelity. Local SQL can derive results conditionally on its inputs and semantics; no supplied query instance establishes a warranted new external-world claim. Generated answers remain semantically unclassified between restatement, derivation and conjecture without concrete candidate evidence. See EPI-RTE-query-results and the epistemic assessment.
The effect envelope depends on configuration and caller. SQL validation rejects disallowed writes on the inspected executeQuery path, not every conceivable mutation. Monty Python and network-enabled Boxlite shell/Python execution have different adapter controls; sandbox availability, experimental flags, secrets and deployment determine the actual grant. No universal human confirmation barrier or isolation invariant follows. The runtime's forcing-route account retains those conditions.
Delegation also has distinct contracts. A beta-gated explore task receives fresh instructions plus an explicit task and returns only its report to the parent model, while sharing ToolContext. It isolates conversation visibility, not the permission principal. MCP ask_nao instead encloses a durable background chat and can return a polling handle after its synchronous response budget. Direct MCP tools are another, host-driven route. Automations run under their owner and can add outbound integration tools; checking that required tools were called establishes completion bookkeeping, not answer truth or exactly-once external delivery. See RT-RTE-explore-task, RT-RTE-mcp-chat and RT-RTE-automation-run.
What survives a conversation
The strongest supported learning-related contribution is implemented trace-fed revision of later model inputs. It has three distinct families: extracted user guidance, continuation summaries and retained diagnostic findings. Their automatic writes and later consumers are wired, but improved capacity for future action is not demonstrated. Profile verification confirms bounded existence findings, not complete inventories or observed benefit.
User-memory extraction runs asynchronously after request dispatch, not after checking the current answer. It samples at most 17 recent user/assistant text messages, with character caps, and asks a separate model for persistent directives and profile facts. Code normalizes output and filters empty content and invalid supersession references before committing it without per-item human approval. Those are structural admission checks, not factual acceptance. See MEM-RTE-memory-extraction and EPI-RTE-memory-admission.
Memory rows are user-scoped rather than project-keyed, despite project enablement controls. Default prompts automatically supply active category/content, excluding current-chat rows, prioritizing rules and newer entries, and applying a 1000-token content budget. The consumer receives neither original evidence nor reasons. This is identity/order/budget selection, not task-targeted semantic retrieval. Overrides or disablement can omit delivery. Supersession retains an inactive predecessor; explicit deletion and disabling differ, and deleting a successor does not automatically restore its predecessor. See MEM-OBJ-memories, MEM-BAP-user-memory and MEM-RTE-memory-maintenance.
Compaction has a shorter horizon. Above the configured threshold, code replaces the eligible old prefix with a model summary while preserving the system instructions and current-user tail. The summarizer receives image-stripped, backward-budget-selected input: older material can disappear before summarization even though its entire eligible span is replaced. A successful streamed marker enables later reconstruction; nonstreaming generation retains the replacement for subsequent in-process steps without that persistence channel. Summary faithfulness and successful context fit are unobserved. Files use explicit path reads, not automatic pointer following; documents can remain outside the window until requested. See MEM-RTE-chat-compaction and MEM-RTE-saved-files.
A connected evaluation workflow
The representative workflow is a CLI analytics test with reference SQL, selected because it connects model work, a second model handoff, executable rejection, bounded repair and delivered reporting. The test author supplies the question and reference SQL. nao runs analytics through nonstreaming generation, then asks a model to write extraction SQL over the fetched result tables. Its prompt receives the question, response messages, expected column names and table descriptions, not expected row values. Code executes that SQL under a local safety policy and offers one repair after execution failure. The CLI then compares extracted rows with reference-query rows and checks tool assertions. A final mismatch records failure; it does not rerun analytics. Results are saved as timestamped JSON and printed summaries, with a failed-batch exit status. This is source-defined delivery, not an observed successful test. See RT-RTE-test-answer and the workflow account.
This supplies a genuine bounded acceptance criterion: tool assertions and reference-data comparison must pass on the data-verification branch. It does not independently verify that extraction SQL faithfully encodes the prose answer, that the reference captures the intended business rule, or that both warehouse queries share an atomic snapshot. Sorting and tolerances affect equality; both-empty datasets pass. Assertions-only and no-verification branches have weaker pass meanings. A saved report is neither an automatic deployment gate nor, by retention alone, a later learning input. See EPI-RTE-test-comparison and the workflow annotation.
Feedback, proposed corrections and later guidance
The recommendation route inspects retained usage/feedback and context costs, requests diagnoses and edits from a restricted agent, repairs references, reconciles fingerprints/status/impact and retains findings. Future diagnostic runs receive compact summaries, not automatically all stored reasons. Dismissed fingerprints can veto repeat admission. Optional automatic PR creation uses a disposable clone rather than mutating live context. The status applied can follow either reconciliation resolution or successful PR opening; it proves neither a PR nor merge or deployment. See RT-RTE-context-recommendation, MEM-OBJ-context-recommendation and EPI-RTE-finding-reconciliation.
Live delivery is separate: UI Git pull fast-forwards, while Python refresh and container startup can hard-reset fetched content; CLI sync writes provider outputs and templates. Later agent construction or reads can consume admitted guidance, but successful admission and subsequent dependence are unobserved. Git/configuration guards establish byte admission, not acceptance of a diagnosis or improvement in answers. See RT-RTE-context-delivery.
The following properties remain independent, as assessed in the epistemic conclusion:
- Theory-builder condition 1, localized content: diagnosis fields and prompts support causal-candidate formulation, but no actual formulated theory instance is established.
- Condition 2, consumption: delivery of findings is wired; decisions depending on the content of a theory are unobserved.
- Condition 3, content-directed criticism: prompts request diagnosis and resolution, but no demonstrated blame-correcting criticism and resulting revision or changed reliance is established.
- Condition 4, iteration: retained findings can reach another diagnostic round; retained criticism actually shaping that round is unobserved.
- Learning from criticism: trace-fed guidance updates are implemented; criticism of a consumed theory improving future capacity is not established.
- Reflection: usage/context self-representation and a proposal-to-guidance pathway are wired; exercised two-way causal dependence remains unobserved.
- Autonomy: computation performs internal diagnostic and proposal steps. Autonomous completion of external review/live admission, or an autonomous theory builder, is not established.
- Self-improvement: the recommendation path offers improvement-directed proposals and separate mechanisms capable of admitting later guidance. No run closes the links from evidence-responsive update through live organizational change to subsequent behavior causally depending on it; successful improvement additionally lacks outcome evidence. User-memory revision and compaction remain positive mechanisms without being upgraded to that conclusion.
For open analytics, the supported result is evidence access, selective context and answer delivery, not certified answers. For reference-SQL evaluation, the supported result is a bounded comparison with explicit failure/recovery semantics. For feedback-directed context maintenance, the supported result is durable diagnostic state and optional review requests, not a demonstrated closed improvement loop. A linked trace of diagnosis, review/admission, changed prompt or context, and subsequent content-dependent behavior would alter the exercised-path assessment. Validated references and extraction-fidelity checks would strengthen evaluation; controlled outcome comparisons would be needed for benefit attribution. Concrete theory/criticism/revision instances would change the individual theory-builder assessments. These requirements follow the gaps in the cited routes, not a product ranking.
Limitations
| Limitation | Affected SRC IDs and record citations | Inspected boundary | Conclusion prevented | Evidence that would resolve it |
|---|---|---|---|---|
| No execution or causal comparison | SRC-1; RT-RTE-web-chat, MEM-RTE-memory-extraction, RT-RTE-context-recommendation, RT-RTE-context-delivery | Frozen implementation and doctrine/design; no target execution | Activation, reliability, actual guidance dependence, improved capacity and exercised or successful self-improvement | Linked execution artifacts showing the update and subsequent dependence; independently specified outcomes and suitable comparisons for benefit |
| Fork consumers are additional to ordinary continuation; full copying/seeding is not inventoried | SRC-1; MEM-RTE-chat-compaction, MEM-OBJ-compaction, RT-RTE-web-chat; report-verification limit | Verification inspected chat-fork.routes.ts: shared-chat and selected-story forks call useLastCompaction; full-story routes can seed query data separately | Complete cross-chat replay and checkpoint-copy comparisons; ordinary continuation findings remain supported | Full fork producer, copied payload, marker and downstream-consumer tracing for each branch |
| Feedback postprocessing adds recommendation producers beyond primary reconciliation | SRC-1; MEM-OBJ-context-recommendation, EPI-RTE-finding-reconciliation; report-verification limit | Verification inspected context-recommendations.feedback-coverage.ts: existing-link attachment, separate model clustering and fallback handling | Complete recommendation population, lineage or model-role inventory; no inference that every finding passed the primary impact floor | Accepted records tracing postprocessor inputs, model role, fallback admission, transformations and later consumers |
| Shipped maintenance skills add a doctrine-level editor/test loop | SRC-1; EPI-RTE-test-comparison, EPI-CLM-reliability-visibility, RT-RTE-context-delivery; report-verification limit | Verification inspected audit-context failure classification/one-change retesting and create-context-tests user validation of references/later saved-output diagnosis | All-routes judgments about reference governance, criticism, saved-report consumption or theory building; instructions do not establish execution or a nao-owned external harness | Full skill-to-editor/test/reader account and linked executions establishing the respective properties |
| Other included memory and context alternatives remain incompletely traced | SRC-1; MEM-RTE-saved-files, RT-RTE-explore-task, RT-RTE-context-delivery, MEM-OBJ-memories; profile verification | Dynamic skills, channel-specific history/memory, story publication/app-state/blob lifecycle, other byte producers/cleanup, CLI/API provider admission, initial memory provenance/control | Exhaustive memory-route, lineage, admission or payload classifications; all ten profile axes remain partial, not absence claims | Producer/selector/consumer records for each named alternative, including opaque payloads rather than only displayed summaries |
| Source truth and evaluator fidelity are unresolved | SRC-1; EPI-RTE-query-results, EPI-RTE-test-comparison, RT-RTE-test-answer | SQL/semantic adapters, extraction and CLI comparison, not full compiler/helper/data-provenance audit | Universal answer correctness, reference correctness, extraction faithfulness or generalization from a test pass | Validated data/reference provenance, concrete answers with checked extraction, compiler/helper inspection and representative evaluation evidence |
| External provider state and deployed enforcement are not frozen | SRC-1; RT-CMP-agent-model, RT-CMP-compaction-model, RT-CMP-verification-model, MEM-CMP-memory-extractor, RT-OBJ-tool-context | nao-side configuration, call roles and adapters; external weights, services, permissions and isolation uninspected | Weight fixity, provider-side learning, complete cross-channel authorization, universal safety, exactly-once effects or deployment guarantees | Immutable provider identities and relevant provider evidence; deployed configuration, adapter/runtime audits and scoped operation traces |