LLM Wiki synthesis
Type: agentic-system-analyses/types/agentic-system-synthesis.md
Bounded synthesis
From incoming sources to retained claims
LLM Wiki is a Python-controlled knowledge-processing system, not a model-directed tool loop. It imports research and security material, asks models to extract and merge claims, saves Markdown, and supplies retained content to later readers and model calls. This account covers the shipped monitors, drivers, API/browser client, maintenance, evaluation and synthesis paths at Git revision d2bd485921f6c7b6071afbb389f44ab8e4e83c69, with cutoff 2026-10-17. SRC-1 supplies inspected implementation and declared design. No target execution, model response trace or causal experiment was supplied or performed. Wiring therefore establishes capacities, not successful operation or benefit (boundary, Source register).
The representative workflow is the repository-root batch invocation, python tools/pipeline_all.py. Python polls monitors, scans raw files, normalizes text, extracts from normalized documents lacking JSON sidecars, integrates every discovered sidecar, and rebuilds a Markdown catalog. Subprocess stages and filesystem scans own the handoffs. Models receive stage-specific text and return completions; the caller determines subsequent actions. The completion helper uses a fresh system/user pair, retries the configured primary endpoint and can send the same input to OpenRouter. It does not itself persist conversational history (RT-RTE-batch-ingestion, RT-RTE-model-call, workflow account).
The durable chain is not lossless evidence transfer. Normalized documents retain source-derived text; extraction creates model-generated JSON claims and requested evidence fields. JSON parsing is an executable syntax gate, not full schema or factual validation. Integration carries claim text, confidence and source ID forward but omits evidence sentences, entities and relationships from its model input. The sidecar remains separately available. A new page writes the projected claims directly. An existing page supplies its old body plus new claims to a merge model: any nonempty response can replace the body, while empty output appends. History preservation and contradiction handling are requested in instructions, not checked invariants (MEM-OBJ-normalized, MEM-OBJ-extracted, EPI-RTE-claim-projection, RT-RTE-knowledge-admission).
The batch delivers files, sidecars, catalog and logs rather than an answer. Its final success-labelled message can follow caught subprocess failures. Existing sidecars suppress extraction, yet all sidecars remain eligible for reintegration; there is no batch-wide rollback. Other drivers are materially different: the daemon writes to a fixed test-integration destination, the CVE monitor has its own immediate route, and API/browser saves can overwrite articles without extraction or merge. Alerts and pending-review listings do not constitute pre-write human approval. These counterpaths prevent treating the system as one transactional, exactly-once pipeline (RT-RTE-batch-ingestion, RT-RTE-scheduled-drivers, RT-RTE-api-mutation, workflow account).
From retained pages to later decisions
Retention has two distinct effects. Natural-language bodies supply context to later models and people. Symbolic fields and access state control Python decisions: monitor IDs suppress acquisition, sidecar existence suppresses extraction, and page metadata selects maintenance tasks. Persistent files and SQLite rows outlive transient queues and schedule counters. Requested Git history is conditional on externally created commits; saving an article does not create a checkpoint (MEM-OBJ-wiki-body, MEM-OBJ-ingested-ids, RT-RTE-api-mutation).
Search rebuilds BM25 from current Markdown tokens on each request and returns positive-score results; the catalog is not a persisted semantic search index. Conflict and unverified status do not quarantine pages from search or synthesis. Ordinary paper analysis selects at most five topic-matching pages and supplies 2,000-character prefixes. Newsletter selection supplies up to five 400-character body snippets per bucket. Files remain available to code, but omitted text is not automatically reachable within the model window (RT-RTE-wiki-consumption, memory consumption annotation).
Checks have different domains and consequences. Deep lint requests model contradiction judgments over 600-character page prefixes and can add conflict notes; it is a real semantic-check route, not evidence that every claim is checked. Weekly retrospective validation instead inspects existing tags and metadata. Repeated integration of the same sidecar can raise source_count; a count/confidence predicate can then raise confidence without additional source evidence. This is a supported possible code path, not an observed confidence-inflation incident (RT-RTE-research-maintenance, MEM-OBJ-wiki-metadata, EPI-RTE-reinforcement-selection, EPI-RTE-confidence-update).
Newsletter generation illustrates a separate reliance boundary. Its context formatter omits the collected status field, though snippets may still contain conflict notes. A nonempty response of at least 200 characters receives verified: true and confidence 0.95 without an evidence-consuming check. Saved under the wiki, it can enter later search and synthesis, including synthesis of prior newsletters. Re-entry supplies neither independent corroboration nor evidence acceptance. Actual extraction, merge and synthesis propositions remain semantically indeterminate without linked outputs: requested novelty does not establish conjecture, and retention does not establish accepted new knowledge (EPI-RTE-newsletter-context, EPI-RTE-newsletter-admission, MEM-OBJ-newsletters, epistemic conclusion).
Ordinary paper generation wires two critique/improve rounds, but the separate review may be discarded before the next round. Long-form generation skips critique. More importantly, report verification found that its section prompts do not receive the computed wiki insights, external-source results or prior generated sections. It cannot be treated as evidence-conditioned like ordinary drafting merely because it calls wiki analysis (RT-RTE-wiki-consumption, report verification).
Feedback, method change and assessment
The strongest supported learning-related contribution is a local automatic trace-fed feedback route: execution outcomes feed durable metrics and experiment/hypothesis/decision state, which can shape later optimizer cycles. This is more than unused logging. It is not evidence of improved future capacity. The proposed prompt is neither installed nor saved as active guidance by the inspected commit method; evaluation runs the existing extractor on at most ten fixtures, uses process/JSON success and model confidence, and fixes conflict rate at zero. Rejection can reset Git history/index even after a failed commit, rather than safely restoring candidate content. Thus the real feedback-to-proposal mechanism survives, but a keep decision establishes neither an operative candidate change nor its effectiveness (RT-OBJ-metrics, MEM-OBJ-eval-set, RT-RTE-prompt-optimization, profile verification).
At the prompt-method boundary, theory-builder condition 1 (localized method content) is wired, and condition 2 (consumption of the current method) is wired. Condition 3 (content-directed criticism with challenged claim and blame placement) and condition 4 (retained criticism shaping a next round) are separately uninspected. Score-based rejection and critique requests do not supply that missing content. Whether criticism of a consumed theory improved future capacity is also uninspected, not disproved by missing traces. Reflection has a supported partial mechanism: metrics represent extraction behavior and mediate later proposal focus, but complete method-revision feedback is not established. Scheduling, proposal generation and admission have computational owners; autonomous theory building remains uninspected because the required criticism and iteration are unestablished. These properties are independent (RT-RTE-prompt-optimization).
For self-improvement, the declared objective is better extraction confidence/validity and fewer conflicts across optimizer cycles. Evidence can shape a proposed update, but installation into behavior-determining prompts is missing; candidate activation and subsequent operation depending on that candidate are consequently unestablished. Exercised or successful self-improvement cannot be attributed. Other retained-context and metadata-feedback paths remain real, without demonstrated improved capacity or causal benefit (RT-RTE-prompt-optimization, epistemic conclusion).
For a scenario requiring persistent compilation and later access, the inspected routes establish concrete file retention, destination-specific reuse, requested search and bounded automatic context supply. For a scenario requiring verified security/research conclusions, preserved evidence and history, safe unattended recovery, or demonstrated improvement, these mechanisms do not establish the required guarantees. An evidence-consuming acceptance gate naming intended use and scope, preserved lineage across projections, enforced review/withdrawal where needed, and inspected recovery paths would change the architectural assessment. Linked runs would establish activation; an installed candidate and independent correctness comparison would be needed to assess prompt effects, with causal attribution no finer than the intervention actually varied (EPI-CLM-source-integrity, RT-RTE-knowledge-admission, RT-RTE-prompt-optimization).
Limitations
| limitation | affected SRC IDs and record citations | inspected boundary | conclusion prevented | evidence that would resolve it |
|---|---|---|---|---|
| No observed execution or causal comparison | SRC-1; RT-RTE-batch-ingestion, RT-RTE-model-call, RT-RTE-prompt-optimization | Frozen implementation and doctrine only | Successful deployment, compliant outputs, activation, benefit and causal improvement | Linked execution artifacts and appropriately controlled outcome comparisons |
| External truth, provider internals and effective deployment remain outside the evidence | SRC-1; RT-CMP-main-model, RT-CMP-openrouter-model, RT-RTE-api-mutation | Client requests and handlers; no live feeds, pinned weights, operator settings or isolation tests | Source correctness, provider learning/fixity, confidentiality and deployed permission/isolation guarantees | Authenticated source evidence, provider/version contracts and inspected deployment configuration and tests |
| Actual generated semantics and retained criticism are unavailable | SRC-1; MEM-OBJ-extracted, RT-RTE-knowledge-admission, RT-RTE-wiki-consumption | Requested extraction/merge/critique and their projections | Faithful preservation, checked derivation, produced accepted novelty, content-directed criticism and criticism-fed improvement | Source-linked outputs, adjudications and successive criticism/revision artifacts |
| Alternate long-form context inventory is partial; carried report-verification limit | SRC-1; RT-RTE-wiki-consumption, memory consumption annotation, epistemic conclusion, report verification | tools/paper_agent_long.py section prompts omit computed insights, external results and prior sections |
Comparison treating long-form sections as evidence-conditioned like ordinary drafting or sharing prior-section context | Corrected per-section context records; changed prompts and linked traces if actual delivery is claimed |
| Initial lineage, generic caller control and complete metric-producer provenance unresolved | SRC-1; MEM-RTE-curated-poll, MEM-OBJ-eval-set, RT-RTE-api-mutation, RT-OBJ-metrics | Known subsequent transformations, not every initial admission or original feedback producer | Complete admission history, authority, trace-learning and trace-source inventory | Provenance and caller/admission records plus all metric-producer/later-consumer chains |
| Withdrawal and history recovery are not completely established | SRC-1; MEM-RTE-rss-reset, RT-RTE-api-mutation, RT-RTE-research-maintenance | Reset, edit/history, lint and validator surfaces; external commits and ignored state unavailable | Cascading invalidation, dependable rollback or a system-wide absence of either | Dependency-aware withdrawal/recovery records and linked state/version histories |
Profile verification declares no additional limit. No unresolved cross-report conflict was supplied. This synthesis adds no source reads or evidence.