LLM Wiki v2 report verification

Type: agentic-system-analyses/types/agentic-system-verification.md

Verification

Judgment and evidence basis

The reports support a bounded source-grounded account of a persistent knowledge pipeline, with local inventory limits below. They do not establish factual preservation, reliable grounded answers, exercised improvement or client isolation. No material blocker was found. The supplied structural report says none; all four correction-answer files are empty, so there are no correction arguments or structural failures to dispose of.

I read all supplied members and contracts, independently checked SRC-1 HEAD against the frozen revision, enumerated its tracked tree and checked that the checkout had no modifications. Source inspection remained within boundary. I neither executed the target nor consulted another source or prior analysis. The evidence below is implementation or attributed doctrine/reported operation, never independently observed execution.

Runtime and workflow

I independently followed Wiki construction, ingest preparation, unchanged-origin checks, locked filing and ask into Librarian, Researcher and the step implementations. RT-RTE-source-filing correctly separates model proposals from Python progression and persistence. RT-RTE-question-answer correctly orders filtering, checked selection, history, conflict guidance, spreadsheet expansion and answering. The workflow account connects two separately triggered invocations through disk state; it does not invent an automatic ingest-to-question trigger or delegated worker loop. Its preparation, consolidation, post-write enrichment, terminal delivery and caller-owned retry findings hold for the selected classic workflow.

I challenged the consequential controls against alternate branches, not only their nominal checks:

  • Citation metadata is intersected with selected/history note paths, but answer text is not rewritten or checked for entailment. History follows previous_version after the field filter; direct reads bypass question selection. Thus the reports correctly reject an authorization interpretation of where. SRC-1 src/llmw2/agents/researcher.py, src/llmw2/agents/steps.py, src/llmw2/bundle/tree.py.
  • The store uses cooperating thread/process locks, stale stamps and individual replacement, not a multi-file transaction or atomic compare-and-swap with arbitrary editors. Filing errors forget cached state; a relating ModelError can occur after admission and leave the note intact. The runtime's stated protocol strength and workflow's partial-effect account are supported. SRC-1 src/llmw2/bundle/store.py, src/llmw2/wiki.py, src/llmw2/agents/librarian.py.
  • CROW route postprocessing covers custom route creation when routed, while planned placement can skip routing. Manual create_folder, area over-limit flags and empty-tree fixed-folder behavior prevent a universal folder-cap or fixed-rebuild invariant. These are reachable public/library paths, not merely malformed typed inputs. RT-RTE-batch-sync and RT-RTE-manual-maintenance preserve the distinctions. SRC-1 src/llmw2/agents/steps_crow.py, src/llmw2/wiki.py.
  • Classifier missing/refused keys are configuration errors, unlike ModelError fallback. Output-cap refusal, caller overrides and relay flattening prevent universal provider-cap or client-tool guarantees. RT-RTE-client-relay correctly leaves client computation and isolation outside server enforcement. SRC-1 src/llmw2/models/classifier.py, src/llmw2/models/llm.py, src/llmw2/models/relay.py, src/llmw2/memory.py.

RT-CMP-llm and RT-CMP-classifier appropriately distinguish external numerical parameters from adapters and caches; model names do not prove immutable weight identity. RT-OBJ-prompts, RT-RTE-runtime-configuration, RT-BAP-model-instructions and RT-BAP-client-instructions correctly separate instruction loading, caller admission and requested model behavior. Settings replacement, prompt precedence and incomplete function dependency hashes match their source anchors.

Memory and epistemic records

I checked every declaration's source-native referent and evidence status, including the operative-object and authority records, rather than treating the analysts' reported checks as verification. The memory objects have distinct consumers: main body and metadata; folder displays; raw extracted text; hidden archive; log; sync state; proposal queue; classifier cache; replay; and evaluator outputs. Their anchors in SRC-1 src/llmw2/bundle/store.py, src/llmw2/bundle/tree.py, src/llmw2/bundle/proposals.py, src/llmw2/sources/sync.py, src/llmw2/models/cache.py and benchmarks/replay.py support the reported storage, lineage and horizons. In particular, full_text omits most frontmatter and See-also, archives have an explicit historical reader despite ordinary-load exclusion, removal marking leaves notes, rebuild loses removal marks, and purge follows source references rather than arbitrary semantic copying.

MEM-RTE-source-reduction matches map/reduce splitting, trimming, combine and warning-only lost-value detection. MEM-RTE-note-read is registered MCP delivery, not a hypothetical Store caller. MEM-RTE-answer-reuse correctly distinguishes exact-key response substitution from learning or current-provider measurement; replay keys omit the writer cap and classifier endpoint. MEM-BAP-retained-knowledge establishes delivery and intended influence, not observed compliance. No uniform human-curation claim is inferred from caller initiation or the proposal's user label.

For epistemic records I traced structural checks, merge value evidence and retry choice; revision eligibility, sentence filtering and assembly; conflict computation and hint consumption; audit proposal and apply; health/lint; purge; and lexical fact scoring. The separate route records preserve different targets, timing, warrant and force. SRC-1 src/llmw2/agents/steps_classic.py, src/llmw2/agents/revisions.py, src/llmw2/agents/conflicts.py, src/llmw2/agents/researcher.py, src/llmw2/wiki.py, src/llmw2/audit.py, src/llmw2/bundle/check.py, benchmarks/note_facts.py support those distinctions. The revision classifier gates depend on classifier presence; the classic/CROW comparison describes default wiring, not a prohibition on injecting a classifier in classic library use.

RT-CLM-factual-preservation and EPI-CLM-source-answers are correctly attributed prompt goals rather than guarantees. EPI-CLM-benchmark-results matches the reported PostHog setting and keeps the unavailable harness and one-run/component-attribution limits. Retrieval evaluates selected-note identity, not answer correctness; lexical fact matching does not establish propositional truth. The epistemic conclusion does not infer ampliative conjecture merely from model generation, or acceptance merely from retention. Learning, internal reasoning, method criticism and exercised self-improvement are not bundled into an unsupported negative.

All declared Part of: relations are supported: source reduction, merge evidence/choice and revision check/admission belong to filing; conflict check/hint belong to answering; origin purge belongs to maintenance. Check families spanning both filing and answering remain separate rather than falsely contained in one. Workflow annotations add workflow findings without redeclaring runtime identity. No evidenced-absence records or supersessions are declared. The reconciliation's decisions against whole-record identity for source/raw, candidate/stored note, caches/components and authority channels are supported; none warrants a supersession.

I assessed the unresolved record-separation conflict, rather than accepting its proposed correction as an established blocker. It cites both containers, the distinct route findings, evidence and the prevented part-level reconciliation. The missing check-result object parts are real, but each consequential check and consumer already has a separate route finding. Neither container emits a common warrant or a complete comparison value. Under the verification type's materiality threshold this is a coarse-inventory limit, not grounds to reject the supported mechanism account. The declaring epistemic analyst would own finer check-result declarations; reconciliation correctly does not allocate them.

Omission questions came from the frozen tree and producer/consumer dependencies: could alternate read paths expose more than selected-note displays, could a shipped custom step implement criticism beyond the defaults, and could raw expansion compute rather than only retrieve content? I inspected HTTP /note, the shipped custom-selection and multi-pass examples, and spreadsheet brief/rows_for/_stats, in addition to the maintenance, cache/replay and history alternatives above. These searches found the local omissions retained below, not a contradiction of the default pipeline account. I stopped optional expansion because these alternatives have bounded dispositions and no identified material challenge remains undisposed. Other benchmark families remain available but incompletely traced; the reports do not claim a complete evaluator taxonomy, system-complete absence of acceptance, or causal performance verification. More evaluator files alone do not justify broadening this review into an exhaustive audit.

Blockers

none

Limits

  • EPI-OBJ-check-results and MEM-OBJ-evaluation-results remain coarse containers, as recorded in the reconciliation conflict. Individual check-result parts are not separately declared. The route-level distinctions remain supported, but readers must withhold a complete part-level inventory or any common retention horizon, evaluator warrant or behavioral authority inferred from these containers alone.
  • MEM-RTE-note-read, MEM-OBJ-note-metadata and RT-RTE-question-answer do not form a complete direct-read inventory. SRC-1 src/llmw2/server.py also exposes HTTP /note with body and frontmatter, unlike the MCP display. Its implementation returns:

    return {"path": n.rel, "title": n.title, "summary": n.summary, "tags": n.tags, "frontmatter": n.frontmatter, "body": n.body} --- src/llmw2/server.py The MCP and answering omissions remain true for those channels. Do not generalize them to every shipped consumer or infer that metadata and related-link text are globally unavailable.

  • MEM-OBJ-raw, RT-RTE-question-answer and EPI-OBJ-answer only partially characterize spreadsheet expansion. SRC-1 src/llmw2/bundle/sheets.py supplies numeric-column count/sum/min/max across all rows when the sheet exceeds the row budget; this arithmetic is additional derived answer context, not merely selected CSV. Its threshold is:

    if numbers and len(numbers) >= 0.8 * len(values): --- src/llmw2/bundle/sheets.py The retained reports still support bounded source-derived context and unverified answer semantics. They do not support a complete derivation inventory or an inference that every numerical value delivered to answering was copied from a selected row. Numeric parsing and source truth further bound the statistics' warrant.

  • The workflow account, RT-RTE-runtime-configuration and epistemic bounded conclusion cover defaults and caller extension capacity, not all shipped custom-step workflows. SRC-1 examples/multi_pass_summarize.py implements draft, source-relative critique and rewrite inside summarization; examples/custom_select.py supplies deterministic field selection and an answer built from metadata. The multi-pass handoff includes:

    system, f"Rewrite the note to fix them. Reply with the note only.\n\nProblems:\n{critique}\n\nNote:\n{draft}", op="rewrite" --- examples/multi_pass_summarize.py These do not contradict the selected classic workflow or establish criticism-driven changes to the system's own methods. Withhold an exhaustive workflow inventory and any system-wide inference that no shipped critic/rewrite route or non-model answer route exists. No example was executed.