LLM Wiki report verification

Type: agentic-system-analyses/types/agentic-system-verification.md

Verification

The pinned reports support a bounded implementation account, with one local inventory limit below. I read all supplied members and contracts. The correction-answer files are empty; report-check reports none, so there is no structural failure to route. I independently checked source behavior rather than treating analysts' validation statements as support. The checkout HEAD matches the boundary, and its working tree is clean. All source inspection below is SRC-1 implementation or doctrine/design at that revision. No target program, test, model call or service was executed; no observed-run or causal layer was established.

Runtime and workflow

I traced complete SRC-1 tools/common.py, tools/prompts.py, tools/ingest.py, tools/normalize.py, tools/extract.py, tools/integrate.py, tools/pipeline_all.py, tools/query.py and tools/index.py. The two model components, RT-CMP-main-model and RT-CMP-openrouter-model, correctly separate configured provider identifiers from pinned weights and inaccessible provider state. RT-OBJ-prompts correctly distinguishes interpreted instructions from symbolic registry selection. RT-RTE-model-call matches fresh message-pair construction, primary retries, cloud fallback and returned-string behavior. Neither delivery nor a configured name establishes compliance or parameter fixity.

RT-RTE-batch-ingestion, RT-RTE-knowledge-admission, and the workflow account preserve the actual stage ordering. JSON parsing precedes sidecar assignment/write; it does not enforce the requested schema or factual support. Integration projects claims to text/confidence/source ID. New pages bypass merge inference; existing pages accept nonempty replacement text, including conflict-marked text, while empty output appends. Metadata defaults and source-count increments match the source. The batch ignores caught subprocess failures at its call sites and reintegrates all discovered sidecars. Consequently completion labels, historical-preservation instructions and post-write catalog notifications cannot establish global success, semantic preservation or prior human acceptance. Workflow declares no new records, and its two annotations add compatible workflow-specific findings without changing runtime identities.

For alternate-route challenges, I independently read SRC-1 tools/daemon.py, tools/unified_daemon.py, tools/cve_monitor.py, tools/watchdog_monitor.py, complete api_server.py, and the browser's read/save/search dispatch in ui/src/App.jsx. RT-RTE-scheduled-drivers correctly distinguishes the fixed daemon destination, raw-adjacent checkpoint test, exception-only requeue, advisory CVE alerts and unified scheduling without the base lint/index loop. RT-RTE-api-mutation correctly retains direct-save bypass, queued rather than completed ingestion, newest-file association risk and dependence on external Git commits. Article path-prefix tests are not a proven containment boundary. These shipped callers are included, not incorrectly excluded as external applications.

Retention, maintenance and epistemic force

The normalized body/frontmatter, JSON sidecar, wiki body and wiki metadata are supported by MEM-OBJ-normalized, MEM-OBJ-extracted, MEM-OBJ-wiki-body, and MEM-OBJ-wiki-metadata. Their distinct text and symbolic consumers are preserved. The corresponding epistemic parts—EPI-OBJ-source-text, EPI-OBJ-extracted-claims, EPI-OBJ-evidence-fields, EPI-OBJ-wiki-body, and EPI-OBJ-wiki-metadata—do not establish additional stores or independently warranted content. EPI-RTE-claim-projection and its symbolic example accurately show evidence-field omission without implying deletion of the sidecar.

I read SRC-1 tools/rss_monitor.py, tools/curated_monitor.py, clear_rss_state.py, and the arXiv/GitHub monitor implementations. MEM-OBJ-ingested-ids correctly treats rejection markers as admission state rather than proof of integration. MEM-RTE-rss-reset deletes RSS admission rows in the command's relative database, not derived knowledge. MEM-RTE-curated-poll correctly traces the retained table's second column to search queries and dated raw writes without claiming authenticated source trust or known initial authorship.

Complete reads of SRC-1 tools/lint.py, tools/retrospective_validator.py and tools/research_agent.py support RT-RTE-research-maintenance, EPI-RTE-reinforcement-selection, and EPI-RTE-confidence-update. Deep lint is a real excerpt-based semantic judgment route; lightweight lint is warning-only. Weekly validation does not call the available contradiction helper. Reinforcement consumes metadata without a distinct-source or conflict-status exclusion. The repeated-integration countercase is valid as a code-path deduction, not an observed incident. Research result labels can denote queries, outlines or identified tasks rather than completed investigations. These findings support the epistemic ledger's separation of checking, software admission, retention and operational use; they do not establish evidence-consuming acceptance or subsequent lifecycle integration.

I traced SRC-1 tools/newsletter_agent.py, tools/paper_agent.py and tools/paper_agent_long.py through selection, context construction and finalization. RT-RTE-wiki-consumption correctly distinguishes reconstructed token BM25 from a persisted semantic index, and requested article delivery from model context. MEM-OBJ-newsletters, EPI-RTE-newsletter-context, and EPI-RTE-newsletter-admission correctly identify status-channel omission, the length gate, unconditional verification/confidence labels and eligibility for re-entry. Ordinary paper drafting omits the analysis contradictions field; critique can discard its separate review; long form bypasses critique. One additional long-form context omission remains a local limit below.

Improvement claims, identities and disagreements

I read complete SRC-1 tools/prompt_optimizer.py, tools/metrics_collector.py and tools/create_eval_set.py. RT-OBJ-metrics, MEM-OBJ-eval-set, and EPI-OBJ-eval-set correctly separate aggregates from fixture inputs and mutable extraction outputs. RT-RTE-prompt-optimization accurately identifies missing candidate installation, evaluation of the current extractor, constant conflict metric, baseline-before-log mutation and reset despite failed commit. Thus RT-CLM-prompt-ratchet is doctrine qualified by implementation, not established improvement. The separate theory conditions and learning/self-improvement conclusions do not turn opaque reasoning into an evidenced absence. The real metrics-to-proposal path survives the missing proposal-to-live-method link.

I checked the claim passages against SRC-1 README.md, complete AGENTS.md, and the trust/integrity section of llm-wiki.md. EPI-CLM-knowledge-compilation, EPI-CLM-source-integrity, and EPI-CLM-retrospective-validation preserve the difference between declared intent and implemented force. The two-independent-sources rule is CVE-specific. Newsletter verification lacks its own evidence check without needing that rule generalized. RT-BAP-prompt-guidance, RT-BAP-wiki-context, and RT-BAP-agent-doctrine correctly distinguish interpreted guidance, selected subject-matter context and host-dependent governance. None establishes model-native tool authority or enforced source truth.

Every declared record's source anchors and evidence interpretation were checked in these groups. No evidenced-absence records are declared. The explicit route Part of: relations describe actual subroutes of their cited containers: curated acquisition, claim projection, reinforcement selection/update and newsletter context/admission. No part replaces its valid parent. The reconciliation correctly supersedes the duplicate epistemic fixture-store record with memory's record, retaining both declarations and their compatible findings. Other object overlaps have different part/container or producer scopes; their nonsupersession is justified. Its dispositions cover memory's integration issues. No unresolved analyst conflict, missing conflicting finding, or inflated evidence status was found. Source-claim mismatches are not themselves cross-report disagreements.

Consequential omission questions and stopping decision

The frozen tracked tree and producer/consumer dependencies guided three challenges beyond simply checking quoted lines. First, could another shipped driver enforce pre-write review or complete the prompt-installation link? The daemon, watcher, API, unified driver and Python-wide caller searches did not supply such a path; the reports already preserve their material differences. Second, could another check justify factual acceptance or an independent optimizer oracle? Searches for contradiction, verification, active-prompt setters, metric producers and withdrawal were followed through the validator, research execution, fixture builder and SRC-1 test_e2e.py, test_phase2.py, tools/test_optimizer.py. These diagnostics check artifacts, labels or decision logic, not an evidence-consuming factual acceptance gate for the assessed writes. No tests were run, and no system-complete negative across all documentation is inferred.

Third, do alternate paper consumers receive the retained evidence that ordinary drafting receives? Full long-form source inspection found the dropped insight dependency below. The ordinary merge, newsletter and ordinary-paper read-back findings remain supported. Monitor inspection also confirmed additional GitHub ETag transport state and a written CVE timestamp whose request filter is disabled; the memory coverage already withholds a complete feed-specific freshness inventory. Neither establishes claim warrant or contradicts the bounded deduplication account (SRC-1: tools/github_monitor.py, tools/cve_monitor.py). Further expansion stopped because each selected challenge has a supported disposition or the explicit local limit, with no identified material contradiction left untreated. Additional design narratives and individual retained page assertions cannot establish operation or causal benefit merely by existing; their unassessed breadth is already stated in the reports.

The long-form limit rests on the complete SRC-1 tools/paper_agent_long.py: section generators receive topic strings and sometimes an unused wiki_insights argument, but their prompts do not interpolate those insights, external-source results or earlier generated sections. For example:

def _generate_methodology(self, topics_str, wiki_insights): prompt = f"""Write a comprehensive Methodology section (1,800-2,200 words) for a research paper on {topics_str}. --- tools/paper_agent_long.py

result = call_local_model("You are writing a technical methodology section.", prompt) return f"## 3. Methodology\n{result}" --- tools/paper_agent_long.py

This is an implementation-level unused dependency, not evidence about generated factual content or exercised harm.

Blockers

none

Limits

• Alternate paper context coverage is incomplete in RT-RTE-wiki-consumption, the memory consumption annotation, and the epistemic bounded conclusion. SRC-1 tools/paper_agent_long.py calls wiki analysis but does not deliver its results into section-generation prompts; those prompts also omit prior generated sections. This differs from ordinary drafting's explicit insight projection. The reports distinguish long form and do not assert that every output consumes retained evidence, so the omission does not undermine their supported file-retention, ordinary read-back, critique-bypass or unestablished-warrant conclusions. Readers must withhold any comparison treating long-form drafting as evidence-conditioned like ordinary drafting, or treating its sections as receiving shared prior-section context. The working-context inventory is partial at this alternate consumer.