HippoRAG report verification

Type: agentic-system-analyses/types/agentic-system-verification.md

Verification

The five pinned members support a bounded source-level account. I independently traced the consequential producer-to-consumer paths rather than accepting the analysts' validation statements as claim support. The checkout HEAD matches the registered boundary. No target code, tests, examples, providers or services were executed. The supplied structural report says none; all four correction-answer files are empty. There are no structural findings to route and no correction arguments to adjudicate.

Runtime and retained-state checks

I checked every declared runtime record and its evidence layer, scope and consumer. SRC-1 implementation reads covered construction and manifest configuration, index/delete, OpenIE load/merge/save, retrieval preparation, normal retrieval, graph seeding/PPR, IRCoT and QA in src/hipporag/HippoRAG.py; the manifest helper in src/hipporag/utils/state_utils.py; the store in src/hipporag/embedding_store.py; and the filter in src/hipporag/rerank.py.

  • RT-RTE-index-state, RT-OBJ-index-manifest and RT-CLM-state-compatibility: passage and metadata persistence precedes online extraction; extraction and derived-store/graph writes are separate operations. Manifest equality and preparation's graph-ID/OpenIE-coverage checks support compatibility over the checked fields, not content truth or a whole-index transaction. Deletion consults surviving source mappings before dropping facts/entities and removes graph source contributions. The report correctly withholds crash-recovery and secure-erasure guarantees.
  • RT-RTE-retrieve-answer and RT-OBJ-query-solution: candidate fact strings reach the model filter; returned candidates seed graph search, while empty/error selection permits dense fallback. QA receives a passage prefix, question and static template, not the full graph or metadata. Supplied solutions bypass retrieval. The answer delimiter is parsing, not semantic validation. These distinctions survive the runtime prose and memory annotations.
  • RT-RTE-ircot-retrieval: thoughts become subsequent queries, duplicate passage scores merge by maximum, and a literal marker or step bound stops continuation. Final QA does not directly receive the accumulated thought list. SRC-1 src/hipporag/utils/qa_utils.py also shows passage formatting and prior-thought concatenation. Neither current-call continuation nor marker detection warrants correctness or durable method revision.
  • RT-RTE-offline-extraction: direct reads of both offline extractor wrappers and the core staging branch confirm the vLLM save-then-raise path and the distinct Transformers indexing path. Their parse-error handling does not inherit online batch-error vetoes. RT-RTE-standard-rag is supported by direct index/delete/retrieve inspection in SRC-1 src/hipporag/StandardRAG.py: it admits and ranks passage embeddings without the graph/filter chain.

The five model components were checked separately: RT-CMP-extraction-llm, RT-CMP-qa-llm, RT-CMP-embedding-model, RT-CMP-vllm-offline and RT-CMP-transformers-offline. Construction, dispatch and generation/encoding code support distinct roles, no parameter update by the traced orchestration, and no enforced immutable weight revision. Direct SRC-1 reads included src/hipporag/llm/__init__.py, src/hipporag/llm/openai_gpt.py, src/hipporag/llm/vllm_offline.py, src/hipporag/llm/transformers_offline.py, src/hipporag/embedding_model/OpenAI.py and src/hipporag/embedding_model/NVEmbedV2.py. Remote resolution and injected Python remain appropriately outside those guarantees.

Memory objects and read-back

All eleven memory object declarations were checked against their producing and consuming code, not merely their storage names. MEM-OBJ-passage-text, MEM-OBJ-entity-text and MEM-OBJ-fact-text correctly distinguish original content, normalized entity identities and stringified relations. MEM-OBJ-passage-embeddings, MEM-OBJ-entity-embeddings and MEM-OBJ-fact-embeddings distinguish stored encoder outputs from encoder parameters and from text interpreted by the filter/QA model. SRC-1 src/hipporag/utils/misc_utils.py confirms that triple subjects/objects, not the whole NER list, determine entity nodes. Core synonym construction confirms thresholded vector-based graph admission, not proof of synonymy.

MEM-OBJ-openie-state retains extracted content and passages, while reconstruction discards raw response/error metadata. MEM-OBJ-graph-state retains source-aware ranking structure, not a justification history. MEM-OBJ-chunk-metadata is indexed by content hash; reindexing identical text can replace its provenance metadata. These findings are supported by SRC-1 src/hipporag/HippoRAG.py and src/hipporag/utils/misc_utils.py. Current-source-ID deletion therefore does not imply independent ownership for identical passages from different sources.

MEM-OBJ-response-cache and MEM-RTE-response-reuse match SRC-1 src/hipporag/llm/openai_gpt.py: normalized request identity selects a raw message/metadata replay; store=True bypasses local caching. Core document deletion does not call cache erasure. This is a separate read-back route across inference roles, not an alias for complete retrieval. The inspected SRC-1 tests/integration/_shared.py supplies a concrete cross-instance index consumer, without establishing that this executable specification was run. Memory write-side, read-back and annotations preserve those boundaries.

Epistemic checks

All epistemic object, route and claim declarations, compact ledger entries and their dependencies were checked. EPI-OBJ-passages, EPI-OBJ-openie-triples, EPI-OBJ-normalized-facts and EPI-OBJ-answer-thoughts distinguish source contents, generated relation contents, normalized relations and per-call candidate conclusions. Their Part of: relations denote actual content within the indexing route or query container, not replacement identities.

The online extractor and SRC-1 src/hipporag/utils/llm_utils.py support EPI-RTE-openie-generation, EPI-RTE-triple-check and EPI-RTE-triple-admission. Separate NER/triple calls, list/arity/string conversion and deduplication justify structural admission, not semantic acceptance. EPI-RTE-text-normalization's C++/C# collision follows directly from casefolding and punctuation replacement. Its indeterminate semantic-preservation finding is supported, and it does not claim that original passages are lost.

EPI-RTE-fact-check and EPI-RTE-fact-selection correctly separate model relevance judgment from nearest-string remapping and branch selection. Their Part of: relations to retrieval are supported. Candidate membership is enforced; relevance, exact model intent and truth are not. The ledger's retention, selection and compatibility rows do not silently promote operational use into accepted knowledge.

Direct reads of SRC-1 main.py, src/hipporag/evaluation/qa_eval.py, src/hipporag/evaluation/retrieval_eval.py and src/hipporag/utils/eval_utils.py support EPI-RTE-metric-evaluation: normalized alias agreement/token overlap and exact document-string recall produce reported scores. No inspected consumer turns those scores into a retained claim-acceptance or method-update decision. The normalization contrast is valid as an implementation deduction, not an observed benchmark error. EPI-RTE-host-check is correctly doctrine only and a part of host guidance, not an enforcing server gate.

The source passages support the attribution and bounded comparisons of EPI-CLM-memory-connections, EPI-CLM-grounded-answer and EPI-CLM-call-guidance. SRC-1 README.md, src/hipporag/mcp_server.py and plugins/hipporag/skills/hipporag-memory/SKILL.md supply the claims, not empirical confirmation. Separate online extraction stages and model-backed filtering refute universal one-call/no-call readings. No unsupported observed or causal status was found. The bounded conclusion keeps learning, reflective criticism and self-improvement unestablished rather than inferring their global absence from opaque weights or missing traces.

Workflow, authority and reconciliation

I read the complete SRC-1 MCP server, both plugin skills and launcher. RT-RTE-mcp-tools correctly limits serialization to one service executor and read-only enforcement to mutating-tool registration. The service still initializes state and may write caches. RT-BAP-extraction-prompt is supported by the shipped triple template; RT-BAP-plugin-skills denotes host-interpreted guidance. Neither establishes observed obedience. The launcher specifies mutable Git installation, not a frozen deployed instance.

WF-RTE-folder-indexing follows one connected shipped procedure. Preparation, confirmation, refresh and final reporting belong to the host; the service receives prepared content and caller provenance. Whitespace rejection precedes indexing; counts follow successful library return. Counts do not supply duplicate/per-file or semantic audits. Delete followed by failed replacement has no workflow rollback. The selected route spans several supplied mechanisms rather than being contained in exactly one; its distinct-identity explanation is supported. Its mixed claimed guidance/wired implementation/uninspected operation statuses are appropriate.

The reconciliation accounts for all integration issues and overlaps. Its single supersession identifies the same retained passage content in EPI-OBJ-passages and MEM-OBJ-passage-text and preserves epistemic source-warrant qualifications. OpenIE relation content is not the full OpenIE container; normalized fact content remains a valid cross-representation account alongside separately classified text, vectors and graph. Answer/thought contents are parts of the per-call container. Filter checks/disposition, broad lifecycle routes, wrapper cache reuse, benchmark evaluation and folder procedure are not duplicates. All explicit Part of: relations have supported containment. No erroneous dependent value, missing material split or unresolved analyst disagreement was found. Source doctrine/implementation discrepancies are consistently reported, not silently reconciled into stronger claims. Empty absence sections do not conceal an asserted system-complete negative.

Omission questions and stopping decision

The frozen tracked tree, shipped entry points and producer/consumer dependencies guided additional questions beyond the listed quotations:

  1. Could a shipped caller or host instruction add consequential state admission, maintenance or consumption omitted by an API-only account? Tree/call-site searches and direct driver, reload caller, MCP and skill reads establish that those responsibilities are already covered. They remain inside the whole-system target; opaque host execution is not attributed to HippoRAG.
  2. Could alternate modes overturn universal extraction, graph retrieval, truth-gating or recovery conclusions? Direct offline-wrapper, StandardRAG, supplied-solution, IRCoT and preparation traces confirm the alternatives and qualifications already retained. Configured filter prompts are loaded as static guidance by the inspected reranker, not optimized from current benchmark outcomes. SRC-1 searches included src/hipporag/prompts/ and the shipped configured DSPy JSON.
  3. Could hidden retained raw output or provider/backend behavior defeat the deletion and model-fixity account? The default response wrapper and core deletion establish the narrower cache-survival result. Scoped dispatch/cache searches also found cache implementations or shared wrapper use in other shipped LLM adapters, but the reports do not claim a complete cache inventory or universal erasure semantics. External-store concurrency, provider behavior and fixture provenance remain unresolved, not evidenced absence.

Further omission expansion stopped because the selected challenges have supported dispositions and no identified material contradiction remains. Bulk extraction fixtures, datasets, backend internals and every provider adapter need not be exhaustively inspected to retain this account. Their unresolved scope prevents complete cache/backend comparisons, deployed reliability, exercised host compliance, empirical accuracy and comparative benefit conclusions; the reports already withhold those conclusions. These faithfully scoped uncertainties add no separate publication defect or new limit.

Blockers

none

Limits

none