ContextPilot report verification

Type: agentic-system-analyses/types/agentic-system-verification.md

Verification

The reports support a bounded implementation account. No material correction is required. I read all supplied members, their types and the record contract. The four correction-answer inputs are empty; the artifact check reports no structural findings. I independently inspected frozen source rather than treating the analysts' validation statements as evidence of correctness. Git HEAD matches the boundary, and the source checkout is clean. No target code, tests, examples, model providers or services were executed.

Record and evidence dispositions

Runtime and authority. I checked all runtime declarations: the RT-CMP-actor-model, RT-CMP-judge-model, RT-CMP-compression-model and RT-CMP-embedding-provider; RT-OBJ-full-history and RT-OBJ-context-masks; RT-RTE-document-qa, RT-RTE-base-inference, RT-RTE-context-edit, RT-RTE-training-rollout and RT-RTE-actor-update; the RT-CLM-fsm-controls; and both RT-BAP-context-policy and RT-BAP-reward-actor authority paths. Their source anchors support the retained distinctions between model proposals, Python effects, text delivery, numerical update authority and unobserved behavior. Endpoint configuration does not pin weights or demonstrate provider compliance. Source-wired updates are not evidence of beneficial learning or exercised self-improvement.

I traced allowed-tool construction through dispatch, not just schema offering. The full-schema fallback is real, and the base caller checks configured names rather than FSM state. The runner supports both classes; the task script selects FSM. Thus the reports correctly qualify review ordering rather than asserting a universal guarantee. Mandatory-review API recovery invokes a store read and advances the protocol without demonstrating relevant model checking. Token-window pruning can exhaust eligible turns above its target. The answer extractor can return assistant prose without successful finish. These findings agree with the workflow account and annotation, including its distinction between protocol completion and artifact delivery. Direct implementation anchors: SRC-1, infer/src/contextpilot_fsm.py, infer/src/contextpilot.py, infer/src/hf_test_runner.py, infer/scripts/eval_task.sh.

Memory. I checked the four distinct stores—MEM-OBJ-notes, MEM-OBJ-simple-notes, MEM-OBJ-memories, and MEM-OBJ-training-notes—plus MEM-OBJ-memory-indexes, MEM-OBJ-trajectory-snapshots, and the MEM-RTE-inference-stores, MEM-RTE-training-stores and MEM-RTE-training-context routes. I also checked every memory annotation against its supplied referent. Keyed mutation is not source authentication. All-summary inventory push differs from requested body delivery; related-memory expansion returns neighbor displays rather than full neighbor bodies or vectors. Simple-note update semantics genuinely differ between inference and training. Merge removes rich fields and does not immediately reindex the result. Restore implementation does not establish default-schema availability. Direct implementation anchors: SRC-1, infer/src/contextpilot.py, infer/tools/context-shaper_tools.json, train/verl/tools/statelm_tools.py, train/verl/tools/contextpilot_memory.py, train/verl/experimental/agent_loop/statelm_agent_loop.py.

I traced synthetic cleanup beyond its producer: suffix isolation and response masks feed non-branchable snapshots; descendant terminal matching supplies credit; missing descendants reject rows; the estimator supplies masked advantages; PPO calls the actor worker. The PPO consumer handles forced-action rollout log-probabilities without discarding those rows merely because their action was synthetic. Conditioning-only cleanup remains a distinct alternative. Branch stores and edit state are copied, while document backend identity can be shared. Checkpoint loading is numerical recovery, not proof of production deployment or semantic trace reuse. These findings preserve the runtime and memory accounts rather than contradicting them. Direct anchors: SRC-1, train/verl/experimental/agent_loop/agent_loop.py, train/verl/experimental/agent_loop/statelm_agent_loop.py, train/verl/trainer/contextpilot/adv.py, train/verl/trainer/ppo/ray_trainer.py, train/sh/run_qwen3-8b_longbenchv2.sh.

Epistemic overlay. I checked EPI-OBJ-document-evidence, EPI-OBJ-memory-content, EPI-OBJ-answer-evaluation, EPI-RTE-memory-merge, all four evaluator records, both claim records and every authority-ledger function. The EPI-RTE-choice-evaluation supports heuristic label agreement, including the ambiguous-answer witness. EPI-RTE-longmemeval-evaluation and EPI-RTE-browsecomp-evaluation have materially different parse-failure denominators. EPI-RTE-terminal-evaluation feeds training, unlike the inspected offline report consumers. Its MCQ branch and open-ended exact-match fallback qualify the evaluator's domain. Direct anchor: SRC-1, infer/src/hf_score_fns.py, train/verl/utils/reward_score/statelm_qa.py.

The context-edit witnesses follow the actual handlers: a selected substring can lose its preceding negation; zero-rate compression can replace content with an empty body and a success wrapper. This establishes lack of a semantic-fidelity check, not that every output introduces a new claim. Manual summaries and remote compression appropriately remain semantically indeterminate. The ledger's bounded lack of post-acceptance integration does not deny implemented retention or operational use. The EPI-CLM-context-management is supported as declared intent and partial implementation; the EPI-CLM-performance-results remains attributed reported operation, not verified superiority or causal benefit. Direct anchors: SRC-1, infer/src/contextpilot.py, README.md, infer/configs/fsm_plan_bm25_mc_prompt.txt, train/configs/contextpilot_system_prompt.txt.

I checked every explicit Part of: relation. Imported evidence is a part of inference history; inference store operations are a subroute of FSM QA with base-handler reuse; training stores and edits belong to the training rollout; terminal scoring belongs to actor-update admission; optional merge is a configurable inference subroute rather than a default operation. Cross-kind containment is explained by those implementation boundaries. No relation requires replacing the container's status with the part's status.

The reconciliation correctly retains the combined epistemic memory-content account alongside the richer and simple store parts. Training arrays are not inference payload snapshots, and an embedding producer is not its cached vectors. There are no supersessions to validate and no unresolved-conflict paragraphs. I independently checked its dispositions of every memory integration issue and the apparent merge/restore and update-rule differences; none is an unresolved contradiction. Workflow declares no competing records. No evidenced-absence record is declared in these reports, and their scoped negative findings do not license a system-complete absence claim.

Omission questions were selected from the tracked tree, shipped callers, registries and producer/consumer dependencies: whether alternate callers invalidate review guarantees; whether saved artifacts reload semantic memory; whether code-authored cleanup reaches training; and whether unmodeled evaluator paths change warrant or force. Inspection of caller selection, payload/render builders, stores, synthetic-action consumers, scorer dispatch and checkpoints supports the emitted central conclusions. The additional inference no-tool recovery selector below makes the recovery inventory incomplete, but does not overturn those conclusions. Further omission expansion stopped because these consequential questions have dispositions, no identified material contradiction remains, and the reports already withhold production-adoption, exhaustive optional-retrieval and exercised-benefit conclusions. Available optimizer/export/service details are not treated as inaccessible or absent; no complete end-to-end deployment or retrieval-quality comparison is warranted.

Blockers

none

Limits

  • The recovery inventory is selective. RT-RTE-document-qa, RT-RTE-context-edit, EPI-OBJ-document-evidence and the workflow account do not detail the FSM's no-tool S3 recovery that automatically reads the highest-ranked search candidate. Code can therefore select evidence delivery as well as perform the reported mandatory store review and cleanup. SRC-1, infer/src/contextpilot_fsm.py establishes this additional branch. The model-directed ordinary workflow and conditional control findings remain supported; withhold an exhaustive comparison of inference recovery selectors or any inference that every chunk read was model-selected.

if (self._fsm_state == FSMState.S3_SEARCHED and self._last_retrieved_chunks and self._no_tool_retries >= max(1, self.max_no_tool_retries - 1)): first_id = self._last_retrieved_chunks[0].get("chunk_id") --- infer/src/contextpilot_fsm.py

result = self._execute_tool("readChunk", params) --- infer/src/contextpilot_fsm.py