ContextPilot boundary

Type: agentic-system-analyses/types/agentic-system-boundary.md

Boundary and evidence

This run analyses ContextPilot as a memory/knowledge/context-engineering system, including its shipped model-driven QA loop, context and memory tools, finite-state inference controller, benchmark drivers and RL improvement pipeline. Its intended use is document-grounded long-context QA and search with a compact working context. It is not just a memory library embedded in an excluded caller: the repository ships the callers, prompts, tool-admission controller, result writers and training consumers. The boundary kind is whole-system for those ContextPilot responsibilities, not a review of every capability of the bundled verl framework (SRC-1, README.md, infer/scripts/eval_task.sh, train/sh/run_qwen3-8b_longbenchv2.sh).

The target depends directly on model calls. The inference driver constructs a fresh agent for each sample, supplies the document and question, calls its run method, and writes final results and inference traces. The payload builder exposes saved memory/note summaries to subsequent turns; the FSM restricts which tools may be offered, including memory review before completion. These shipped callers and consumers are included, rather than treated as external hosts (SRC-1, infer/src/hf_test_runner.py, infer/src/contextpilot.py, infer/src/contextpilot_fsm.py). A minimum implementation passage establishing the caller is:

last_payload = agent.run(question, max_turns_to_fail=max_turns_to_fail) --- infer/src/hf_test_runner.py

The prompt supplies maintenance instructions as part of the system, not merely explanatory documentation (SRC-1, infer/configs/fsm_plan_bm25_mc_prompt.txt, train/configs/contextpilot_system_prompt.txt):

1. Before finishing, if readNote or loadMemory is available, review relevant saved entries using only keys listed in the current Available Notes/Memories sections. --- infer/configs/fsm_plan_bm25_mc_prompt.txt

The training plane is included because it creates and consumes intermediate context-editing snapshots, not just final answers. Its agent-loop worker assigns descendant terminal rewards to snapshots, and the dedicated estimator consumes these rewards. Checkpoint save/load and validation paths are also within scope (SRC-1, train/verl/experimental/agent_loop/agent_loop.py, train/verl/trainer/contextpilot/adv.py, train/verl/trainer/ppo/ray_trainer.py). The estimator states the division of responsibility:

Partial rollouts and subtree-mean reward assignment happen in the agent-loop worker before this estimator runs. Consequently, each input row already represents one snapshot whose reward is the mean terminal reward of its descendant continuations. --- train/verl/trainer/contextpilot/adv.py

The frozen evidence is solely SRC-1 at the full commit above, with applicability cutoff 2026-10-19, the supplied run date. Overall evidence is code-grounded: source text and shipped instructions were inspected, but no target code, tests, examples, model calls or services were executed. Source inspection establishes implementation and intended contracts, not successful deployment, measured quality or causal benefit. The README's performance claims remain attributed claims, not observed results (SRC-1, README.md).

Repository coverage

Every top-level tracked entry is classified below. Included paths remain available to later source inspection at the same commit; “not yet inspected” does not mean unavailable. Initial inspection traced the inference caller through state consumption and result writing, and the training recipe through tools, snapshot reward assignment, advantage computation and checkpoint recovery.

All source-dependent entries above refer to SRC-1; paths are commit-relative. Coverage does not assert exhaustive inspection of every included function.

External dependencies and evidence limits

• Model services and weights: inference launches vLLM and uses model/tokenizer paths; training requires a supplied SFT checkpoint and GPU environment. OpenAI-compatible judges score selected tasks and open-ended training answers. Optional embedding/reranking paths are present in the inference caller. These external implementations, model weights and live endpoint behavior are not frozen here, preventing conclusions about actual model behavior, service reliability, achieved accuracy or runtime cost (SRC-1, infer/scripts/run_full_pipeline.sh, infer/src/hf_test_runner.py, train/README.md, train/verl/utils/reward_score/statelm_qa.py).
• Retrieval: Elasticsearch is an external service; the included tool code owns document indexing and cleanup, but the Elasticsearch implementation and deployed index contents are outside this source. This prevents claims about production retrieval performance and isolation beyond inspected request construction (SRC-1, train/verl/tools/statelm_tools.py, infer/README.md).
• Benchmark content: LongMemEval's checked-out file is an LFS pointer rather than benchmark records; root documentation also identifies BrowseComp+ as LFS-tracked. NovelQA's private answer/book bundle is explicitly not distributed, InfBench is loaded from Hugging Face, and training data is downloaded separately. No fetch or LFS materialization was performed. Benchmark adapters and scoring/launch code remain included, but unavailable external payloads prevent dataset-content audits and reproduction of benchmark or training outcomes (SRC-1, infer/data/LongMemEval/longmemeval_s_cleaned.json, README.md, infer/scripts/eval_task.sh, train/README.md). BrowseComp+ shard contents were not inspected; their materialization is not asserted.
• Durability and reuse: this boundary includes within-trajectory memory read/write and subsequent payload consumption, inference artifact persistence/resume bookkeeping, and training checkpoint save/reload. It does not infer cross-task autobiographical memory from the word “memory,” nor assume that inference trajectory JSON is reloaded as a later agent's knowledge store. Those conclusions require further tracing, not an exclusion of the shipped driver (SRC-1, infer/src/hf_test_runner.py, infer/src/contextpilot.py, train/verl/tools/statelm_tools.py, train/verl/trainer/ppo/ray_trainer.py).
• Evidence outside this checkout: the linked paper, live demo, upstream StateLM/verl repositories and remote model/dataset pages were not acquired and cannot supply evidence for this run. No implementation claim or experimental conclusion may rely on them (SRC-1, README.md).

Source register