ContextPilot synthesis
Type: agentic-system-analyses/types/agentic-system-synthesis.md
Bounded synthesis
From a document question to selected evidence
ContextPilot is a long-document question-answering system in which a model selects retrieval, finding-retention, context-editing and answer-submission operations. Software controls their execution. A separate reinforcement-learning plane trains the operation-selecting policy. This account covers the shipped inference, memory, evaluation and training paths in Tencent/ContextPilot at commit 782cbb6611fb610c4cf6fafda6022b7e89cae191, with an analysis cutoff of 2026-10-13. SRC-1 supplies implementation, shipped design instructions and attributed performance reports, but no observed execution or causal experiment. External services, weights and unavailable dataset payloads are not frozen evidence. The whole-system boundary is functional coverage, not an exhaustive audit of every bundled backend or configuration (boundary, Source register).
The representative shipped workflow starts when an evaluation runner constructs a fresh agent for one sample. The launcher selects the finite-state-machine (FSM) variant. The runner supplies document, question, endpoint, instructions and tool configuration; the reference answer normally goes to later logging/evaluation, not to the answering policy. Resume skips previously recorded samples. Retry-with-hint can give a fresh agent its prior wrong answer and attributed incorrectness, but does not reload the old finding store (RT-RTE-fsm-inference, workflow).
The normal FSM progression is document analysis, planning, indexing, search, another plan, chunk reads, finding writes, cleanup, saved-content review and finish. Planning is the same model submitting strategy text, not delegation to another planner. Local tools receive parsed arguments and state; the external policy receives rendered messages rather than the complete Python store. Keyword search returns ranked chunk IDs and previews; reads supply selected passages. Optional embeddings and reranking broaden the implementation's retrieval capabilities, but are not default inference grants. Relevance determines exposure, not source truth or full-document coverage. Dataset adapters can also flatten document boundaries or remove metadata, so chunk addressing does not guarantee complete source lineage (RT-RTE-fsm-inference, EPI-RTE-document-selection).
Retain findings before removing bulky history
ContextPilot separates structured memories from simple notes, both held in task-local dictionaries outside the rendered history. The model authors their contents; software admits writes and updates through key, mode and argument checks without per-finding human approval or independent entailment validation. Structured memories can carry entities, episodes, relations and provenance, but these remain supplied content rather than authenticated warrants (MEM-OBJ-notes, MEM-OBJ-simple-notes, MEM-RTE-findings-write).
Every subsequent prompt advertises all current keys and summaries, without query ranking or a separate summary-list token budget. A requested key read returns full selected content. Structured reads additionally push graph-ranked neighbor projections: summaries, episodes and matching reasons, not all neighbors' full content. The graph uses normalized entity/time/facet/relation overlap and optional embedding similarity. Its reasons explain matching, not why a remembered proposition is true. Arbitrary prose may remain readable while failing to supply parseable graph features (MEM-RTE-findings-delivery, MEM-OBJ-mem-edges, MEM-OBJ-mem-embeddings).
The normal FSM makes finding writes available after a chunk-read count threshold, then requires cleanup of tracked messages and review of applicable saved-content types. One successful reader call per applicable type satisfies review; it need not cover every key or establish incorporation into the answer. After a provider failure during mandatory review, the runtime can read the latest key itself and advance the protocol. These are evidence-access and progression mechanisms, not independent factual acceptance (workflow, RT-RTE-fsm-inference, EPI-RTE-findings-retention).
Cleanup edits a shared history view rather than isolating a subtask in another agent's context. Deletion stubs, selected spans, model-supplied summaries and compressor output replace what later calls receive while originals remain host-reachable. The implemented inference restore action is not default-granted; automatic eviction does not itself confer restore eligibility. Compactness is not semantic preservation: a source-derived substring countercase removes a denial while retaining its embedded assertion, and compressor failure can fall back to sampled words. Neither example is an observed false answer. Inference eviction can also exhaust eligible history while still exceeding the requested window; training instead terminates when its budget reclamation cannot make sufficient progress (RT-RTE-context-control, EPI-RTE-context-replacement, RT-RTE-training-rollout).
Completion, recording and evaluation are different decisions
The FSM rejects wrong-state actions before normal execution, but this is state-local protocol enforcement. Empty state/tool matches fall back to the configured full set; ordinary inference, direct trusted executor calls and training do not inherit the same schedule. Repeated empty searches, missing tool calls and errors can force best-effort exits or termination. Thus neither universal sequencing nor a successful grounded finish follows from the normal progression (RT-CLM-fsm-gating, RT-RTE-ordinary-inference).
Successful finish stops the loop without an independent answer-truth veto. Caller-side delivery is artifact recording, which can follow an inference exception. The runner's logger selects the latest truthy tool-result final_answer, without checking the tool name, or records No final answer found.. A different helper can fall back to assistant prose for trajectory snapshots; that is not the benchmark prediction handoff (EPI-OBJ-answer-content, workflow).
Offline evaluation judges saved predictions against task-specific references after emission. Letter heuristics and external semantic/rubric judges have different domains and parsing consequences: LongMemEval can count unparseable output as false, while BrowseComp+ excludes parse-error judgments from its denominator. These labels control files and aggregates, not pre-answer admission or acceptance of stored claims. Operational storage, organization and retrieval therefore do not establish post-acceptance integration of warranted claims. No produced accepted ampliative claim is established; this does not prove that the model never derives or conjectures (EPI-RTE-offline-evaluation, EPI-RTE-findings-retention).
What the training path adds
The strongest supported improvement contribution is a connected, trace-responsive parameter-adaptation pathway. Training creates context-management snapshots, ranks branch opportunities by context-size change and entropy sensitivity, clones pre-action state and scores terminal continuations. Snapshot rewards average terminal descendants by ancestry; rows lacking terminal descendants are dropped. Query-relative advantages drive actor optimization, updated tensors enter rollout serving, and configured checkpoints support training reload. This closes an implementation path from experience to retained decision machinery and a later consumer, rather than merely saving logs (RT-RTE-branch-credit, RT-RTE-policy-update, RT-BAP-rollout-weights).
The objective is reference-answer correctness with format/tool/budget penalties, not a criterion invented by each update. Multiple-choice grading and an external open-answer judge supply distinct answer oracles. Terminal-lineage and finite-gradient checks admit rows and updates; periodic validation is not a mandatory promote-only-if-better gate. Descendant comparisons do not independently vary one context edit, so they do not isolate its causal contribution. No supplied run demonstrates beneficial learning, particular update-dependent later behavior or checkpoint adoption by a separate inference deployment (EPI-RTE-answer-reward, RT-RTE-policy-update).
Automatic history-derived replacements retained for later calls also qualify as trace-fed memory updates, independently of parameter training. That write-route finding does not establish improved capacity. Separately, the accepted runtime does not establish theory-builder conditions 1–4: localized shipped guidance exists, but its status as a formulated theory is not assessed; delivery does not establish decisions depending on its theoretical content; content-directed criticism is not established; and retention of such criticism's result shaping another round is not established. Scores selecting updates are not stated criticisms assigning blame to a theory, test or auxiliary assumption. Criticism of a consumed theory improving future capacity consequently remains unestablished. Reflection and autonomous theory-building are independently unestablished, not refuted by the absence of traces. Computational tool and optimizer decisions do not by themselves establish either property (RT-OBJ-system-prompt, RT-RTE-policy-update, EPI-RTE-context-replacement).
At the included training boundary, the source supports a dormant improvement-directed self-change capacity: objective-bearing evidence is wired to update determination, parameters are the system's own decision organization, and synchronization supplies operative authority for later generations. Exercised self-improvement remains unestablished because no subsequent operation is shown causally depending on a particular update. Successful improvement requires additional outcome evidence. The README's compact-context performance gains remain attributed reports, not reproduced results (RT-RTE-policy-update, EPI-CLM-performance-gains).
For bounded document QA, the architectural contribution is retaining selected findings while deliberately reducing the evidence window and enforcing parts of the access/review sequence. For durable multi-session knowledge, warranted claim acceptance, or action-specific training efficacy, the evidence supports a narrower conclusion. Linked traces of writes, rendered inputs, completion and scoring would establish exercised routes; update-linked later behavior and controlled comparisons would change the learning assessment. A semantic acceptance gate, budget enforcement covering injected summaries, or a shipped store-reload consumer would change the corresponding architectural assessment. None is inferred from the present implementation (MEM-RTE-findings-delivery, RT-RTE-context-control, RT-RTE-policy-update).
Limitations
| Limitation | Affected SRC IDs and record citations | Inspected boundary | Conclusion prevented | Evidence that would resolve it |
|---|---|---|---|---|
| No execution or controlled comparison is registered. | SRC-1; RT-RTE-fsm-inference, RT-RTE-policy-update, EPI-CLM-performance-gains | Shipped inference, evaluator and training source; no target execution. | Observed activation, reliable recovery, improved future capacity, exercised self-improvement, reproduced gains and single-edit causal benefit. | Provenance-linked execution and update-dependent later-operation traces; controlled outcome comparisons at the claimed treatment granularity. |
| Dataset payloads, weights and deployed service internals are not uniformly frozen. | SRC-1; boundary, RT-CMP-vllm-client, RT-CMP-judge-answer, MEM-CMP-embedding-provider, MEM-CMP-reranker-endpoint, RT-CMP-llmlingua2 | Repository adapters/configurations; external dependencies and missing task payloads. | Exact evaluation reconstruction, source/reference authentication, exact deployed policy identity, provider retention and transformation guarantees. | Frozen datasets, model digests, endpoint implementation/configuration and relevant service traces. |
| Finding stores have task-local consumers; generic initialization provenance and separate inference checkpoint adoption remain unresolved. | SRC-1; MEM-OBJ-notes, MEM-RTE-findings-delivery, RT-RTE-policy-update | Fresh-sample construction, in-task rendering, branch copying and configured training reload. | Durable cross-session note reuse, complete initial lineage/admission classification and deployed inference use of trained weights. | A shipped, linked persistence/reload path with initialization control and deployment provenance. |
| Successful structural checks do not validate finding or replacement semantics. | SRC-1; EPI-RTE-findings-retention, EPI-RTE-context-replacement | Inspected write, edit, review and renderer paths. | Universal semantic preservation, accepted warranted findings, exhaustive review or guaranteed grounded answers. | Candidate/source comparisons and evidence-consuming semantic acceptance decisions with traced consequences. |
| Alternate grants, generic callers and configuration sequences were not exhaustively audited. Optional merge lacks established shipped invocation and drops structured metadata; training note updates and graph parsing have local differences. | SRC-1; RT-CLM-fsm-gating, MEM-RTE-memory-merge, MEM-OBJ-simple-notes, MEM-OBJ-mem-edges, MEM-OBJ-chunk-embeddings | Default grants plus selected alternate implementations, schemas and consumers. | Universal sequencing, schema/store parity, complete graph coverage, arbitrary reindex coherence or lossless merge. | Caller/grant traces, branch-specific schema and lifecycle inspection, and coherence evidence for the claimed sequences. |
| The partial memory profile omits a separate learning-authority entry for retained history at its summary/span/fallback update consumer. | SRC-1; RT-OBJ-full-history, RT-RTE-context-control, EPI-RTE-context-replacement; profile-verification limit | Accepted records establish experience feeding retained replacement context and later QA calls; the profile separately lists training's learning authority. | Complete route-level comparison of learning authority across compaction and parameter training; positive compaction trace-learning and training findings remain valid. | A profile revision separately enumerating the compaction update consumer and its authority from these records. |
Report verification declares no remaining limits or blockers; profile verification declares the carried omission above. No current unresolved reconciliation conflict was supplied. This synthesis used only supplied members and added no source evidence (report verification, profile verification).