WikiSkill synthesis
Type: agentic-system-analyses/types/agentic-system-synthesis.md
Bounded synthesis
WikiSkill's strongest supported contribution is a documented learning pathway that preserves experience even when a proposed procedure fails admission. Task traces feed an editable wiki; the wiki and intervention history guide later skill proposals; a validation gate selects active procedures. The paper reports a benchmark benefit for its persistent knowledge-maintenance/access bundle. Whether the described pathway operated as specified, and which retained content caused later gains, remain uninspected. The reported contrast and its treatment boundary are retained in RT-CLM-wiki-benefit.
This account concerns the arXiv:2608.27454v1 text capture arxiv-2608.27454v1-text-2026-10-15, with an analysis cutoff of 2026-10-15. SRC-1 supplies design descriptions, algorithm, prompts and author-reported results, not executable source or an inspectable end-to-end run. The target is a memory/knowledge/context-engineering system, including its skill-improvement plane. Its boundary is complete artifact, partial loop, and its evidence tier is doc-grounded. No target execution was performed. The boundary and evidence and Source register define the included surfaces and excluded dependencies. All mechanisms below are declared designs unless explicitly described as reported results; none is upgraded to verified implementation or observed execution.
From task work to retained explanations
A caller supplies training and validation tasks, a metric and an iteration budget. The outer harness initializes empty skills and wiki, establishes an empty-skill validation baseline and owns subsequent sequencing. It delegates task execution to an Inference Agent, wiki editing to a Wiki Maintainer and single-skill proposals to a Skill Proposer. The harness, rather than either authoring model, owns skill admission and rollback. There is no described per-proposal human approval; caller and initialization control remain unspecified (RT-RTE-validation-gate).
The task agent receives all active skill content in its system prompt. Its effect boundary depends on the task: direct tool-free reasoning for LiveMath, bash-driven spreadsheet editing, interactive simulator actions for ALFWorld, or document/search tools for question answering. A spreadsheet prompt promises directory confinement while advertising arbitrary shell commands; unavailable wrapper code prevents treating that promise as an isolation guarantee. Model names are given, but exact weight snapshots and endpoint fixity are not established. The evolution algorithm changes textual artifacts, not specified model parameters (RT-RTE-task-execution, RT-CMP-model-backends).
The Raw Layer retains training interactions: reasoning, actions, tool outputs and answers. Declared immutability preserves history, not the truth of reasoning inside it. External documents and observations are acquired evidence with unknown source warrant; model-generated answers are separate candidates, not imports merely because they use that evidence (MEM-OBJ-raw, EPI-OBJ-imported-evidence, EPI-OBJ-produced-predictions).
After training, the harness supplies the maintainer with current wiki context and at most eight traces: up to five failures and three successes, each capped at 15,000 characters. The maintainer returns JSON requesting pattern creations or patches, a complete replacement index and an appended iteration summary. The surrounding system applies the edits. Sampling and caps limit delivered evidence; they do not establish deletion of stored traces. Pattern pages are requested to contain causes, exact command evidence and remedies. The index compresses problem/cause/fix information to guide relevance judgments; the log records findings and actions (RT-RTE-wiki-maintenance, MEM-OBJ-patterns, MEM-OBJ-index, MEM-OBJ-log).
These are explanatory candidates, not automatically accepted knowledge. Diagnosis, claim formulation and retention are distinct functions. No complete input/output pair establishes whether a particular generated explanation preserves existing claims, follows by entailment or adds an ampliative conjecture. No separate claim-level acceptance criterion is described for wiki content. It can persist and influence later proposals before any skill passes validation (EPI-RTE-wiki-diagnosis, EPI-RTE-wiki-formulation, EPI-RTE-wiki-retention).
From retained experience to active procedures
The proposer starts with a smaller view than the maintainer: wiki index, intervention history and a summary of all training outcomes, including predictions and ground truth. It chooses detailed pattern and trace reads through read_file, then submits one skill creation, patch or no_action through finish. Wiki-first inspection and at least four trace reads are prompt requirements, not verified enforcement. Automatic initial supply and model-requested detail reads are separate operations; file paths resolve requests rather than demonstrating targeted unsolicited retrieval (RT-RTE-skill-proposal).
A skill contains procedural SKILL.md text and a companion PURPOSE.md recording origin, addressed patterns and evolution history. Retaining reasons does not establish their separate delivery to task inference or synchronized maintenance during patches. Procedural authority comes from model-interpreted system-prompt guidance, not executable enforcement (MEM-OBJ-skill, MEM-OBJ-purpose, RT-BAP-skill-injection).
The harness provisionally applies the candidate and evaluates it on validation tasks against benchmark references using domain-specific scores. Only a mean score strictly exceeding the incumbent best admits the changed configuration. Ties and decreases restore incumbent skills. This is evidence-consuming acceptance for measured procedural use, not acceptance of every explanation, assurance of transfer or proof that any particular clause caused improvement. Candidate instructions already reach validation inference before acceptance. Neutral changes that might enable future gains are rejected by design (EPI-RTE-validation-scoring, EPI-RTE-skill-disposition, RT-RTE-validation-gate).
Crucially, rollback covers skills, not the wiki. The harness retains proposal diffs, scores and decisions after either outcome. Later proposers can use failed interventions without reinstalling them. A faithful audit would establish what was tried and scored, not the causal truth of a diagnosis or the general uselessness of a rejected approach. Accepted instructions become active for later full prompt injection; default inference does not directly receive wiki access, although a wiki-enabled experimental alternative is described (MEM-OBJ-skill-impact, EPI-CLM-objective-audit, EPI-RTE-skill-integration, MEM-BAP-wiki-guidance).
Reported progression and assessment
The representative ALFWorld case makes this retention policy concrete. The source reports rejection of goal-directed-action at Iteration 0, followed by an accepted break-repetition-loop proposal at Iteration 1 informed by retained feedback. New loop evidence leads to a further rule at Iteration 4, whose admission outcome is not established. This simplified narrative is not a full transcript or independent demonstration of causal history dependence. At the outer-loop boundary, delivery means returning skills and wiki after the iteration budget or a best validation score of 1.0, not deployment or human sign-off (WF-CLM-loop-evolution, workflow account).
For bounded benchmark optimization, the architecture separates durable diagnostic memory from reversible procedure admission. The paper reports average performance rising from 48.7% to 63.7% when proposer wiki access is provided with default inference wiki access disabled. Removing that access also removes the maintainer, so the contrast concerns a maintenance/access bundle, not retrieval alone, a pattern format or a particular file. It supports an attributed partial learning result at that comparison level; raw runs and independently inspectable causal evidence are unavailable (RT-CLM-wiki-benefit). For longer operation, full skill injection, unspecified wiki context budgets and the declared lack of automated wiki pruning prevent conclusions about sustained context fit. For explanatory knowledge, performance admission does not establish accepted causal explanations (RT-RTE-wiki-maintenance, EPI-CLM-knowledge-growth).
The independent property assessments remain separate:
- Theory-builder conditions 1–4: localized explanations/procedures (1) and content-guided decisions (2) are claimed. Content-directed criticism naming a challenged claim, result, blame and revision (3) is uninspected. Retained material guiding further rounds (4) is claimed, but criticism-specific iteration is uninspected. Page/file/span addressability does not close these gaps (RT-RTE-wiki-maintenance, RT-RTE-skill-proposal).
- Learning from criticism of a consumed theory: author-reported gains and rule refinement are positive evidence of claimed future capacity, but do not establish that content-directed criticism of a consumed theory caused that improvement. The missing criticism chain is not evidence of its absence (RT-RTE-skill-proposal).
- Reflection and autonomy: the design represents task-agent actions and skill adherence and can return revised instructions to that agent. Exercised two-way dependence remains uninspected. Computational ownership of the internal roles is specified, but reflective and autonomous theory-builder status remain uninspected; role assignment does not establish all four theory conditions or every exercised operation (RT-RTE-wiki-maintenance).
- Self-improvement: over benchmark-evolution rounds, the declared pathway links objective-bearing task evidence to updates, updates to the system's instruction organization and those instructions to later task agents. This supports a claimed dispositional pathway. Subsequent operation causally depending on the changed instructions remains uninspected, preventing exercised self-improvement attribution. Validation improvement alone establishes neither that dependence nor unseen-task success (RT-RTE-validation-gate).
A pinned executable harness and scorer would change the assessment of admission, restoration and isolation. Linked contexts, edits, evaluations and subsequent rollouts would test whether retained content actually shaped updates and later behavior. An explicit claim/result/blame/revision chain would change the criticism assessment. Independently varying history or individual memory components, rather than removing the whole maintenance/access bundle, would permit finer attribution. These are evidence requirements, not additional observations. No source files were read for this synthesis; it uses the accepted member records and their verification judgments.
Limitations
| Limitation | Affected SRC IDs and record citations | Inspected boundary | Conclusion prevented | Evidence that would resolve it |
|---|---|---|---|---|
| Design and reported results do not expose implementation or full-loop operation | SRC-1; RT-RTE-task-execution, RT-RTE-validation-gate, boundary | Frozen v1 text, algorithm and prompts; no executable distribution or run bundle | Verified enforcement, exercised behavioral dependence, crash safety and independently established self-improvement | Pinned harness/tool/scorer source and linked end-to-end execution artifacts |
| Unresolved conflict: necessarily complete maintainer tracker delivery | SRC-1; RT-RTE-validation-gate, MEM-OBJ-skill-impact, EPI-RTE-impact-retention; report-verification limits | §3.1–3.2 broadly describe history/full-wiki contexts; E.2 enumerates index/log/patterns without tracker; E.3 establishes proposer access | Joint claim that the maintainer necessarily receives the complete tracker; proposer access and persistence remain supported | Context-construction implementation or exact maintainer input records |
| Unified diffs versus full rejected-content wording | SRC-1; MEM-OBJ-skill-impact, EPI-RTE-impact-retention; report-verification limits | §3.2.4 diff description and E.3 full-content wording; no serialized history | Complete replay archive or restoration of full rejected files | Serialized histories with base versions and demonstrated reconstruction semantics |
| Rationale delivery and companion maintenance unresolved | SRC-1; MEM-OBJ-purpose, MEM-OBJ-skill; report-verification limits | §3.1, §3.2.1 and E.3 describe creation/injection without separate rationale delivery or synchronized patches | Every task sees current reasons; companion files are consistently patched/restored | Exact injection payloads and companion patch/restore implementation |
| Exact log path and recovery/return particulars unresolved | SRC-1; MEM-OBJ-log, MEM-OBJ-raw, RT-RTE-validation-gate; report-verification limits | §3, A.1 and E leave log filename, validation-trace retention, durable checkpoints, early-return and no_action handling unresolved |
Exact filesystem, full replay and crash-recovery comparisons; documented sequence and skill-only rollback remain supported | Concrete file layout, dispatch/return code, snapshots and retained validation artifacts |
| Benefit attribution no finer than bundled treatment | SRC-1; RT-CLM-wiki-benefit, WF-CLM-loop-evolution | §5.1 maintenance/access contrast and simplified §5.3 narrative | Independent component/file effects, general transfer or causal proof from the case | Inspectable runs and comparisons independently varying the claimed component |
| Explanation warrant and criticism remain unestablished | SRC-1; EPI-RTE-wiki-formulation, EPI-CLM-knowledge-growth, RT-RTE-skill-proposal | Root-cause prompts and reported refinement without full input/output or criticism chain | Accepted ampliative explanations, established entailment or content-directed criticism causing improved capacity | Specific claims and inputs, evidence-consuming claim assessment and linked criticism/revision/later-use records |
| Context, alternative consumers and additional retention coverage remain partial | SRC-1; MEM-BAP-wiki-guidance, RT-RTE-wiki-maintenance, RT-RTE-task-execution, RT-CMP-model-backends; profile-verification | Default role supplies and full injection described; wiki-enabled selector, provider state, initialization, context disposal, ALFWorld window and cross-run reload unresolved | Complete memory/consumer inventory, sustained context-fit, isolation or cross-session persistence guarantees | Context-selection/configuration code, exact payloads, retention/reload contracts and bounded long-run artifacts |