WikiSkill synthesis
Type: agentic-system-analyses/types/agentic-system-synthesis.md
Bounded synthesis
WikiSkill implements an experience-to-instruction adaptation loop: retain task outcomes, ask a Maintainer for reusable Wiki lessons, ask a separate Proposer for revised instructions, and let software decide whether those instructions replace the incumbent. Its strongest supported contribution is this persistent, evaluator-gated path from experience to later instruction delivery, including trace-fed updates on inspected research paths. Whether the delivered changes improve future capacity remains unestablished. WikiSkill changes text and control state, not the external models' weights (RT-RTE-product-round, MEM-OBJ-research-evidence, RT-BAP-skill-consumption, RT-CMP-host-model, RT-CMP-research-model).
This is a code-grounded analysis of the independent implementation at https://github.com/Stahl-G/WikiSkill, commit da6984caca09d5f81b2336cf3067b59c0c18bb1e, with an applicability cutoff of 2026-10-15. SRC-1 supplies implementation, doctrine and attributed operation; no target execution, tests, provider calls or fresh experiments were performed. The whole-system functional boundary includes product CLI/MCP, native handoffs, feedback comparison, research controllers, role assets and deployment. It does not freeze the host's subagent machinery, model services, arbitrary user evaluators, BriefLoop or the original paper implementation. Whole-system scope is not complete behavioral inspection: additional callable research controllers remain incompletely characterized (boundary and evidence, Source register, report-verification limits).
From a recurring task to recorded experience
The ordinary product is a host-driven controller, not an enclosing agent runtime. The user and foreground coordinator choose examples, criterion, starting skill and budget. WikiSkill fixes workspace conditions, issues requests and reconstructs progress from a hash-linked event journal. The host chooses inference and tools, executes requests and supplies results. CLI and MCP reach the same product functions. Research execution differs: the evolution engine and isolated Spreadsheet study launch Codex through their own execution paths (RT-OBJ-product-journal, RT-RTE-product-round, RT-RTE-research-evolve, RT-RTE-isolated-study).
The representative source-supported workflow is baseline validation → training → maintenance → proposal → candidate validation → retained result. It connects the ordinary request to both persistent lessons and a separately selected successor. Tasks must finish before maintenance; maintenance commits before proposal. In native mode, WikiSkill stages role instructions and payloads, the host creates a fresh child, and the coordinator binds its returned ID, waits and collects a result-file pointer. Executors receive the filtered task and selected skill, not the learning Wiki or saved baseline answers. Fresh context is a host protocol, not attested filesystem isolation: privileged host tools may reach other workspace files. Direct APIs also do not inherit every containment check of native collection (workflow account, RT-RTE-native-handoff).
Task outputs, optional traces and score/feedback rows persist separately from journal integrity metadata. A configured scorer receives the full task, including reference information, and saved output; without one, the caller supplies a finite score. Authorization fingerprints the scorer's direct files, not every imported dependency, and the scorer runs with normal host permissions. Score validation therefore establishes admissible numeric input, not a truthful or valid evaluator. Journal checks establish local consistency against recorded hashes, not protection against a writer changing both content and metadata (MEM-OBJ-experience, RT-RTE-product-round, RT-OBJ-product-journal, RT-CLM-mcp-separation).
From experience to Wiki, then to a candidate
Product maintenance receives completed current-round training records, all current patterns, original feedback, incumbent skill and gate history. It admits named lessons with nonblank content and source IDs drawn from training, feedback or existing patterns. This preserves local lineage, including lesson-to-lesson derivation, without checking that a lesson follows from its sources. Pattern retention precedes skill evaluation. Reusing a name replaces its current value; old journal versions remain. Product-generated Markdown is a view: later authoring contexts consume replayed mappings, not edited page bytes (MEM-OBJ-patterns, MEM-OBJ-feedback).
The Proposer receives the committed Wiki and returns a full nonempty skill or no_action. Instructions request applicability, exclusions, concrete procedures, evidence and counterexamples. Product admission does not semantically validate those features or authenticate computational authorship. No separate human decision approving each pattern or candidate is required. Wiki content supplies evidence and guidance to learning authors; skill text supplies instructions to task executors under existing host policy, not additional permissions (MEM-OBJ-skill, MEM-BAP-wiki-authors, RT-BAP-role-guidance, RT-BAP-skill-consumption).
Research uses a distinct Markdown Wiki, selected execution views and impact records. Default maintenance samples up to five failures and three successes, supplies stdout/event tails and the Wiki snapshot, and stages fuller evidence for optional reads. Default proposal stages up to twelve eligible traces and supplies index/impact summaries. Paper-style views middle-trim long traces; study reference feedback is paged. These are bounded context selectors, not lossless delivery or demonstrated foreground-chat compaction. Automatic bundles coexist with model-requested exact-file reads. Product context has no inspected token cap (MEM-OBJ-research-evidence, MEM-OBJ-research-wiki, memory read-back).
Reasons can survive as original notes, source links, Wiki narratives, purposes, proposal rationale and gate history without reaching every later consumer. Native numeric history omits candidate bodies and notes. Default research proposal gets a shortened impact rendering with fuller files optionally available. Paper mappings retain named SKILL/PURPOSE pairs, but task injection joins only SKILL bodies. Structural patch and trace-read checks constrain editing and evidence access; they do not establish comprehension, semantic fidelity or explanatory criticism. The paper adapters require injected invocation transport; default CLI activation is not established (MEM-OBJ-skill-impact, MEM-OBJ-paper-skills, MEM-RTE-paper-adapters, EPI-RTE-paper-contract).
Selection, recovery and delivery
Admission depends on the operating mode, not a universal improvement rule:
- Numeric product: candidate mean score must improve on the incumbent by strictly more than the configured direction-adjusted threshold. Baseline and candidate reuse validation cases; this is not an independent held-out generalization result (RT-RTE-product-round).
- Feedback-first: ordinary feedback requires more better than worse cases and no regressions. Explicit human requirements allow all ties only when every required check is fulfilled, no case is worse and no regression is reported. Verdicts, evidence and human-origin declarations are caller reports, not authenticated observations. Initialization without an explicit skill can clear the active inherited skill while retaining inherited knowledge (RT-RTE-feedback-gate, MEM-OBJ-feedback, memory annotations).
- Research: the named evolution engine requires higher mean validation accuracy; the isolated one-round Spreadsheet study requires a higher correct count. The latter freezes final artifacts without scheduling a second learning round or subsequent deployment. Its additional tool and OS controls must not be attributed to the ordinary product (RT-RTE-research-evolve, RT-RTE-isolated-study).
Rejection retains the incumbent but generally leaves committed Wiki learning and proposal evidence available. Thus rejecting instructions is not retracting their supporting lessons. Scorer failure preserves execution output, blocks progression and requires coordinator diagnosis and explicit retry; native recovery can reuse that execution and delegation instead of launching another child. Research resume uses distinct outcome, maintenance and proposal checkpoints. These controls preserve comparisons and reduce repeat work on their named paths, not across every shipped caller (workflow recovery, RT-RTE-native-handoff, RT-RTE-research-evolve).
Completion exposes retained skill, Wiki and history; it does not mean improvement occurred. Shipped guidance treats rejection/no-action as valid results and calls for Wiki/report delivery when no skill was retained. Export and installation copy retained bytes; replacement and restoration use explicit flags, backups and hash checks. They neither prove host reload nor erase the originating Wiki. MCP omits privileged approval/install tools, but this separation constrains only a client restricted to that surface, not an agent with arbitrary shell access (workflow account, RT-RTE-skill-install, RT-CLM-mcp-separation).
What the loop establishes
For a bounded task curriculum with a credible evaluator, the implementation supports selecting instructions against that evaluator and retaining experience for later authors. For requirement-led revision, it supports a different objective: reported fulfillment without regression, potentially without score gain. Neither comparison accepts every explanatory claim in a Wiki or skill. Exact-label scoring, structural contracts and snapshot recomputation have narrower licenses: reference agreement, valid edits/read preconditions and internal numerical/hash consistency respectively. Snapshot verification does not rerun tasks or control candidate admission (EPI-CLM-wiki-lessons, EPI-RTE-label-score, EPI-RTE-paper-contract, EPI-RTE-snapshot-check).
The theory-builder findings are separate, not a combined grade:
| Property | Supported finding and unresolved question |
|---|---|
| Condition 1: localized theory content | Afforded by whole skill text and scoped procedural guidance; no inspected actual proposal establishes that condition in operation. |
| Condition 2: content-dependent consumption | Skill delivery/injection is wired; decisions depending on what a theory says remain uninspected. |
| Condition 3: content-directed criticism | Evidence, counterexamples and localized revision afford criticism; actual claim/result/blame articulation and resulting changed reliance remain uninspected. |
| Condition 4: criticism shaping another round | Wiki, accepted skills and impact history have wired later-consumer paths in multi-round routes; iteration of formulated criticism remains uninspected. The isolated study supplies a Maintainer-to-Proposer bridge, not another criticized skill generation. |
These condition-level bounds follow RT-RTE-product-round, RT-RTE-feedback-gate, RT-RTE-research-evolve, RT-RTE-isolated-study and MEM-RTE-paper-adapters. Addressability ranges from full skill/named-pattern replacement to unique local patches; it does not guarantee individually evaluated assumptions.
Learning: automatic trace-fed durable updates with later consumers are wired, but improved future capacity—particularly improvement caused by criticism of a consumed theory—is uninspected. Reflection: selected execution records can update procedural representations, which can reach later instruction paths; realized representation-mediated behavioral change and reflective theory building remain uninspected. Autonomy: computational author roles and software gates do not establish that computation actually performed every internal theory-building operation. These independent limits do not negate the implemented adaptation paths (MEM-OBJ-research-evidence, MEM-RTE-paper-adapters, RT-RTE-product-round).
Self-improvement: within the controller-plus-retained-instructions boundary, over configured rounds and explicitly inherited workspaces, the system implements an improvement-directed self-change pathway toward an independently specified task metric or requirement objective. This is a dispositional architectural finding, not exercised or successful self-improvement: evidence is supplied to update authors, software can change retained instructions, and those instructions enter later delivery channels. Actual evidence-responsive model determination and subsequent behavior causally depending on the changed content remain unobserved. Installation alone does not close those links (RT-RTE-product-round, RT-RTE-feedback-gate, RT-BAP-skill-consumption, RT-RTE-skill-install).
Linked inputs, actual lessons/proposals, evaluator results and later executor traces would alter the assessment of semantic transformation, criticism and exercised change. Independent later-task evaluation would address future-capacity claims; independent variation of Wiki maintenance, criticism or isolation would be needed for component attribution. System changes that semantically check lesson support, deliver retained reasons to the relevant consumer, or enforce host isolation would change the corresponding guarantees only once their paths were inspected. None follows from the present gate or role instructions (epistemic bounded conclusion, RT-RTE-native-handoff, MEM-OBJ-skill-impact).
Limitations
| Limitation | Affected SRC IDs and record citations | Inspected boundary | Conclusion prevented | Evidence that would resolve it |
|---|---|---|---|---|
| Partial research-controller inventory; carried report-verification limit | SRC-1; RT-RTE-research-evolve, RT-RTE-isolated-study, report-verification limits | Named engine/default callbacks and isolated study; verification additionally inspected OfficeQA promotion and Spreadsheet candidate-file resume. LiveMath and SealQA iteration libraries were located but not fully traced. | Complete research-controller enumeration and universal checkpoint, provenance, recovery or acceptance-route comparisons. No deployed caller or operation of these additional libraries is established. | Canonical route records tracing the additional OfficeQA, Spreadsheet, LiveMath and SealQA callers, their persistence, admission and recovery; linked execution evidence for operation. |
| Implementation is not observed activation or improvement | SRC-1; RT-RTE-product-round, RT-RTE-research-evolve, RT-BAP-skill-consumption | Source inspection without executed tasks, providers or experiments; bundled results not independently evaluated | Successful instruction use, improved future capacity, exercised self-improvement or a component-causal benefit | Linked update/consumption traces and objective outcomes; independent later-task evaluation and appropriate intervention contrasts. |
| Model transformations and lesson warrant remain unresolved | SRC-1; MEM-OBJ-patterns, MEM-OBJ-research-wiki, EPI-OBJ-workbook-lesson | Identifier/schema checks, requested consolidation and one available bundled explanatory candidate | Entailment, non-entailing conjecture, semantic acceptance, or attribution of the bundled lesson to a checked production route | Actual linked inputs/outputs, provenance, content-directed tests and evidence-consuming acceptance decisions for the named claims. |
| Evaluator validity and source reports are external or only partly assessed | SRC-1; RT-RTE-product-round, RT-RTE-feedback-gate, EPI-RTE-label-score | Product scorer interface, caller pair reports and one exact-label scorer; other domain evaluators incompletely assessed | Arbitrary-task scoring validity, authentic human intent, truthful pair evidence, generalization or mechanism correctness from gate success | Evaluator/domain validation, authenticated provenance where relevant, inspected domain scoring and independent test cases. |
| Fresh contexts and privileged-operation separation are conditional | SRC-1; RT-RTE-native-handoff, RT-RTE-isolated-study, RT-CLM-mcp-separation | Host-reported product handoffs, weaker direct submissions, separate unexecuted isolated-study policy and MCP-only surface separation | Product-wide sandboxing, actual fresh-context isolation or host-wide prevention of scorer approval/install | Deployed host/tool policies, attested context behavior and tests of alternate submission and privilege paths. |
| Retained reasons need not reach later consumers | SRC-1; MEM-OBJ-skill-impact, MEM-OBJ-research-evidence, MEM-OBJ-paper-skills | Aggregate histories, sampled/trimmed views, optional full-file reads and SKILL-only task injection | Lossless evidence delivery, actual reading/comprehension, preserved rationale in deployed instructions or fully classified session fallback content | Consumer-level context/read traces and linked derivations showing what survived and shaped updates. |
| Opaque inputs, generic caller control and external inference remain unresolved | SRC-1; MEM-OBJ-experience, MEM-OBJ-feedback, RT-CMP-host-model, RT-CMP-research-model | Optional caller traces, arbitrary output payloads, declared origins and externally hosted model parameters | Universal trace origin or form, authenticated human/computational authorship, fixed provider weights or complete memory classifications | Payload/provenance inspection, admission-principal evidence and provider identity/version evidence. |
| Deployment and callable adapter availability do not establish later use | SRC-1; RT-RTE-skill-install, MEM-RTE-paper-adapters | File installation/restoration and injected-transport adapter code | Host reload/behavior, external read-back direction or selector, and default CLI activation of the paper adapters | Actual host discovery/consumption and caller wiring records, plus linked operation traces. |
No unresolved cross-report conflict was carried by the supplied reconciliation. Report verification declares the partial-controller limit above; profile verification declares no additional limits. Its accepted partial classifications do not supply complete-value comparisons (report-verification limits, profile verification). No new source reads or evidence were added for this synthesis.