SkillRL synthesis
Type: agentic-system-analyses/types/agentic-system-synthesis.md
Bounded synthesis
From experience to two behavior-changing products
SkillRL implements automatic experience-fed changes to both a textual skill bank and an agent's numerical policy. The strongest supported contribution is the connected architecture: failure-responsive rules can enter later training prompts, while reward-bearing rollouts feed parameter updates. Whether those changes improve future action remains unestablished. This account uses executable source, declared design and attributed benefit claims from SRC-1, https://github.com/aiming-lab/SkillRL, at commit 4d984d455b9b02f0bad283cbe9ab3d66a7e5b89d, with an applicability cutoff of 2026-10-14. No target execution, provider call, supplied execution trace or controlled comparison supports it. See the boundary and Source register.
The functional boundary is the whole SkillRL builder or improvement plane: offline experience processing, bank generation, selection and prompt delivery, environment interaction, policy fitting, validation, dynamic skill admission and persistence. It includes shipped training drivers and environment managers, but is not an audit of every feature of the bundled training framework. External model weights, services, rollout provenance and the external supervised fine-tuning trainer are not frozen. The product is a configured training run with artifacts and metrics, not a general-purpose deployed assistant.
The reusable skill library contains natural-language principles and mistakes, organized by general and task-specific categories. Symbolic IDs, category keys and ordering control selection and admission; optional dense vectors support ranking. Their roles differ from the numerical actor weights that determine generation. Storing these parts together does not give them the same authority. The bank text supplies model-interpreted directions, not executor permissions. See MEM-OBJ-skill-content, MEM-OBJ-skill-metadata, MEM-OBJ-retrieval-index and RT-BAP-policy-parameters.
Offline production has two material routes. The original ALFWorld generator extracts bounded patterns from supplied memories, requests model-generated skills and overwrites a JSON bank. A newer data pipeline cleans supplied rollout dumps, generates derived experiences, aggregates skills, then inserts generated reasoning around recorded successful actions and emits training pairs. The latter produces data for external fitting; it does not independently replay actions or verify the generated explanations. Aggregation does not preserve explicit individual-trace links in each resulting skill. See RT-RTE-offline-production, MEM-RTE-trace-abstraction and RT-RTE-sft-production.
A representative run: action, fitting, failure feedback and later delivery
The shipped ALFWorld skills driver is representative because it enables both policy training and dynamic bank updates. An operator chooses model, bank, data, environment and resources. The Ray-based runner creates separate training and validation managers, policy workers, reward machinery and a trajectory collector. Code schedules the outer run; the policy chooses action text within episodes; simulator workers execute projected commands. These workers are not independently prompted specialist agents. The workflow account traces this sequence, supported by RT-RTE-environment-rollout and RT-RTE-policy-update.
At reset, the manager selects and caches skills. Selection is not first-call delivery: the initial ALFWorld prompt omits skills, and later delivery requires a noninitial observation with positive history length. Each policy call receives a rebuilt user message containing current observation, a recent history suffix and, on that branch, selected skill text. The episode buffer retains older entries in host memory until reset, but this path exposes no policy-controlled arbitrary-history reload tool. It trims by code rule rather than generating a continuation summary. See RT-RTE-environment-rollout and MEM-OBJ-episode-history.
The default template selector places all dynamic general skills before static ones, so the requested six slots are not a hard cap. Optional embedding selection can instead omit dynamic rules. General rendering drops applicability conditions; mistake rendering drops explicit reasons. Thus a producer's qualification need not reach the action consumer. The alternate experience selector has manager consumers and similarity-based examples, but its budgets are approximate and its skill-bearing branch does not enforce the advertised token limit. See MEM-OBJ-skill-content, RT-BAP-skill-conditioning and MEM-RTE-experience-selection.
The collector submits decoded action text through projection to the simulator. ALFWorld projection flags formatting problems but does not enforce membership in the listed admissible actions; malformed text can still reach execution. Simulator success supplies bounded outcome feedback, not a corrective-action oracle or proof that a selected rule works. The collector stops at all-done or its configured step bound, subject to backend calls returning, and emits active-step rows with episode statistics. Those rows are not individually complete conversations. See RT-RTE-environment-rollout and RT-OBJ-trajectory-batch.
In the selected driver, actor fitting precedes scheduled validation. Reward and advantage computations feed numerical optimization. Nonfinite-gradient checks can skip an update; they are not a quality test or held-out successor-policy veto. Some configurations accept supplied reward scores rather than simulator-derived episode rewards. See RT-RTE-policy-update.
The driver enables dynamic updates while leaving the training-source flag unset, selecting validation-driven admission. A task success rate below 0.4 triggers failure collection, but the low-success labels do not restrict the collected failures to those categories. Validation deduplicates outputs by the first occurrence of a trajectory identifier before reconstruction; this aligns counts without establishing a full failed conversation. The updater receives at most five examples, their last five parsed steps, observation prefixes of 200 characters and existing skill titles—not existing principles. The alternative training-driven route uses periodically selected training rows without that trajectory deduplication and disables validation-driven updates. See RT-RTE-validation-admission and RT-RTE-training-admission.
A distinct remote model proposes new rules. Software parses a JSON array, requires ID/title/principle keys, reassigns IDs and caps proposals before appending them to the training bank and saving a step-labelled file. Requested applicability is not required by the parser. No per-content human approval or comparative candidate-effectiveness test intervenes on this chain. Structural rejection is real, but it does not establish semantic novelty, correct diagnosis or accepted knowledge. The updater's history retains counts and IDs, not originating criticism and evidence. See EPI-RTE-proposal-check and the two admission records.
At a subsequent training reset, the changed bank becomes eligible for selection and later positive-history delivery. Validation writes only the training bank, but its failures can therefore shape subsequent training: separate managers do not establish causal isolation. Actual model dependence on the added rules remains unobserved. Final delivery consists of metrics, actor/optional critic and loader checkpoints, and separately saved skill JSON. Checkpoint resume does not reload the dynamic bank; coupled recovery requires a separately chosen matching bank. Provider-call exceptions can return no proposals, while missing credentials during construction and bank-save failures are not covered by that catch. See the workflow account, RT-RTE-validation-admission and RT-RTE-policy-update.
What checks license—and what they do not
Other paths qualify any uniform account. Online Search selection expects task_specific_skills, while shipped Search banks use query_type_skills; query-type instructions are lost on that online path, although general rules and mistakes remain possible and offline distillation supports the Search key. Search retrieval imports passages with unknown source warrant. Its terminal scorer measures normalized reference agreement over answer tags extracted from combined history, not independently verified world truth or reasoning validity. WebShop can replace observations longer than 13,000 characters with a no-history template, dropping history and selected skills for that invocation. See MEM-CLM-search-category, EPI-RTE-search-import, EPI-RTE-answer-check and report verification.
For the traced dynamic skill routes, outcome evidence triggers generation; it does not verify the candidate rule. Operational admission, retention and use are not an evidence-consuming acceptance decision against a named effectiveness criterion. These paths therefore retain behavior-shaping candidates, not established accepted ampliative knowledge. Without generated instances, the semantic relation between failures and new rules remains indeterminate: requests for new rules do not prove either faithful compression or conjectural novelty. See EPI-RTE-proposal-check and RT-RTE-validation-admission. Unassessed alternatives prevent extending this into a system-wide absence of other checks.
Independent capacity assessments and conditions for change
The localized rule representation is inspectable and editable at whole-principle granularity. On the traced proposal/consumption pathway, the four theory-builder conditions remain separate:
| Condition | Supported assessment |
|---|---|
| 1. Localized formulation | Wired: natural-language principles state guidance in identifiable entries. |
| 2. Content-dependent consumption | Delivery is wired; actual decisions depending on what those principles say are uninspected. |
| 3. Content-directed criticism | Uninspected beyond the bounded prompt/parser account. Title-only hints and failure excerpts do not establish a challenged claim, placement of blame and resulting revision or changed reliance. |
| 4. Criticism-shaped iteration | Repeated additions are wired; keeping a criticism's result so it shapes a next round is uninspected. Counts and IDs do not establish retained criticism. |
These assessments come from EPI-RTE-proposal-check, with content and consumer evidence in MEM-OBJ-skill-content and RT-BAP-skill-conditioning. Criticism of a consumed theory improving future action is not established; neither the necessary criticism nor improved capacity is evidenced. This does not negate the implemented trace-fed update routes.
Reflection is wired at a limited aspect: selected task behavior can update its representation in failure-derived instructions, and representation-mediated prompt operations can affect later behavior. Accuracy and exercised influence remain unobserved. Computational proposal and admission are autonomous within the operator-started route, while launch and configuration remain human contributions. Autonomous theory building is unestablished because criticism and criticism-shaped iteration remain unestablished. See EPI-RTE-proposal-check.
For self-improvement, the bank pathway supports improvement-directed self-change capacity over later training episodes, with benchmark task success as an independently specified objective. Evidence-responsive proposals can change the system's own bank and enter a live advisory channel. The unestablished link is subsequent operation causally depending on that change. Exercised self-improvement and successful improvement therefore remain unestablished. Reward-driven fitting is a separate wired adaptation route, not a substitute witness for skill-specific effects. See RT-RTE-validation-admission, RT-RTE-training-admission, RT-BAP-skill-conditioning and the assessments in EPI-RTE-proposal-check.
For bounded training with automatic guidance accumulation, the architecture supports successive writes and later consumers. For verified transferable skills, faithful trajectory explanations or exact instruction/policy recovery, its evidence is insufficient. A trace linking failure, admitted rule, final prompt and content-dependent later action would establish exercised parts of the bank pathway. A controlled comparison independently varying bank changes while holding policy fitting and other inputs fixed could establish a skill effect; new-task comparisons would be needed for transfer. Full-principle criticism with retained blame/revision and a next-round consumer would alter the theory-builder assessment. Coupled bank/checkpoint restoration and tested recovery would alter the recovery assessment. The README's reuse, recursive-evolution and 10–20% compression/utility claims remain attributed claims rather than measured conclusions: RT-CLM-reusable-patterns, EPI-CLM-recursive-evolution, EPI-CLM-context-efficiency.
Limitations
The accepted reconciliation leaves no unresolved cross-report conflict. Both verification stages have no blockers; their two declared limits are carried below alongside the consequential evidence boundaries. No additional source reads were needed for this synthesis.
| Limitation | Affected SRC IDs and record citations | Inspected boundary | Conclusion prevented | Evidence that would resolve it |
|---|---|---|---|---|
| No execution or causal comparison | SRC-1; RT-RTE-environment-rollout, RT-RTE-policy-update, RT-CLM-reusable-patterns, EPI-CLM-context-efficiency | Frozen source and attributed README claims; no target runs | Observed compliance, improved future capacity, causal gains, transfer and reported compression/utility magnitude | Inspectable runs and measurements; controlled interventions matched to the claimed component and task horizon |
| Initial provenance and exact numerical components unresolved | SRC-1; MEM-OBJ-skill-content, MEM-OBJ-experience-records, RT-CMP-actor-rollout, RT-CMP-skill-updater, RT-CMP-embedding-model, MEM-CMP-retrieval-encoder | Shipped instances, configured paths and opaque/unpinned service or encoder models | Authenticated experience origins, complete lineage and reproducible instantiated model behavior | Producer-linked source histories, immutable weights/service versions and deployment records |
| Proposal semantics and criticism unresolved | SRC-1; EPI-RTE-proposal-check, RT-RTE-validation-admission, RT-RTE-training-admission | Bounded failures, title-only existing-bank input, key checks and counts/IDs history | Generated-instance entailment/novelty, accepted effective fixes, full content-directed criticism and criticism-shaped iteration | Candidate/input pairs, evidence-consuming effectiveness decisions, explicit challenged principles and retained criticism/revision consumed next round |
| Failure reconstructions are limited projections | SRC-1; RT-OBJ-trajectory-batch, RT-RTE-validation-admission, RT-RTE-training-admission | First-occurrence validation outputs or training rows; additional prompt caps | Treating each example as a full episode or its proposed diagnosis as warranted | Full linked state/action/outcome traces and audited reconstruction into updater inputs |
| WebShop final-prompt delivery inventory incomplete; report-verification limit | SRC-1; MEM-OBJ-skill-content, RT-RTE-environment-rollout, MEM-RTE-experience-selection; verification limit | Verified manager fallback replaces observations above 13,000 characters with a no-history template; full inventory remains incomplete | Uniform cross-manager delivery preservation/context eviction, guaranteed inclusion of selected WebShop rules or observed influence | Complete manager-path inventory and final-prompt traces showing selection, fallback and consumption |
| Partial authority inventory omits update-input learning units; profile-verification limit | SRC-1; RT-RTE-validation-admission, RT-RTE-training-admission, RT-RTE-offline-production; verification limit | Failure excerpts at updater and supplied derived experience at original bank generator | Complete unit-level learning-authority comparisons across skill production; supported learning and downstream instruction positives remain valid | Separate accepted inventory units for those inputs and their actual update consumers |
| Delivery and context budgets are pathway-specific | SRC-1; RT-RTE-environment-rollout, MEM-RTE-experience-selection, MEM-CLM-search-category | Initial/history gates, template overflow, embedding omission, approximate budgets and Search schema mismatch | Universal delivery of new rules, uniform category consumption or a strict skill/token budget | Audited final-prompt accounting on each mode and schema-compatible Search selection |
| Offline cleanup, generated reasoning and repairs do not authenticate experience | SRC-1; EPI-RTE-trajectory-cleanup, MEM-RTE-trace-abstraction, RT-RTE-sft-production | Action-only deletion; supplied outcome labels; reasoning insertion; final WebShop mismatch counting; earlier repairs and some parsing/generation stages incompletely inspected | Faithful state-preserving cleanup, verified explanations, uniformly admissible training data or cross-environment semantic equivalence | Complete transformations with original-to-output lineage, state-aware preservation checks and independent execution/effectiveness evidence |
| External SFT fitting not frozen | SRC-1; RT-RTE-sft-production | Pipeline emits pairs; external trainer excluded from evidence | Actual fitted-policy behavior or benefit from emitted pairs | Frozen fitting implementation, configuration, resulting weights and evaluated later operation |
| Callable maintenance is not an automatic learning or rollback workflow | SRC-1; MEM-RTE-bank-maintenance, EPI-RTE-skill-withdrawal, MEM-RTE-experience-save, MEM-RTE-experience-selection | Bounded searches find no shipped removal/saver caller; saver lacks live-index refresh; direct caller identity unresolved | Automatic quality-driven withdrawal, immediate revocation, automatic live experience accumulation or complete write-agency classification | Shipped callers, admission decisions, index refresh and later-consumer traces; explicit cached/disk revocation behavior |
| Legacy cache correspondence not guaranteed | SRC-1; MEM-OBJ-retrieval-index | Cache checks model name, not memory content/order or encoder revision | Reliable vector-to-record correspondence after replacement; no actual mismatch is observed | Content/order/version-aware invalidation and matched cache/record artifacts |
| Recovery and execution guarantees have narrow coverage | SRC-1; RT-RTE-policy-update, RT-RTE-environment-rollout, RT-RTE-validation-admission | FSDP synchronous path; separate bank saves; no checkpoint bank reload; constructor/save failure limits; async synchronous-call incompatibility | Exact coupled recovery, universal backend compatibility, admissible-action enforcement, wall-clock termination or uniform alternate-optimizer guarantees | Coupled state restoration, failure-path handling, backend-specific source traces and execution evidence; separate enforcement/timeout contracts |
| Other evaluators and authority routes incompletely assessed | SRC-1; RT-RTE-environment-rollout, EPI-RTE-search-import, EPI-RTE-answer-check | Detailed ALFWorld outcome and Search scorer paths; external search corpus/service, full WebShop evaluator and other dispatched environments unassessed | System-wide absence of additional checks, uniform evaluator/safety claims, corpus truth or world-truth license from normalized reference scores | Evaluator and source-lineage inspection for included alternatives, with scoped acceptance criteria and independently warranted references |