SkillLift synthesis
Type: agentic-system-analyses/types/agentic-system-synthesis.md
Bounded synthesis
SkillLift is a builder or improvement plane: it generates and revises skill packages for another agent, and revises the rules used to score those packages. It does not own the benchmark agent's scheduling, tool permissions or memory. This account covers the main repository at commit 599358b4d4c4a27c0e004df8228ab93026600653, with an applicability cutoff of 2026-10-14. SRC-1 supplies executable source, prompt/design intent and attributed README results. No target code, tests, model services or benchmarks were executed. Unpopulated benchmark submodules, the sibling tau2 installation and external hosts/providers leave a complete artifact, partial loop boundary. These distinctions are fixed by the boundary and evidence and Source register.
The strongest supported contribution is an experience-responsive path from recorded benchmark outcomes to durable scoring guidance used by SkillLift itself in later rounds. This is more than exporting a better-looking skill file: the optimizer can change its own subsequent verifier inputs and editing guidance. The unresolved question is whether those changes improve future capacity or outcomes. Source wiring establishes neither exercised benefit nor component causality. See RT-RTE-receipt-revision and MEM-OBJ-receipt-rubrics.
From a task request to candidate experiments
The representative workflow is the shipped WildClaw single-task path. Caller settings select a task, endpoint, run root and bounded search configuration. A runner constructs model clients, an adapter and a store; a Python coordinator owns progression. Planner, editor and verifier roles receive explicit requests rather than a shared benchmark conversation or independent scheduling authority. Model endpoints are inference dependencies, not authenticated frozen weight revisions; provider mutability remains unresolved. See RT-RTE-task-evolution, RT-CMP-llm-client and the connected workflow account.
The coordinator checks task/seed/configuration identity, reconstructs any accepted parent and evaluates an anchor. Invalid anchors return an indeterminate frozen portfolio; an already-terminal unadapted seed returns without ordinary final trials. Otherwise a planner receives the public task, current files, prior directions and scalar outcome history. An editor proposes scoped changes. Staged patch application checks changed and forbidden paths, preserves curated identities and validates skill structure before publishing a candidate tree. These are invariants of the inspected portfolio gate, not semantic safety guarantees for the resulting instructions, and they do not cover the separate category/domain generators. RT-RTE-task-evolution and MEM-OBJ-candidate-record retain the admission and reconstruction evidence.
Mode A refines each branch against a receipt: natural-language binary criteria with signed scoring points. Below the configured threshold and within the iteration budget, criterion hits guide another edit. Code keeps the edit only if the verifier score is nondecreasing; errors or declines retain the previous branch. This inner method spends no benchmark calls, but the outer round still evaluates candidates. A verifier judges visible skill text, not task execution. Parsing checks criterion IDs and booleans; arithmetic computes a conditional score. Supported lexical fallback branches are weaker still and must not be projected onto ordinary model-backed scoring. RT-RTE-branch-refinement, RT-CLM-mode-a and EPI-RTE-verifier-check separate these mechanisms.
Working context is selected explicitly. The editor first attempts the scoped files; more than 20 files, incompatible snapshots or overflow can trigger a model file selector, with mandatory paths retained and optional count/byte limits enforced. Requests carry selected snapshots by value, not references the model can dereference. Planner overflow has a separate core-evidence fallback, followed by a rejecting preflight. Persistent omitted files remain reachable to code, but no host chat-compaction or automatic foreground-memory mechanism is established. See the memory annotations on RT-RTE-branch-refinement and RT-RTE-task-evolution.
Outcome selection, calibration and delivery
The adapter submits valid candidate portfolios to benchmark execution. WildClaw serializes its runner calls under a lock despite coordinator concurrency. Task, configuration, artifact containment and loader-manifest checks associate returned reward with the candidate; SkillsBench task evaluation additionally checks deployment/loading receipts. These protocols depend on accurate external receipt producers and graders. They do not prove instruction compliance, grader correctness or host isolation. The successor requires valid reward improvement, or equal reward with a strictly higher available verifier score. Thus the verifier can affect selection on ties; inner score nondecrease is not a guarantee of task-quality nondegradation. See RT-RTE-task-evolution, RT-RTE-cell-recovery and EPI-CLM-local-safety.
Mode B follows successor selection. It compares the pre-round champion and evaluated candidates using their existing rewards, verifier scores and criterion contrast. At least two usable portfolios and common scores are required. A rubricator proposes revised criteria; structural checks and bounded repair can reject them. Code retains the best tested rank alignment, not guaranteed threshold attainment. The admitted receipt guides later rounds; it does not retroactively choose the current winner. Criteria removal requires bookkeeping and nonempty reasons/references, not authenticated semantic support. Finite-set alignment licenses a local scoring policy, not universal prediction or a causal explanation. See RT-RTE-receipt-revision, EPI-RTE-receipt-check and EPI-CLM-ranking-surrogate.
Evidence reaching that rubricator is caller-dependent. Only an exact SKILL.md key receives full-body protection under a configured evidence budget. Task portfolio conversion preserves prefixed paths, so budgeted packing can truncate or omit nested instruction bodies as auxiliary content. Ordinary task Mode B supplies no such budget and includes all bodies; its separate character preflight can reject an oversized request instead. Neither universal body preservation nor default-path truncation follows. EPI-RTE-evidence-packing, EPI-OBJ-evidence-package and the memory annotation on RT-RTE-receipt-revision preserve this condition.
Logical evaluation recovery reuses valid results, asks adapters to recover started attempts and otherwise retries physical calls. An infrastructure fuse leaves audit markers without permanently closing an indeterminate cell. This is not exactly-once remote execution; later control flow must still reach the cell. Content recovery has a separate defect: a changed, accepted Mode A refinement retains the original candidate record, while resume reapplies its original patch. The reconstructed tree can therefore differ from the evaluated/promoted tree and encounter later hash rejection. This is a conditional implementation risk, not an observed incident. See RT-RTE-cell-recovery and RT-CLM-refinement-replay.
Ordinary completion freezes the chosen portfolio and requests two final logical trials. Reporting uses the maximum valid reward, or null when neither trial is valid; it does not guarantee improvement over the anchor. Delivery consists of the returned frozen-portfolio result and runner-written summary/task records. Later agents receive skill instructions through loaders, not necessarily optimizer history or reasons. Instruction delivery is wired; activation and causal benefit are unobserved. See the workflow account and RT-BAP-skill-delivery.
Retention and alternate training paths
Memory is principally filesystem skill trees, patch recipes, JSON state and full-content checkpoints. Natural-language instructions and rationales have model consumers; symbolic hashes, IDs, points and validity fields have fixed-rule consumers. Failed proposals can persist without promotion. Rejected directions constrain future novelty and enter later planning; admitted receipts enter later verification/refinement. Requests for replay, planning, fusion and testing pull specified retained inputs. These witnesses do not establish a repository-wide pull-only inventory. Receipt diagnosis/removal rationale can reach later models in full serialized receipts, but delivery does not prove reliance or sound criticism. Attempt archives establish repair and audit retention, not general experience retrieval. See MEM-OBJ-search-state, MEM-OBJ-receipt-metadata, MEM-OBJ-revision-attempts and the profile verification.
The SkillsBench category driver uses full completed-round snapshots rather than task patch replay. It selects batch winners and can fuse multiple winners into a new package. Package/name checks precede final-tree publication, but no new benchmark result is required before that publication. Held-out testing is later evaluation, not inherited certification of the fused output. RT-RTE-category-training, MEM-OBJ-category-round and EPI-RTE-category-fusion support this distinct path.
The tau2 domain driver also uses snapshots, but its checkpoint loader lacks a configuration-signature comparison and its update helper does not impose category Mode A's immediate verifier-nondecrease gate. Each cluster's final artifact exports all current skills, and fusion loads all their metadata entries and SKILL.md bodies without consulting the separately saved oracle rank. Lower-ranked skills can therefore enter fusion. The master guide undergoes structural admission, followed by output equality/hash checks at held-out loading; copied input hash labels do not authenticate every upstream rule. Conflict handling, preservation and deduplication are requested, not verified semantic transformations. See RT-RTE-domain-fusion, MEM-OBJ-cluster-checkpoint and MEM-OBJ-master-skill.
Tau2 additionally retains heuristic message-derived behavior summaries, but shared receipt input projects that feedback out and supplies scalar score/pass/rank instead. Saved diagnostics do not establish summary-driven instruction induction or full-trajectory learning. Its wrapper configures external natural-language assertion defaults; unavailable evaluator code prevents treating every reward as inspected deterministic proof. Post-run artifact audit supplies reports and process exit status, not demonstrated optimizer rollback. See MEM-OBJ-behavior-summary, EPI-RTE-tau2-evaluator and EPI-RTE-artifact-audit.
What the improvement evidence establishes
For task-scoped experiments, SkillLift has wired proposal, local surrogate checking, benchmark selection and reusable scoring-policy revision. For later category/domain use, it has wired retained-content delivery but weaker semantic gates on newly fused products. No supplied candidate/trace identifies an accepted ampliative factual claim or its subsequent integration; policy adaptation is the supported account, not certified knowledge production. The README's ablations remain attributed reports rather than inspectable causal comparisons. See the epistemic bounded conclusion and EPI-CLM-ablation-mechanism.
The independent properties must be kept separate, at the receipt-revision boundary and across subsequent rounds/task re-entry:
- Theory-builder condition 1, localized content: wired. Criteria expose individually inspectable/revisable text, IDs and points.
- Condition 2, consumption: wired. Criteria determine verifier inputs and score-dependent decisions.
- Condition 3, content-directed criticism: claimed. Prompts request diagnosis of challenged criteria and evidence-bounded revision; actual blame placement and justified criticism remain unobserved.
- Condition 4, iteration: wired. Selected receipt revisions persist and feed further scoring/rounds; this does not establish the criticism's quality. All four conditions' operation statuses are uninspected.
- Learning as improved capacity for future action is unestablished. Automatic evaluation-fed durable updates and later consumers are wired, satisfying the separate memory notion of trace learning. That positive mechanism does not prove capacity improvement or attribution to criticism of a consumed theory.
- Reflection is wired for scoring policy. Outcome/ranking feedback can update the optimizer's representation of useful criteria, and that representation can change later optimizer behavior. Accuracy and benefit remain unobserved.
- Autonomous proposal, admission and iteration are wired, without per-content human approval on these inspected paths. Autonomous theory building and reflective theory building are not established because the content-directed criticism operation remains unestablished.
- Self-improvement has an improvement-directed pathway in retained receipt policy, not merely external skill output: objective-bearing reward contrast can shape its update, and the policy can govern later verifier/refinement behavior. Exercised self-improvement is not established because no subsequent operation is shown to causally depend on the change. Successful self-improvement additionally lacks independently assessed outcome evidence.
These statuses follow RT-RTE-receipt-revision, RT-BAP-receipt-scoring and MEM-OBJ-receipt-rubrics; they are not a grade or a progression ladder.
A linked run showing particular criteria, diagnosis, revision and consequential later use would change the criticism and exercised-path assessments. Independently assessed subsequent outcomes, with controlled receipt-revision comparisons, would change benefit/attribution conclusions. Persisting the evaluated refined patch/tree would address the replay risk. Semantic fusion checks and authenticated upstream contents would strengthen final-content claims; inspecting external evaluators and host loading would narrow the partial-loop boundary. Neither report verification nor profile verification declares blockers or additional limits, and the supplied reconciliation retains no unresolved conflict. No frozen-source reads were needed for this synthesis.
Limitations
| limitation | affected SRC IDs and record citations | inspected boundary | conclusion prevented | evidence that would resolve it |
|---|---|---|---|---|
| Source wiring, not observed operation or causal comparison | SRC-1; RT-RTE-task-evolution, RT-RTE-receipt-revision, EPI-CLM-ablation-mechanism | Main-repository implementation and attributed README results at the reviewed commit; no supplied runs or experiments | Observed reliability, improved future capacity, exercised/successful self-improvement and component causality | Linked execution artifacts plus independently assessed outcomes and controlled interventions at the claimed comparison level |
| Missing benchmark, evaluator and host/provider implementations | SRC-1; boundary, RT-BAP-skill-delivery, EPI-RTE-tau2-evaluator, RT-CMP-llm-client | Main wrappers/local host launcher; unpopulated benchmark gitlinks, unavailable tau2 sibling and external hosts/providers | End-to-end loading/compliance, grader validity, isolation, exact model-version or parameter-fixity claims | Pinned dependency/host/evaluator source and loading/evaluation traces; provider version/mutability evidence |
| Changed accepted refinements may not survive replay faithfully | SRC-1; RT-CLM-refinement-replay, MEM-OBJ-candidate-record | Task Mode A when refinement changes the tree and that branch is accepted; original-record replay | Resume-stable identity of every evaluated refined successor; not an observed failure claim | Persisted refined patch/tree tied to acceptance and a resumed identity comparison |
| Surrogate and structural checks have narrower licenses than semantic safety | SRC-1; EPI-RTE-verifier-check, EPI-RTE-receipt-check, EPI-CLM-local-safety, EPI-CLM-ranking-surrogate | Visible-text scoring, receipt bookkeeping, nondecreasing inner score and finite candidate-rank alignment | Guaranteed task nondegradation, authenticated diagnosis, universal/out-of-sample preference prediction or threshold convergence | Linked judgments and candidate content, semantic evidence checks and independent held-out outcome comparisons |
| Budgeted evidence coverage depends on exact paths and caller | SRC-1; EPI-RTE-evidence-packing, EPI-OBJ-evidence-package, RT-RTE-receipt-revision | Positive evidence budgets can omit prefixed instruction bodies; ordinary task Mode B omits that budget and can fail character preflight | Complete-body support for every budgeted diagnosis; neither universal preservation nor default truncation | Captured consumer payloads with exact keys/budgets, or packing changes protecting every required instruction body |
| Separate checkpoint/update gates and recovery scope | SRC-1; RT-RTE-cell-recovery, MEM-OBJ-category-round, MEM-OBJ-cluster-checkpoint | Task logical cells, category configuration-compatible snapshots, domain loader without configuration-signature comparison | Exactly-once remote effects, automatic reopening of stopped decisions, universal configuration-complete resume or shared nondecrease guarantees | Crash/resume traces and authenticated external recovery; domain configuration checks and route-specific update comparisons |
| Fused final content is structurally admitted, not semantically certified | SRC-1; EPI-RTE-category-fusion, RT-RTE-domain-fusion, MEM-OBJ-master-skill | Category batch-winner fusion; domain all-current-skill fusion with copied input hash labels and checked output association | Inherited winner performance, verified rule preservation/deduplication, upstream authenticity or fused-guide generalization | Authenticated inputs, linked fused outputs, source-rule/conflict checks and independent evaluation of the fused product |
| Rationale delivery and summary retention do not establish criticism or learning through those parts | SRC-1; MEM-OBJ-receipt-metadata, MEM-OBJ-revision-attempts, MEM-OBJ-behavior-summary, RT-RTE-receipt-revision | Full receipt delivery, within-episode repair, diagnostic archives and summaries excluded from shared receipt input | Actual rationale reliance, reliable content-directed criticism, general archive retrieval or summary/full-trajectory-driven instruction induction | Linked consumer requests/results and diagnosis/revision evidence; inspection of raw-payload readers and other diagnostic consumers |
| Initial control and remaining shipped alternatives lack complete coverage | SRC-1; MEM-OBJ-skill-instructions, RT-RTE-task-evolution, RT-RTE-domain-fusion; profile verification | Selected task/category/domain routes; unresolved seed authorship/admission, auxiliary payloads, raw readers, wider baselines/entry points and validator/injection branches | Complete memory/authority/admission inventory, repository-wide pull-only or no-trace-consumer claims, exhaustive security/validator assessment | Part/consumer-level records covering initial content control and consequential remaining alternatives at the frozen boundary |
| No individuated factual output and acceptance trace | SRC-1; EPI-OBJ-verifier-score, RT-OBJ-portfolio-ref, epistemic bounded conclusion | Generation/checking implementation without linked concrete candidate claim instances | Produced accepted ampliative factual knowledge and post-acceptance integration | Candidate claims linked to generating evidence, semantic acceptance criteria/intended scope and subsequent integration/use |