ModularRSI synthesis
Type: agentic-system-analyses/types/agentic-system-synthesis.md
Bounded synthesis
An improvement plane with its own downstream solver
ModularRSI wires execution experience to changes in its own executable harness. Its strongest supported contribution is a complete implementation path from selected task evidence through isolated editing and admission to a generation that later solver tasks can consume. This establishes improvement-directed self-change capacity, not that a particular edit improved future capacity or caused a later success. The product is a Python strategy library, not updated foundation-model weights. See RT-RTE-generation-change and RT-OBJ-module-library.
The evidence is SRC-1, the IQuestLab/ModularRSI repository frozen at b5c72c36b0d08ff93f00ee202a8fbdebe849dfb9, with an applicability cutoff of 2026-10-18. The whole-system functional boundary includes the improvement drivers, their retained state, candidate gates, solver, and supporting Harbor execution paths. It is not an assessment of Harbor as a separate product. The accepted reports are code-grounded accounts of implementation and shipped instructions. No target code, tests or model calls were executed, and individual bundled evaluation trials were not inspected. Available sanitized evaluation artifacts are not a full training history. External provider internals and unfrozen task contents remain outside the evidence; available but unread variants and trials are not access failures. See the boundary and Source register.
The README reports accuracy and transfer gains, including Terminal-Bench accuracy moving from 47.57% to 52.43% for its stated model. Those numbers remain attributed source reports, not independent measurements or component-effect estimates from this analysis: RT-CLM-harness-improvement.
From configured support tasks to proposed interventions
An operator supplies model access, support tasks, resource settings and a module lock, then starts evolution. Phase0 hands this bounded curriculum to the online driver. The driver snapshots the installed library, initializes or resumes generation state, and launches task groups using a captured generation. Initial dynamic composition selects five slots—agent loop, observation, context management, tools and verification—and an optional helper subset. Later rolls can replay the first bundle. The dynamic composer judges task text against filtered module descriptions without verifier rewards; software validates the returned names. Descriptions are model-interpreted capability claims, whereas the chosen implementations are executable policy. Static, cached and forced selection are distinct alternatives. See RT-RTE-bundle-selection and RT-BAP-description-selection.
The selected loop owns solver progression. In the baseline, model output supplies commands, software parses them, tools execute terminal actions or helper callbacks, observations return to chat, and the selected verification strategy decides when interaction ends. The baseline's repeated model completion signals are not independent correctness checks. Harbor subsequently runs the task-provided verifier and returns reward or error. That reward supplies the reference outcome used by evolution; its adequacy remains bounded by the external task tests. See RT-RTE-solver-task.
Completed groups trigger background reflection while solver launches continue. A window can therefore contain evidence from earlier generations; shared task identity does not create a controlled experiment. Investigators receive selected summaries, permitted trajectories and staged code in separate sessions. The requested diagnosis compares passing and failing attempts where available and asks whether a concrete module intervention would plausibly change the outcome. This is an ampliative causal conjecture: the proposed intervention was not applied in those unchanged-code comparisons. The implementation establishes that the question is asked, not that actual generated explanations are sound. See EPI-OBJ-contrast-finding and RT-RTE-generation-change.
Findings and proposals retain those reasons across windows. Model responsibility, intervention-equivalence and successor-coverage judgments govern organization of proposed work; symbolic dispositions, support counts, attempts and leases govern eligibility, priority and recovery. These are not acceptance of the causal diagnosis as true. Software can select still-open proposals later, but a retryable unattached finding is merely nonterminal: the inspected online caller does not schedule its resupply. Historical passing-trial pointers can reconstruct earlier contrast evidence if the referent survives and still passes. See MEM-RTE-evidence-brief, EPI-RTE-finding-disposition and MEM-RTE-history-contrast.
Isolated builds, bounded checks and delivery
The representative workflow is one two-lane evolution window, not a reconstructed successful training episode. Software selects up to one incumbent and one novelty proposal; a lane may remain empty. Each editor receives only its assigned proposal, causal hypothesis and supporting finding displays in an independent staging tree from the same parent. Missing evidence is disclosed, and each finding display is capped at 6000 characters. Full investigator histories do not merge into editor context. See the workflow account and MEM-RTE-evidence-brief.
Candidate admission checks committed edits, smoke/static contracts, action shape and niche, model review, required variant identification, and forced activation/crash sanity. These stages have different licenses. A substantive structured reviewer acceptance is an evidence-consuming, fallible judgment that the executable proposal is worth trying in the bounded experiment, considering trajectory-linked mechanism and task-specific overfit. An explicit rejection can veto. However, reviewer timeout or exception passes as skipped; missing verdicts can initially pass before breaker handling. Sanity-crash repair can subsequently alter reviewed bytes without renewed review/action/niche stages. There is no invariant that a reviewer approved every promoted final tree. See EPI-RTE-review-decision and RT-RTE-generation-change.
Forced sanity establishes required activation and checks classified code breaks. Task failure or infrastructure timeout alone need not reject a candidate, parent reward is reference logging rather than an admission threshold, and the ordinary-routing probe never gates. Passing therefore means neither reward superiority nor likely ordinary selection. A reviewer-classified implementation defect can instead trigger one fresh build from the parent with a repair brief; rejection of the direction does not earn that branch. Preparation failure and started work have different lease/attempt consequences. These are implemented criticism-delivery and recovery mechanisms, not observations of correct criticism or successful recovery. See RT-RTE-generation-change and the workflow account.
Passing lanes can become numbered generations. If both pass, the default order promotes the incumbent first and rebases the second; conflicts reject it, while successful rebases rerun tree-dependent checks. Filesystem promotion and archive synchronization are not one transaction: sync failure leaves the generation standing. Delivery means the driver receives the new generation path and uses it for later launches, not that already-running tasks switch implementations. Evaluation-only invocation loads a selected frozen library without running this improvement loop. See RT-RTE-generation-change and RT-RTE-solver-task.
The retained library has direct Python authority. Discovery imports eligible files before ordinary variant selection, and selected factories and methods subsequently control solver behavior. Editor path and module-lock guards mediate supported file actions; they do not sandbox imported candidate Python. Task-environment isolation does not establish isolation of local candidate imports. See RT-BAP-module-execution.
What persists and what later reliance means
The principal durable behavior artifact is executable code. Findings and causal hypotheses supply upstream reference and editing guidance; eligibility metadata supplies fixed-rule routing, ranking and vetoes; descriptions inform composition. Historical reasons survive separately in backlog state and need not accompany code into solver context. Journals and manifests must not be conflated with those live consumers. The editor-memory formatter exists, but the searched shipped callers establish no compact-history invocation or evolution-log injection into editors. The boundary's ambiguous description of editor use therefore supports no joint inference of automatic journal delivery. See MEM-OBJ-proposal-record, MEM-BAP-backlog-control and MEM-ABS-editor-readback.
Outcome confirmation provides another real adaptation route. It computes candidate-versus-baseline pass-rate differences on recorded shared tasks and can mark regressors superseded, affecting ordinary later dynamic selection. Defaults require at least three shared tasks and a deficit exceeding 0.34. This arithmetic does not independently vary components or establish transfer. Retirement is best-effort selection withdrawal, not file deletion or universal revocation of static/cache/forced paths. Confirmation consumes task/bundle/reward tuples; their granularity does not establish the richer linked-trajectory source used by diagnosis. See EPI-RTE-paired-confirmation and MEM-OBJ-archive-entry.
Within a solver task, compression is also a retained-context route: budget/overflow rules trigger model summary, questions and answers that reach a later invocation. Tail message pairs can be removed before summarization, so neither summary labels nor saved trajectory references guarantee complete evidence or automatic reload. Installed and released baseline fallback behavior differs. Observation clipping and bounded scrollback impose further limits; editor file/range reads offer a separate request-driven reload capability. Optional solver skill-catalog injection remains an incomplete instruction-discovery inventory, not an established learned-memory route. See RT-RTE-context-handoff, the memory annotations and the report-verification limit.
Under the memory comparison's write-route criterion, diagnosis-to-code, automatic continuation production and outcome-fed archive revision qualify as wired trace learning. That term identifies automatic trace-fed retention with a later behavior-shaping consumer, not demonstrated improved capacity. At the next inference call, a QA handoff supplies context and guidance, not an evidenced parameter update. Initial provenance, additional strategy mechanisms, provider state and unestablished receiving consumers keep comparison coverage partial. See profile verification and MEM-RTE-evidence-brief.
Independent property assessments and what would change them
At the builder-plus-solver boundary over successive task launches, the following assessments remain separate. Their supporting account is RT-RTE-generation-change, with the review-specific mechanism in EPI-RTE-review-decision.
| Property | Supported assessment | Unresolved question |
|---|---|---|
| Theory-builder condition 1: localized content | Afforded through hypothesis/delta fields and named whole strategies. | Actual formulated theory content was not inspected. |
| Condition 2: consumption | Wired: hypothesis content is explicitly supplied in implementation/review decisions. | Actual semantic dependence and model compliance remain uninspected. |
| Condition 3: content-directed criticism | Afforded through counterfactual diagnosis and proposal-versus-implementation blame. | No actual sound content-directed criticism was inspected. |
| Condition 4: iteration | Wired through retained support, later briefs and bounded criticism-fed rebuilds. | Actual iterative criticism of a formulated theory remains uninspected. |
| Learning as improved future capacity | A retained evidence-to-code-to-later-consumer path exists. | Whether criticism of a consumed theory improved capacity, and its cause, remain uninspected. |
| Reflection | Wired: represented bundles, actions and outcomes can mediate later harness revision. | Accurate self-diagnosis and observed exercise remain uninspected. |
| Reflective theory builder | Uninspected. | The evidence does not establish criticism and revision of the investigative/editor method texts themselves. |
| Autonomous theory builder | Afforded within the configured curriculum: computation supplies diagnosis, selection, editing, review and promotion. | Reliable performance of every internal theoretical operation is not established. |
| Self-improvement | Dispositional improvement-directed self-change is wired for task success, with efficiency secondary. | Link 4—subsequent operation causally depending on a particular edit—is uninspected; exercised self-improvement and successful improvement are unestablished. |
For the scenario of a configured support-task experiment, the system supplies a live, evidence-responsive harness-change architecture with bounded admission and recovery. For the scenario of frozen downstream evaluation, it supplies a selected executable product, not continuing improvement. Neither scenario's implementation alone establishes causal gains. An inspectable evolution episode linking evidence, criticism, exact admitted bytes and later change-dependent operation would alter the exercised-path assessment. Matched outcome comparisons could establish benefit at the treatment granularity actually varied; individual module attribution needs independent variation of that module. Fail-closed substantive review, review renewed after repair, revision-bound replay and stronger execution isolation would change the corresponding architectural guarantees rather than retroactively establish past success. See RT-RTE-generation-change, RT-RTE-bundle-selection and RT-BAP-module-execution.
Limitations
| Limitation | Affected SRC IDs and record citations | Inspected boundary | Conclusion prevented | Evidence that would resolve it |
|---|---|---|---|---|
| No inspected full evolution episode or independent gain comparison | SRC-1; RT-RTE-generation-change, RT-CLM-harness-improvement | Implementation and attributed README results; bundled individual trials not inspected; training/verifier history omitted | Exercised edit-dependent self-improvement, actual theoretical criticism, improved capacity, reproduced gains or transfer | Linked evolution artifacts showing evidence, criticism, admitted bytes and later dependent execution; appropriately controlled outcome comparisons |
| Opaque providers and unfrozen external tasks | SRC-1; RT-CMP-lite-llm, RT-RTE-solver-task, boundary | Client inference constructors and task interfaces, not provider internals or remote payloads | Stable served-weight identity, provider-side learning, task contamination or oracle adequacy, deployment security | Pinned model/service identity and inspected state contracts, frozen task/test contents and deployment configuration |
| Additional executable strategy alternatives remain uninspected | SRC-1; RT-OBJ-module-library, RT-RTE-context-handoff | Baseline and selected alternative mechanisms, not all variant bodies | Uniform context/stopping policy, exhaustive memory mechanisms or complete-system negative about accepted claim production | Route-level inspection of the material remaining variants and their actual consumers |
| Initial generation ancestry is sanitized | SRC-1; MEM-OBJ-archive-entry, RT-OBJ-module-library | Released source/archive plus live initialization path | Complete historical lineage, admission agency or trace origins for initial entries | Original training/admission records linked to released content |
| Review and forced sanity do not guarantee final-byte approval or superiority | SRC-1; EPI-RTE-review-decision, RT-RTE-generation-change | Timeout/exception skips, missing-verdict handling, post-review repair, activation/crash gates and non-gating routing probe | Every promotion substantively reviewed, normally selected or reward-improving | Final-tree review records and fail-closed/renewed-review enforcement; ordinary-routing evidence and separate comparative outcome checks |
| File policy is not candidate execution isolation | SRC-1; RT-BAP-module-execution | Editor file guards, smoke/library imports and helper execution | Safe execution of hostile candidate code from no-shell editing or task isolation | Explicit host isolation and independently checked enforcement across candidate execution paths |
| Unresolved editor-memory delivery conflict | SRC-1; boundary account, MEM-OBJ-editor-memory, MEM-ABS-editor-readback, report verification | Boundary wording suggests editor use; memory's symbol search in shipped src/ and scripts/ finds formatter/writers but no journal consumer |
Joint conclusion of automatic journal-to-editor delivery; archive outcome use cannot substitute | A traced shipped journal consumer or explicit narrowing of the boundary wording; independent proposal read-back remains supported |
| Working-context inventory is locally incomplete | SRC-1; RT-RTE-solver-task, memory annotation, report-verification limit | Conditional solver environment-skill frontmatter/catalog supply independently checked by verifier; actual configured skills and later bodies not observed | Complete comparison of solver instruction discovery and subsequent skill-body consumption | Records tracing configured catalog selection, injection and later body reads; local compression/editor findings remain valid |
| Retryability and handoff production do not establish receiving schedules | SRC-1; EPI-RTE-finding-disposition, MEM-OBJ-proposal-state, MEM-OBJ-implementation-record | Snapshot-local finding ingestion; cross-module output and manifest helper APIs | Automatic later retry of unattached findings or additional cross-module/manifest consumer routes | Inspected scheduling/resupply and receiving-driver wiring, with the consumed payload identified |
| Secondary efficiency diagnoses lack an established durable path | SRC-1; MEM-RTE-evidence-brief, memory integration issues | Primary snapshot persistence precedes secondary efficiency results; all-passing batches skip reflection in inspected caller | Effective efficiency-only durable updates through that function | A persisted secondary-result route and later selection/implementation consumer |
| Persistence, retirement and replay have different consistency boundaries | SRC-1; MEM-RTE-progress-resume, MEM-RTE-history-contrast, RT-RTE-bundle-selection, EPI-RTE-paired-confirmation | Best-effort checkpoints/archive sync; dangling history pointers; current-record confirmation; static/cache/forced alternatives and instruction-only cache key | Transactional recovery, complete resumed confirmation history, universal retirement or revision-safe replay | Cross-artifact consistency/recovery evidence, preserved referents, historical confirmation inputs and revision-bound replay/withdrawal enforcement |
| Retained displays and continuation are lossy and unchecked | SRC-1; MEM-RTE-evidence-brief, RT-RTE-context-handoff, memory annotations | Finding truncation, pre-summary tail unwinding, bounded observation history, differing installed/released fallbacks | Complete support at consumers, summary fidelity, uniform recovery or automatic reload of dropped chat | Actual input/output fidelity comparisons and inspected recovery/reload routes for the selected variant |
| Confirmation source granularity and comparison coverage remain bounded | SRC-1; EPI-RTE-paired-confirmation, profile verification | Task/bundle/reward tuples and descriptive shared-task statistics; unresolved variants/provider/receiving paths | A linked-trajectory source classification for confirmation, isolated component causality or complete memory-value inventories | Original input records establishing source content; independently varied components; records resolving remaining included units |