OpenRSI synthesis
Type: agentic-system-analyses/types/agentic-system-synthesis.md
Bounded synthesis
OpenRSI implements an improvement plane for machine-learning engineering: it constructs executable tasks, searches for solution programs, retains evaluated experience and supplies selected experience to operator training. Its strongest supported contribution is the wiring of these mechanisms, including durable experience-to-context and experience-to-parameter-update paths. The unresolved question is whether a particular update entered later operation and improved capacity. The evidence does not establish that historical chain. The project's own claim is bounded meta-evolution, not solved general recursive self-improvement. See RT-CLM-meta-evolution, RT-RTE-sft-training and MEM-RTE-summary-cache.
This account covers SRC-1, https://github.com/FrontisAI/OpenRSI, at commit 71ae803a035d5e3b78c19fa49ed9f67d0550cbaa, with an applicability cutoff of 2026-10-14. The source-release boundary includes Gym task construction and evaluation, ERL experience production and training, and Evo test-time search. These are separately runnable stages, not one demonstrated autonomous cycle. Evidence is inspected implementation plus attributed design and reported operation. No target execution, supplied execution trace or causal experiment establishes successful operation or benefit. External weights, full corpora, task releases, services and the upstream RL trainer are outside the captured evidence. See the boundary and Source register.
From task construction to an evaluation contract
Gym uses a graph to acquire task information, inspect files, generate a description, generate and retry preparation code, and generate a metric module. Code owns stage routing; models author descriptions and executable artifacts. Explicit query construction selects what each model sees. Preparation retries receive a recent code/feedback tail, not the complete preceding reasoning. Empty graph-message updates do not establish erasure under the append/merge reducer. See RT-RTE-gym-build and its memory annotation in the memory account.
The generated metric later becomes executable scoring machinery. Compilation establishes syntax, not fidelity to task intent. Separately invoked package checks inspect required files and CSV relationships and execute a schema-valid sample submission for a finite score. That score is a smoke test, not a model-quality baseline. Optional semantic review receives selected package evidence and code. Deterministic failures can gate that review and affect CLI status; report-schema validation does not verify the truth of each model reason, and a low recommendation alone is not an enforced task-use veto. These distinctions bound the claim of “verifiable” packages. See RT-OBJ-metric, EPI-RTE-package-validation, EPI-RTE-quality-judgment, EPI-RTE-quality-disposition and EPI-CLM-verifiable-packages.
From configured search to delivered code
The representative workflow is the shipped MLE-Bench invocation. An operator supplies tasks, endpoints and budgets; the adapter writes configuration and launches preparation and a runner. The runner creates task/sample subprocesses. Each solver chooses operators and parents; configurable Draft, Improve, Debug, Crossover and Analyze clients propose or assess programs. Separate client roles do not establish distinct or pinned model weights. Generated Python crosses into executable authority only at the selected task backend. See RT-RTE-evolution-search, RT-CMP-generic-llm and RT-BAP-execution-authority.
Evaluations supply validity and fitness; buggy programs enter bounded Debug attempts. Non-buggy candidates compete for island admission against average island fitness, not necessarily against their parent or the best solution. Score authority depends on protocol and configuration: self-validation can trust model-printed scores, analysis can supply judgment, and test-score override can change selection. With experience disabled, Analyze runs for every node before deterministic finite fitness overrides its metric and bug flag; experience-enabled operation instead prioritizes deterministic evidence before fallback. Thus executable evaluation is not a universal independent oracle. See RT-RTE-sandbox-evaluation, the epistemic ledger and the workflow repair annotation.
Budget checks govern scheduling and admission, not strict preemption of every external call. Async search serializes commits and can reject completed work after a stop condition. Execution grants also differ: isolated Gym preparation, direct imports, NatureBench local Python, Docker and remote execution are not interchangeable security boundaries. Actual containment depends on deployment and remains unobserved. See RT-RTE-async-search, RT-RTE-gym-build and RT-RTE-naturebench-execution.
Delivery is weaker than solution certification. Final selection can fall back to a seeded random code-bearing node when no scored candidate qualifies, or write empty code when none exists. Code files are written before optional test-phase evaluation. A zero subprocess exit is distinct from the stricter resume-completeness predicate. NatureBench can reuse search-stage scores instead of obtaining fresh final evidence. Neither valid_code_final.py nor process success establishes a valid or improved solution. See WF-RTE-final-delivery.
From evaluated attempts to later behavior
Search history, active solutions and derived experience are separate objects. The journal retains code, plans, logs, scores and relationships, including attempts no longer eligible as active solutions. Cards and a strategy board derive structured experience from those attempts. Optional summaries describe a method and compare it with its parent. Software admits two nonempty summary fields; it does not establish their semantic truth. Existing node/card/cache content takes precedence over regeneration. See MEM-OBJ-journal, MEM-OBJ-experience-card, MEM-OBJ-rich-summary and MEM-RTE-summary-cache.
Later operator calls receive code-selected context rather than the entire prior conversation. Improve selects ancestry and shared-parent siblings; Debug prioritizes matching error signatures. Summary fields are clipped to 420 characters, with count limits rather than a global token ceiling. Legacy memory operators remain alternatives. Checkpoint/cache loading is requested read-back; subsequent automatic prompt selection supplies context without a foreground model retrieval request. This establishes retained, trace-fed context wiring, not actual reliance or improved capacity. RL's similarly named sibling renderer selects children of the current parent, so it is not equivalent to Evo's selector. See MEM-RTE-targeted-memory, RT-BAP-operator-guidance and the memory annotation on RL training.
SFT makes a different projection of experience. Software selects examples using score/feedback, length, validation/test-gap, top-k and reward-deduplication criteria. Reconstructed conversations reach tokenization and loss masking; selection metadata can remain in a separate manifest without reaching the loss consumer as reasons. Actor updates, checkpoints and rollout weight refresh are implemented. The shipped launcher's asynchronous timing is only coarsely covered: rollout refresh depends on update_weights_interval, not an unconditional refresh after every update. See MEM-OBJ-alignment-dataset, RT-RTE-sft-training, RT-CMP-training-models and report-verification limits.
RL instead returns rewards through hooks to external training machinery and retains programs in SQLite for later selection. Negative cheating/holdout verdicts can bypass ordinary execution and incur penalties while still permitting storage. Some judge failures allow continuation; missing credentials are a separate exception. Database membership therefore certifies neither legitimacy nor uniform parent eligibility. SQL pruning, active-population filtering and reward-baseline consumers have different rules. The unavailable upstream optimizer cannot be inferred from the vendored SFT trainer. See RT-RTE-rl-training, MEM-OBJ-rl-program, EPI-RTE-hack-judgment and EPI-RTE-hack-disposition.
What the supported mechanisms establish
For bounded task search, the release implements consequential feedback: scores affect parent selection, Debug and final output; retained experience can reach later proposals; selected traces can drive numerical updates. For explanatory knowledge production, the same evidence is narrower. Outcome comparisons do not isolate the cause of success, and cached comparative prose is candidate experience rather than an evidence-accepted explanation. Admission for training use is not acceptance of every proposition in an example. See the epistemic bounded conclusion.
The independent property conclusions remain separate:
- Learning: durable trace-fed updates and numerical adaptation are wired. Improved capacity for future action, including improvement caused by criticism of a consumed theory, is unestablished. No causal comparison supports that stronger attribution.
- Theory-builder conditions 1–4: the runtime does not establish a formulated-theory route with separately evidenced localized theoretical content, content-dependent consumption, content-directed criticism and retained criticism shaping iteration. Code replacement, score selection and requested comparison alone do not establish those conditions. This is not a finding that hidden model processing lacks them.
- Reflection: representations of attempts and requested parent comparisons are supported; reliable reflection and the required two-way causal relation are not established merely by their delivery.
- Autonomy: computation chooses, evaluates, repairs and selects candidates inside an operator-configured bounded search without per-candidate human approval. This does not establish an autonomous theory builder or an autonomous release-wide training/deployment cycle.
- Self-improvement: the training pathway supports a marked dispositional, improvement-directed parameter-update claim at the improvement-plane boundary. Exercised self-improvement remains unestablished: the exact changed artifact's entry into live Evo serving and subsequent operation causally depending on that change are missing. Successful improvement requires additional outcome evidence even if those links are closed.
These conclusions follow the separate route limits in RT-RTE-evolution-search, RT-RTE-sft-training, RT-CLM-meta-evolution, the workflow account and the epistemic conclusion; they are not a system-wide epistemic grade.
A linked run record connecting evaluated experience, admitted dataset, exact checkpoint, deployed model identity and later behavior would change the exercised-path assessment. A controlled comparison varying the relevant update would address benefit attribution at that comparison's scope. Concrete summaries checked against their input evidence would permit stronger semantic classifications; traces of claim-specific criticism and consequential revision would address theory-building questions. None is supplied here. No additional source files were read for this synthesis.
Limitations
| Limitation | Affected SRC IDs and record citations | Inspected boundary | Conclusion prevented | Evidence that would resolve it |
|---|---|---|---|---|
| Source wiring is not observed operation or causal benefit; external artifacts and historical deployment are uncaptured. | SRC-1; RT-CLM-meta-evolution, RT-CMP-training-models, RT-RTE-rl-training | Frozen source release; no target execution or registered run/experiment source | Exact reproduction, historical lineage, exercised self-improvement, successful learning and component-effect attribution | Registered traces, corpora/checkpoint identities, deployment records and suitably controlled outcome comparisons |
| Static-guidance inventory is partial: optional task-name-selected Markdown skill injection supplements retained experience. | SRC-1; RT-BAP-operator-guidance, MEM-RTE-targeted-memory, report-verification limits | Verifier inspected MLE-Bench adapter and task-skill guidance helper; not every skill/loading branch | Exhaustive static-guidance selection, loading and budget comparisons; not journal-derived read-back | Complete selector/template/skill coverage and resolved delivery budgets under named configurations |
| SFT scheduler inventory is coarse: the launcher uses asynchronous training with interval-conditioned rollout refresh. | SRC-1; RT-RTE-sft-training, RT-CMP-training-models, report-verification limits | Launcher and verifier-inspected async update condition; synchronous account is not complete timing coverage | Exact synchronization, staleness and scheduler equivalence; not the implemented numerical-update pathway | End-to-end async scheduler and synchronization account under explicit configuration, with traces for operational timing |
| Persisted Gym reports lack dedicated object-level units in the partial memory profile's storage, form, lineage and admission inventories. | SRC-1; EPI-OBJ-task-evaluation, EPI-RTE-quality-disposition, profile-verification limits | Accepted report records establish normalized JSON/CSV findings and reasons; profile inventory omits separate units | Object-by-object report-retention/admission comparison; neither omission nor persistence establishes endorsement or additional later read-back | Dedicated classifications derived from the report records; a supported later-consumer route for any read-back claim |
| Continuation is not lossless restoration, and summary cache validity is structural rather than semantic. | SRC-1; MEM-OBJ-journal, MEM-RTE-summary-cache, MEM-RTE-sft-search | Nested metric-info export versus flattened reload, separate cards/caches, node-ID reuse, SFT preload with blank feedback/log fields | Complete provenance restoration, exact replay, fresh/correct summary guarantees | Matched export/reload schemas, complete linked state, content-sensitive cache checks and resumed-run evidence |
| Task truth, evaluator independence and deployment isolation vary by route and remain unverified in operation. | SRC-1; RT-OBJ-metric, RT-RTE-sandbox-evaluation, RT-RTE-naturebench-execution, RT-BAP-execution-authority, WF-RTE-final-delivery | Generated metrics, score-source alternatives, local/container/remote grants and terminal fallback | Universal task-semantic validity, independent final confirmation, containment or validity from delivered filenames/status | Independently checked task/reference contracts, fixed score protocols, deployed-grant evidence and fresh final evaluation traces |
| Included alternatives remain partly uninspected or opaque; positive witnesses are not complete inventories. | SRC-1; MEM-RTE-targeted-memory, MEM-OBJ-rl-program, RT-RTE-rl-training, RT-RTE-gym-build, the epistemic coverage | Other vendored/database copies, RL algorithms/payload parts, NatureBench context, metadata/tool admission, sandbox operational stores and provider/deployment branches | Whole-release memory equivalence, complete profile values or system-wide absences of acceptance, learning or other mechanisms | Route-local inspection resolving each included part, actual consumer, admission rule and opaque payload; external contracts where needed |
| Comparative prose and model judgments lack instance-level semantic and causal validation. | SRC-1; MEM-OBJ-rich-summary, MEM-RTE-summary-cache, EPI-RTE-quality-judgment, EPI-RTE-hack-judgment | Structural summary/report acceptance, evidence-conditioned package review and pre-execution source review; no concrete validated generated instance | Accepted explanatory knowledge, reliable cheating detection, content-directed theory criticism or improved capacity from such criticism | Input/output comparisons, evidence-consuming semantic dispositions, calibrated judgments and linked consequential revision/benefit evidence |