PrimeScientist synthesis

Type: agentic-system-analyses/types/agentic-system-synthesis.md

Bounded synthesis

PrimeScientist implements a feedback pathway from evaluated coding attempts to later search decisions and task guidance. Its strongest supported contribution is this wiring: outcome scores change durable selection state, and retained diagnostics feed requests for successor plans. Whether criticism of a consumed hypothesis improves future capacity remains unestablished. The evidence is source inspection, not execution or a causal comparison.

The analysis covers the shipped research-replication and code-optimization workflow at commit 29971beac6f4f4b41309b1326762e1b83ceece98, with cutoff 2026-10-14. SRC-1 supplies implementation, design intent and attributed benchmark reports. The whole-system boundary includes the search driver, linear baseline, reflector, retained state and AutoLab/FIRE-Bench integrations; it does not include external coding-agent or model-provider internals. Supporting task and preparation coverage remains partial. No target run, generated candidate trace or experiment was supplied or performed. See the boundary and evidence and Source register.

From task request to selected experiment

An operator supplies the task, models and search limits. Python then owns the ordinary sequential search loop without per-plan human approval. It reloads a filesystem experiment tree or delegates root-plan writing to a Codex reflector. A selected node without a packet is executed; one with a packet is expanded. Although the selector is named puct_select_leaf, its inspected policy is stochastic BAVT: retained Q/P values affect sampling weights. Q is a backpropagated running mean that can include descendant results, not the node's own measured reward or a truth probability. See RT-RTE-search-execution, RT-OBJ-stats and RT-BAP-score-selector.

The representative AutoLab path connects the shipped components in one workflow. The adapter stages the selected plan as a native skill, supplies the task instruction and creates a fresh workspace. A positive-Q parent with a retained workspace can instead seed the child. Codex edits and executes through its external runtime; the adapter then invokes the task verifier and returns a reward packet. With repeated attempts, the driver keeps the maximum primary score and winning artifacts, not an uncertainty estimate. Worker completion is an intermediate handoff: even a nonzero worker exit can proceed to evaluation when a run ID is returned. This is source-supported delegation, not an observed successful optimization. See the workflow account, RT-OBJ-sandbox and RT-RTE-score-evaluation.

The runtime map separates four responsibilities: Python selects, admits and retains; the reflector proposes plans; the external coding agent performs task work; benchmark machinery scores the result. Plan text is model-interpreted guidance, whereas scores have fixed computational force. Staging a skill does not establish actual loading or compliance. A directory named sandbox does not establish confinement: AutoLab's coding worker and reflector request host full access, while other runners use different host/container grants. The verifier container is not the coding worker's containment boundary. See RT-BAP-plan-worker and the runtime's forcing cases and guarantees.

What the evaluator licenses

The two shipped benchmark families provide different evidence. AutoLab uses task tests rather than the worker's success narrative. The inspected Gaussian task checks one deterministic 256×256 image against a reference with tolerance, then gates performance scoring on that result. Its performance reward uses the median of three times reported by the submitted binary. This supports a finite-input correctness check, not universal correctness or independently verified general speed. Other task oracles are not fully assessed. See EPI-RTE-gaussian-check.

FIRE-Bench instead selects a log fragment, asks a model to generalize it while omitting concrete values, methods and artifact references, then uses RAGChecker to compare extracted claims with an imported expected answer. Precision, recall and F1 are arithmetic over model judgments. The reference's initial warrant and the transformation's fidelity remain unresolved. Consequently F1 can support reference-relative agreement of generalized narrative, not verification that the original experiment occurred or that its proposed mechanism is true. This bounds the reflector prompt's stronger language equating score with conclusion correctness. See EPI-RTE-conclusion-extraction, EPI-RTE-reference-check and EPI-CLM-score-correctness.

The FIRE-Bench packet supplies normalized scores but deliberately omits reference-derived false-positive/negative diagnoses and the agent conclusion. Missing or malformed score attachment can become zero; zero is therefore not necessarily evidence of falsity or experimental failure. The optional model/file-search error-analysis helper exists but is disabled in the ordinary CLI and has no established corrective consumer. Neither packet withholding nor hidden files guarantees denial of filesystem access. See EPI-RTE-score-attachment and EPI-RTE-error-analysis.

From retained outcome to successor guidance

The driver stores packets, logs, workspaces and ledgers. Optional pruning compares a child's score with positive parent aggregate Q minus a threshold. It hides the rejected branch, keeps its artifacts and writes a failure digest, but does not backpropagate the rejected score. This rejects future search use after execution; it does not undo prior agent effects. The digest can carry requested hypothesis/rationale fields, or short plan and verifier excerpts. Missing fields can remain placeholders rather than explanations. See RT-RTE-search-execution, MEM-OBJ-packet and MEM-OBJ-pruned-children.

On expansion, code makes a diagnostic log brief available: at most the first 100 and last 500 lines, with the full log still reachable. The reflector is asked to diagnose verifier/runtime results and produce different hypotheses. Python admits child directories by plan existence and prior defaulting/clamping, not semantic validity, evidence fidelity or scientific provenance. FIRE-Bench requests an executable plan transformation and change explanation, but bootstrap does not require that transformation to have run. Existing raw proposal directories can enter the same gate without a fresh author call. Reasons are requested in a durable prior file for later reflection; their actual authorship and use are unobserved. See MEM-RTE-log-brief, RT-RTE-plan-admission, MEM-OBJ-prior, MEM-OBJ-diff and MEM-OBJ-changes.

Memory therefore has several consumers, not one retrieval store. Python reloads symbolic controls; the reflector can read diagnostics and reasons; workers receive selected plans and code. Automatic trace-fed briefs, digests, history and numeric updates have durable later-consumed outputs. They establish wired trace-fed adaptation, not measured learning or provider-weight change. The reflector also requests an external resumed session whose payload is opaque, so files are not the only continuity interface and the excerpt budget does not bound total model context. The profile verification accepts these local distinctions while retaining partial coverage on every comparison axis. See MEM-RTE-session-resume and RT-OBJ-stats.

FIRE-Bench automatically supplies nonempty other-task insight files, filtered by task identity rather than semantic relevance or a numeric context budget. The producer and initial admission of these files remain unresolved; copying and later delivery do not close a cross-task learning loop. The linear baseline is a separate route: it stages the last ten attempt records with summaries truncated to 400 characters, seeds from best-so-far code, and replaces that code only on strict score improvement. It bypasses reflector/tree-prior admission. See MEM-RTE-insight-delivery, MEM-OBJ-insights, MEM-OBJ-history-shown and RT-RTE-vanilla-retention.

Delivery, learning and independent properties

Normal termination exposes a retained tree, evaluated packets, winning workspaces/logs, ledgers and a printed best-Q summary. There is no additional final verification of that printed node, production installation or recorded acceptance of an individuated scientific claim. The meaningful acceptance criteria are for continued benchmark search: score-based branch eligibility or strict baseline improvement. Retention and operational use are not scientific acceptance or lifecycle integration. Resource limits are also bounded: ledger/time checks occur between delegated operations, initialization precedes the loop timer, and failed unrun nodes do not consume the evaluation count. The estimated-dollar “hard stop” is best effort, not a preemptive spending invariant. See the delivery account, RT-CLM-budget-stop and the epistemic conclusion.

The runtime's theory and self-change findings concern task-plan hypotheses over rounds of one task, not an established revision of PrimeScientist's method. Each remains independent, as supported by RT-RTE-plan-admission:

  • Theory-builder condition 1, localized content: afforded. Whole plans and named hypothesis fields can carry a theory-like unit; no generated theory is supplied.
  • Condition 2, consumption: afforded. Plans reach an interpretation path, but decisions depending on their specific assertions are not demonstrated.
  • Condition 3, content-directed criticism: claimed. Prompts request result-grounded diagnosis; no stated criticism places blame on a theory, test, data or auxiliary and shows consequent revision.
  • Condition 4, iteration: afforded. Files and sessions can carry criticism forward; exercised criticism-to-successor dependence is unobserved.
  • Learning from criticism: uninspected. Improved future capacity and causal benefit have not been established; wired trace-fed updates do not answer this question.
  • Reflection: wired at the operational representation level. Outcome representations feed a proposer whose admitted guidance can affect later attempts. Model-mediated dependence remains unobserved. Reflective theory-building over method texts is uninspected.
  • Autonomy: computation owns internal scheduling and admission without per-content approval. Autonomous theory-building is afforded, not established as exercised, because the four theory operations are not demonstrated.
  • Self-improvement: a dispositional improvement-directed self-change pathway exists. Retained plans and starting code can enter later behavior under the independently specified benchmark-score objective. Evidence-responsive model update determination and later causal dependence on the update remain unobserved. Exercised self-improvement and successful improvement are unestablished; better task code alone need not improve the framework's method.

For a budgeted benchmark-search scenario, this boundary supports an account of repeated attempts with outcome-sensitive selection, reuse and prospective guidance. It does not support exact budget enforcement, universal containment, scientific discovery or causal superiority to the linear baseline. Generated plans with localized criticisms, their retained successors and subsequent action traces would establish whether the theory affordances are exercised. Controlled comparisons of evidence-conditioned updates with unchanged guidance would address capacity and benefit at the tested horizon. Independent result replay and task-specific evaluator validation would change the warrant assessment. Enforced semantic proposal checks, live-operation budget controls or verified executor containment would change the corresponding architectural guarantees. No such evidence or system changes are present in this run.

Source-read log: none. This synthesis uses the supplied members and their records only.

Limitations

Limitation Affected SRC IDs and record citations Inspected boundary Conclusion prevented Evidence that would resolve it
Failed-resume recovery coverage remains a reconciled conflict: runtime/workflow describe clearing, while memory records timeout and subsequent-save exceptions. SRC-1; RT-RTE-plan-admission, MEM-RTE-session-resume, workflow recovery; report verification limits. search/reflector.py: nonzero return unlinks state, timeout bypasses clearing, and failure stdout can save an identifier again. Guaranteed fresh-next-call recovery or clearing after every failed process; complete restored-context fidelity. Agreed conditional recovery contract and traces covering timeout, nonzero return, stdout parsing and the following invocation, including restored payload.
Baseline replay and workspace replacement provide incomplete continuity. SRC-1; MEM-OBJ-history-shown, RT-RTE-vanilla-retention, RT-OBJ-sandbox; report verification limits. run_vanilla.py: live formatting uses conclusion-or-summary, replay uses summary only; best-workspace replacement is destructive and nontransactional. Complete narrative replay or a reusable best workspace guaranteed solely by recorded improvement. Restart traces comparing delivered history and atomic/recoverable workspace replacement with failure-path evidence.
Packet pointers can become stale when pruning renames directories. SRC-1; MEM-OBJ-packet, RT-RTE-search-execution; report verification limits. run_search.py: paths are updated before the pruning rename, not afterward; hidden files and digest remain. Reliable later dereference through every retained packet path. Post-rename path updates and traces showing subsequent consumers resolving retained artifacts.
Cross-task insight production, initial admission and withdrawal are unresolved. SRC-1; MEM-OBJ-insights, MEM-RTE-insight-delivery; report verification limits. search/reflector.py, run_search.py, FIRE-Bench worker loaders: touching/copying/loading are known; local copies can survive deletion and supply has no semantic relevance budget. Closed outcome-to-insight learning, verified source quality or evidence-based withdrawal; bounded relevant-only delivery. Recorded producers and admission decisions, outcome-to-insight provenance, withdrawal propagation and inspected selectors with delivery traces.
Preparation and method-guidance inventories are partial. SRC-1; epistemic coverage, memory coverage, RT-RTE-plan-admission; report verification limits. benchmarks/fire_bench/benchmark/tree_parser.py, benchmarks/fire_bench/benchmark/plan_generator.py, benchmarks/fire_bench/benchmark/reflect_program.md: verifier found preparation writers and method-editing doctrine, not separately declared analyst routes or ordinary consumers of the default enriched instruction/program. Exhaustive preparation lineage, global-guidance doctrine or system-complete theory/epistemic-route comparisons. Analyst records covering preparation transformations, method doctrine and actual consumer wiring, with operation evidence for any claimed loop.
Evaluator and effect-envelope coverage is task/path bounded. SRC-1; RT-RTE-score-evaluation, EPI-RTE-gaussian-check, EPI-RTE-reference-check, RT-RTE-search-execution; report verification limits. AutoLab task tests and enroot helpers, FIRE-Bench utility calls and low-level RAGChecker/provider checking remain incompletely assessed. Exhaustive task-oracle validity, container-path equivalence or downstream model/context inventories; generalizing the Gaussian witness to all tests. Source-grounded records for material alternatives, validated task oracles, deployment grants and evaluator/measurement comparisons.
No execution or causal evidence establishes exercised criticism, benefit or scientific acceptance. SRC-1; RT-RTE-plan-admission, RT-RTE-search-execution, EPI-RTE-conclusion-extraction, EPI-RTE-reference-check. Implementation and prompt intent only; no generated hypotheses, criticisms, acceptance decisions or interventions. FIRE-Bench checks generalized narrative against references. Exercised theory-building, improved capacity, successful self-improvement, component attribution or independently accepted scientific findings. Linked candidate/check/criticism/successor/action traces, scoped scientific acceptance records and controlled comparisons with independent outcome validation.
External payloads, model resolution and arbitrary workspace parts remain opaque. SRC-1; MEM-RTE-session-resume, RT-BAP-plan-worker, RT-OBJ-sandbox, RT-CMP-coding-model, RT-CMP-reflector-model, RT-CMP-eval-model, RT-CMP-ragchecker-models; profile verification. Saved session metadata and staging/copy interfaces are inspected; provider weights/version resolution, session contents, native-skill loading, compaction and task-specific data/model consumers are not. Filesystem-only or completely bounded context, immutable provider parameters, complete memory-form/authority inventories or observed skill compliance. Pinned deployment identities and inspected host/provider memory contracts, payload/skill activation traces and task-specific workspace-consumer records.