Ingest: ScienceFlow: A Long-horizon Agent for ML Research, Scientific Discovery and Beyond

Type: kb/sources/types/ingest-report.md

Source: scienceflow-long-horizon-agent-for-ml-research-and-discovery.md Captured: 2026-08-18 From: https://arxiv.org/abs/2608.14354v1

Classification

Genre: scientific-paper — an arXiv preprint that specifies a system and reports benchmark, ablation, replay, and telemetry experiments. Domains: agentic-systems, long-horizon-agents, context-engineering, autonomous-research Author: nineteen authors from Noah's Ark Lab, Huawei. The authors released an implementation in Huawei's official noah-research repository, but the evaluation remains an affiliated self-report.

Summary

ScienceFlow is a long-horizon research-agent harness built around recoverable executable state. It divides work into validated Stages. Each accepted Stage records a workspace snapshot, structured memory, evidence, and resource state. ESTRA then chooses a live or archived anchor and whether to continue or redirect the trajectory. A separate controller admits and monitors physical jobs using resource availability, budget, and validated progress. The paper reports results on machine-learning competitions, scientific modeling, and mathematical optimization, including 70.22 ± 1.18% Any-Medal on the full 75-task MLE-bench. The implementation confirms the main state-management and control mechanisms, but the inspected repository does not include the paper's run artifacts, so the reported outcomes remain paper-only.

Code Grounding

The inspected revision is huawei-noah/noah-research@f16be15660284898354e2a5d0fe195f97e4685c4. This monorepo revision postdates arXiv v1 by four days, so it is a durable inspection target rather than a claim that the exact experiment commit was recovered.

Implemented mechanisms. Static inspection confirms content-addressed workspace capture and restoration in workspace_snapshot.py. The main solver implements Stage transitions, ESTRA decisions, and pending archived-state restoration in solver.py. Bounded folding with a retained raw Stage ledger appears in stage_memory.py. Reject-capable evaluator admission is implemented in gates/service.py. Evidence-aware job admission, leasing, monitoring, and review surfaces appear in resource_runtime/admission.py and resource_runtime/runtime.py.

Artifact-supported operation. The repository includes task packages, example configuration such as tasks_circle_packing_example.yaml, and focused tests for workspace snapshots, Stage memory, and Gate integration. These artifacts show that the paper's mechanisms are operationalized. They do not establish that the published benchmark runs used this exact revision or produced the reported scores.

Paper-only outcomes. The MLE-bench scores, 4.92-point comparison, scientific-modeling and optimization results, ablation effects, replay regret, telemetry, and storage-reduction measurements were not independently confirmed. No inspected result bundle linked the code revision to those outcomes.

Execution status. No project code or tests were run. The checkout had no prepared virtual environment and lacked declared dependencies such as Pydantic and OmegaConf. The code-grounded ingest procedure excludes dependency installation, data or weight downloads, and full evaluations. The evidence here is static inspection, not reproduction.

Connections Found

ScienceFlow supplies a code-grounded whole-system example for active work state being distinct from retrospective memory or chat history: its recoverable state includes the executable workspace, memory, evidence, and resource records. Its retained raw Stage ledger and bounded folded views support the requirement to preserve evidence without loading history. The design also instantiates the scheduler, context engine, and execution-substrate runtime decomposition and a reject-capable search, evaluation, and retention loop. Compared with ACM, ScienceFlow couples bounded context to exact executable-state restoration. Compared with Exo, it similarly restores mutable state without erasing the surrounding record, but preserves research evidence and cumulative resource accounting rather than an event log around self-modification.

Extractable Value

  1. Put recoverable work and irreversible evidence on different sides of the restoration boundary. ScienceFlow restores an executable workspace and its anchor-specific state while preserving the archive and cumulative resource accounting. The broader mechanism is that rollback can recover exploration without letting an agent erase failed attempts or spent budget. [quick-win]
  2. Make a checkpoint an evaluated state transition, not merely a saved directory. The Gate can reject a proposed Stage before it becomes an accepted anchor. This operationalizes the existing proposal-selection claim with an executable, reject-capable retention step. [quick-win]
  3. Retain exact evidence while assembling a bounded working view. The Stage ledger and content-addressed workspace objects remain available even when folded memory is what enters the model's context. This is a concrete implementation of separating retention from loading. [deep-dive]
  4. Replay scheduler choices from a state-matched checkpoint. The paper evaluates re-anchoring decisions by branching alternative actions from the same captured state. This is a reusable assay design because it reduces pre-decision trajectory confounding, although it still depends on the evaluator and short replay horizon. [experiment]
  5. Use validated progress as a resource-control input. Admission and lease decisions combine budget and capacity with Stage evidence rather than relying only on elapsed time or worker demand. The code confirms the control surface; the paper's execution-control ablation remains the only outcome evidence. [deep-dive]
  6. Treat the result as evidence inside a fixed decomposition. ScienceFlow shows that its compound harness sufficed on the tested tasks. It does not show that the supplied signals, task adapters, evaluators, checkpoint trigger, action vocabulary, memory policy, or worker topology are the best general abstractions for long-horizon research. [just-a-reference]

Limitations (our opinion)

This is a v1 preprint evaluated by the system's authors. The official code confirms implementation but not the reported outcomes. The inspected revision was published after the paper submission, has no paper-experiment tag, and contains no inspected run bundle that ties it to the headline results. Tests were not executed during ingestion.

The headline comparison is not a controlled harness-only comparison. Models, hardware, time budgets, and prior reported baselines differ. A simpler account for part of the gain is the combination of a strong 2026 model, a generous 24-hour and two-GPU task budget, and benchmark-specific task packaging. The 22-task Lite ablation uses three seeds and varies ESTRA or execution control, but it does not isolate every component or reproduce the full 75-task result. The scientific-modeling study uses one run per system-task pair. Several optimization comparisons are descriptive rather than compute-normalized.

The fixed decomposition matters. Inputs to the model include workspace observations, task-specific result signals, accepted Stage cards, evaluator scores, resource telemetry, peer summaries, and folded or unfolded history. The model can edit and run code, select a live or archived anchor, continue or redirect, and request memory or compute operations. It can optimize programs and task artifacts within the supplied schemas. It cannot change the task adapters and evaluator oracles, benchmark candidate representation, Stage trigger, two-axis ESTRA vocabulary, folding policy, worker topology, resource-controller rules, task definitions, or baseline protocol. The ablations vary only selected mechanisms inside that larger structure. Under the fixed-decomposition lens, the experiments support those mechanisms within the tested design; they do not validate the surrounding hypothesis and response spaces.

The paper's state-matched replay is useful but short-horizon. A branch that looks worse over the replay window may have delayed payoff. Retrospective telemetry on 54 of 75 tasks can describe behavior but cannot by itself establish that the controller caused the final gains.

Write a code-grounded whole-system analysis at kb/agentic-systems/scienceflow.md, centered on the recovery boundary between reversible workspace state and preserved evidence or resource accounting, and keep implemented mechanisms separate from paper-only outcomes.