h t t p s : / / a r x i v . o r g / a b s / 2 6 0 9 . 1 9 6 4 4

Ingest: ScientistTwo: Pioneering the Human Knowledge Frontier with Autonomous AI

Type: types/ingest-report.md

Classification

An empirical research-system paper by Jaehyun Nam and colleagues at Google Cloud AI Research and the University of Waterloo, posted as arXiv v1 in September 2026. The authors describe and evaluate their own system. The evidence includes aggregate benchmark results, automated and human manuscript assessments, generated research examples, and selected experiment, criticism, and audit records. These are author-reported results, including the embedded agent audits.

Summary

ScientistTwo turns problems drawn from accepted machine-learning papers into revised methods, executable experiments, and manuscripts through specialized agents. It screens ideas on subsets, evolves them using successful and failed execution traces, removes or replaces components after ablations, and runs simulated review and rebuttal experiments. On 107 problems with available benchmark infrastructure, the authors report 86 successful outcomes and a 25.2% average relative performance gain across successful cases; these heterogeneous gains describe improvements within the supplied research tasks, not a controlled comparison of alternative research architectures. Automated reviewers favor many generated papers, but their decisions are not conference acceptances. The most useful evidence for Commonplace is the visible revision of a proposed explanation after criticism, together with examples showing why reproducibility, method-code agreement, and validation-set guarantees each support narrower claims than scientific validity as a whole.

Quotes

No source quotes have been retained yet.

Connections Found

The paper supplies a concrete boundary case for exact implementation does not validate a requirement against its objective. The Procrustes-DS example passes a reported audit against its revised method after criticism removed or replaced several original mechanisms. That agreement cannot validate the superseded explanation. A second example is sharper: DynaSpec-RAG minimizes a normalized sum of validation MSE and MAE while retaining baseline fallback. This bounds that validation objective, but its own test table records regressions on two of seven datasets. The stronger claim that the method prevents negative transfer exceeds what the guard checks.

It also bears on a proximate target is checked for achievement, not for warrant. ScientistTwo pursues research quality through benchmark gains, attribution, and reviewable manuscripts. Automated review scores check selected aspects of those outputs; their relation to scientific novelty and venue readiness still needs outcome evidence. The held-out Stanford reviewer adds a useful check beyond the ScholarPeer reviewer used for revision. Across the two rebuttal rounds, however, ScholarPeer scores rise monotonically while the held-out scores peak after the first round. This is evidence about those review rounds within the fixed pipeline, not a test establishing that its division into specialized agents is preferable.

Learning Claims (our opinion)

The adaptation mechanism acts on proposed methods and their experimental record. An idea carries a formulation, code, results, a decision, and diagnostic feedback. Later proposals consume successful and failed traces; ablation and review criticism can change the method, after which the system reruns downstream experiments and drafting. Performance comparisons can reject a revision and retain the prior best output. Against the four conditions of a theory builder, conditions 1 and 2 are met: formulations, code, and manuscript claims are stated, and they direct which experiments run. Condition 3 (criticism) is met, with more evidence than a score-only selection loop, because stated criticism names the component it blames, as the Procrustes-DS record below shows. Condition 4 (iteration) is met: diagnostic feedback and failed traces are kept, later proposals consume them, and a revised method is rerun against the same benchmark, which is a real test of the new conjecture. As evaluated, ScientistTwo is inside the term. Idea evolution, ablations, and rebuttal experiments all revise one method for one supplied problem, so the persistence grade reached is across rounds of one run, and across runs on one task in the BPE-tokenization sequence below.

The Procrustes-DS record makes the mapping to addressable theory unusually concrete. Criticism identifies nearly uniform layer weights, unhelpful alignment, harmful feature gating, and badly scaled regularization. The revision changes the corresponding commitments: different weights, direct blending, gating at the score level, and scale-matched shrinkage. The reported final improvement is consistent with gains from formulated criticism, but learning is a separate claim. The record does not isolate criticism's causal contribution against equally resourced search without such formulations, and the final method's success does not vindicate the original story.

The effective update space includes research hypotheses, implementation, experimental plans, and manuscript claims. The paper does not demonstrate revision of the ScientistTwo harness, its stage boundaries, or its acceptance policy. The distinction in learning inside a fixed decomposition inherits its mistakes therefore applies: improving research artifacts within this workflow does not establish that the workflow's fixed decomposition is necessary or preferable. Three successive improvements to one BPE-tokenization task reuse prior outputs as new context, but on the same problem; they show no finding taken up by a different problem, which the highest persistence grade would need.

DynaSpec-RAG introduces a separate adaptation mechanism inside one generated method. A small residual module learns how strongly to use retrieved trajectories while the forecasting backbone and retriever remain frozen. It trains for 40 epochs on legacy training splits; deployment on five new domains avoids retraining weights but still uses domain calibration and retrieval pools. That is learned residual prediction and calibrated transfer, not by itself evidence of stated conjecture and criticism within the forecasting module; the residual module's weights fail condition 1.

Extractable Value

  • [quick-win] A precise limit to validation guarantees. DynaSpec-RAG's baseline-inclusive search guarantees a non-worse normalized validation sum. Neither improvement on each constituent metric nor non-regression on unseen test data follows. Its own Table 7 records two of seven test datasets regressing. This is a compact example for the existing requirement-validation note, with broader relevance to KB review criteria that check a proxy for the desired outcome.
  • [deep-dive] A visible sequence from criticism to replacement. The Procrustes-DS appendix retains an initial method, criticisms of specific components, a redesigned method, and a reported audit of that successor. It can support a case study of addressable revision while preserving the difference between evidence against the original explanation and evidence for the final implementation. The layer-weight ablation varies weights while holding the checkpoint, feature blend interface, scoring components, and benchmarks fixed; it attributes only that comparison.
  • [experiment] Separate in-loop review improvement from external judgment. On the 49 successful ICML-derived cases, ScholarPeer ratings rise from 5.2 to 6.9 to 7.6 across drafting and two rebuttal rounds; held-out Stanford ratings move from 5.6 to 5.8 to 5.7. This motivates checking whether revisions that improve Commonplace's own assays also improve judgments or outcomes outside those assays. It does not establish that a second round generally harms research quality.
  • [just-a-reference] A bounded autonomous-research cost and success reference. The reported 86/107 success rate concerns selected accepted-paper problems, and the mean gain aggregates successful cases with different metrics. The cost analysis reports about 2.5 days and $3,765 per research cycle on its NeurIPS-derived sample. These provide context for a substantial experimental workflow, not a Commonplace implementation estimate.

Limitations (our opinion)

Benchmark selection and comparisons. Existing papers, reproducible code, and benchmark suites give the system a substantial starting structure. Success on these problems does not establish discovery from an underspecified scientific question. Relative gains were extracted from result tables with model assistance and averaged across heterogeneous successful cases. Comparisons to other research systems also change tasks, models, resources, or stopping rules. The appendix explicitly says that the AutoSOTA comparison uses different reproduced baselines and sometimes different metrics; it cannot isolate the effect of ScientistTwo's architecture.

Review validity. ScholarPeer is inside the optimization loop. The Stanford evaluator is held out from it but remains an automated judge. Human assessment covers 33 successful papers with nine reviewers; the reported account does not establish a detailed blinding or inter-rater reliability protocol. These observations support manuscript-quality comparisons under the reported assessments, not actual venue acceptance or independent confirmation of every scientific claim.

Audit scope. The integrity table covers 49 successful ICML cases after filtering. The initial configuration has 50 cases, one of which is removed for a specification violation, so its rows are not a simple matched ablation of an auditor. Method-code agreement can also be repaired by correcting the manuscript to describe the code. The Procrustes-DS reproduction record explicitly audits the new method rather than reproducing its baseline. No implementation was independently inspected or executed for this ingest.

Claims can exceed the paper's own evidence. DynaSpec-RAG's abstract and conclusion claim prevention of negative transfer, while its guard constrains a validation sum and Table 7 reports test regressions. The five-domain transfer table also includes one increase in MAE. Its frozen backbone does not make the entire method untrained. Likewise, the TABHARMONY rebuttal tests feature enrichment with inner-validation selection and frequent fallback on capped datasets; small average gains do not establish uniform benefit or transfer of its backbone-specific efficiency mechanism.

Mechanism attribution remains local. Component ablations can challenge or support the component varied under the other fixed choices. They do not establish that an entire explanatory decomposition is correct. The DynaSpec-RAG tables also show that simpler neighbour aggregation can match or outperform its learned attention on individual metrics. More components and more experiments therefore do not, by themselves, establish the necessity of the proposed explanation.

Update exact implementation does not validate a requirement against its objective with the DynaSpec-RAG validation-sum guarantee and reported test regressions as a bounded example of a locally satisfied condition that does not warrant the stronger objective claim.