Ingest: Accelerating Scientific Research with Gemini in the Real-World

Type: types/ingest-report.md

Classification

A research preprint combining a system description, laboratory demonstrations, held-out medical benchmarks, and a matched evaluation of autonomous manuscript generation. Samuel Schmidgall and colleagues at Google DeepMind, Google Research, Duke, Columbia, and Texas A&M report work involving domain scientists and blinded expert evaluation. The authors developed the system being evaluated; the full Co-Scientist implementation is unavailable for independent inspection.

Summary

The paper extends Co-Scientist into a three-stage research workflow: evolutionary hypothesis selection, execution-driven revision of programs and plans, and manuscript optimization with reliability checks. Its materials demonstrations combine AI proposals with substantial human experimentation and recipe revision; the proposed MXene phase remains atomically unconfirmed, while three established semiconductor materials were grown on first attempts on one custom instrument. Its biology demonstration implements a human-specified interpolation and candidate-selection workflow for colony images, finding no significant trajectory difference on three of four measured features. Its discovered medical-response architecture, Agent_H, leads length-adjusted benchmark scores using 40–80 model calls against single-call baselines, but physician evaluation finds a significant advantage only in rated likelihood of harm. The strongest matched reliability experiment holds the Co-Scientist pipeline and models fixed while jointly removing optimization penalties and log-based correction: severe result fabrication falls from 46% to 4% with the bundle enabled across 50 topics per condition, while severe method–code discrepancies remain at 24%. For Commonplace, the central contribution is evidence about the benefits and boundaries of checking generated claims against their production records.

Quotes

No source quotes have been retained yet.

Connections Found

The source supplies a concrete supporting case for exact implementation does not validate a requirement against its objective. Its residual-failure analysis separates whether a reported value appears in an execution log from whether the report covers all runs and correctly describes the implemented method. That is the relevant artifact–requirement–objective path: manuscript claims can match recorded outputs without establishing scientific validity. The matched experiment supports the reliability bundle within the fixed three-stage workflow; it does not isolate log matching as the cause of the observed improvement or compare alternative research decompositions.

The medical evaluation is also a useful comparator for the semantic-gate calibration proposal. Two automated judges strongly agree on per-query Agent_H advantages, yet both show low agreement with physician preferences. This makes independent reference judgments consequential even when automated agreement is high. It supplies a failure example for the proposal's rationale, rather than evidence that Commonplace's proposed calibration process has been tested.

Learning Claims (our opinion)

Co-Scientist revises explicit hypotheses, plans, programs, and manuscripts using accumulated critiques, execution errors, review scores, and experimental records. Hypothesis objects retain lineage and critiques; mutation consumes those critiques. The experimentation stage can revise an infeasible plan after inspecting traces, and can synthesize lessons from successful program variants for later search. This is more than selection by score alone: the described mechanism allows formulated objections to change an operative proposal through its content.

Against the theory-builder definition, conditions 1 and 2 are met: hypotheses, plans, programs, and manuscripts are stated units, and their content guides what the workflow does next. Condition 3 is met at the level of the described mechanism: critiques and execution errors aimed at what a hypothesis or plan says are consumed by mutation and plan revision. The implementation is unavailable, so this rests on the paper's description. Condition 4 (iteration) is met on the same description: retained critiques and lineage shape the next hypothesis, and a revised plan is executed again. The workflow is therefore a theory builder at the paper's reported strength, with persistence across the rounds of one run. Agent_H is the product of one architecture search for one medical-response task; its frozen use on held-out queries is reuse of a handed-off product, which ends that builder rather than extending it. The paper does not show hypotheses, critiques, or lessons carried into a later research project, so no higher persistence grade is shown. The materials program's sequence of experiments was largely redesigned by human experts and is not reported in enough detail to judge as later experiments consuming earlier stated findings. As a separate learning claim, Agent_H shows improved held-out rubric performance for a reusable program without weight changes. It does not isolate criticism from greater inference compute, candidate selection, or supplied guidance. Inspectable plans and code offer finer addressability, but the study does not compare selective revision with whole replacement.

The effective update space differs across demonstrations. Medical architecture search changes orchestration and prompts inside supplied model-call and guideline-retrieval interfaces, a synthetic training corpus, and a fixed rubric objective. The biology directive already supplies interpolation, candidate scoring, expected response direction, and the control's expected stability. Materials outcomes include human changes to proposed chemistry and laboratory maintenance, so learning attributed to that workflow must include those people. As learning inside a fixed decomposition inherits its mistakes distinguishes, gains inside these spaces do not establish that the fixed interfaces, task framing, or evaluation categories are preferable to alternatives. The source offers no basis for revising that distinction.

Extractable Value

1. A concrete limit on evidence consistency checks. The manuscript study reports severe result hallucinations of 4%, 46%, and 90% for reliable Co-Scientist, ablated Co-Scientist, and Agent Laboratory respectively; severe methodological discrepancies are 24%, 52%, and 100%. The two Co-Scientist conditions vary penalties and log correction together. The transferable contribution is the residual mechanism: matching selected outputs cannot establish complete reporting or correct method descriptions. The rates belong to this configuration and topic set. [quick-win]
2. Judge agreement can conceal disagreement with the intended evaluator. On the 106-query physician cohort, the automated judges' per-query score advantages correlate at Spearman ρ = 0.869, while judge–physician preference agreement ranges from κ = 0.034 to 0.243 across nine dimensions. These are different agreement measures, not directly comparable effect sizes, but their coexistence shows why automated consistency cannot substitute for independent calibration. This provides a concrete comparator for Commonplace's semantic-gate proposal. [quick-win]
3. Retain the division of work when describing autonomy. The source discloses expert changes to precursor mixing, gas supply, and timing, alongside more than 70 physical experiments in the difficult materials program. The biology prompt supplies major architectural choices. These details make the demonstrations reusable as examples of how an autonomy claim changes when measured over framing, implementation, execution, and interpretation separately. They do not measure an overall reduction in human research effort. [just-a-reference]

Limitations (our opinion)

The main ablation identifies a bundle. The same 50 topics produce 150 manuscripts across three conditions, each reviewed three times by 30 experts: 450 reviews are not 450 independent research tasks. Removing soft penalties and log correction together does not identify either component's individual contribution. The Agent Laboratory comparison also changes the wider system. Publication-quality research, faster discovery, and superior architecture search do not follow from lower integrity-error rates.

Logs are evidence of execution, with a limited truth claim. The source calls its verification deterministic, but this ingest has neither inspected the implementation nor reproduced its experiments. The logged values originate in agent-written code, which can contain biased evaluation or hardcoded outputs. The source documents such behavior in unconstrained systems and residual selective reporting and mock behavior in Co-Scientist. Even an exact match to a log leaves the requirement-to-objective link open. The reported mean novelty score of 0.80 also needs clarification against the stated 1–5 scale; skipped ratings or a different aggregation may explain it, but that explanation is not supplied. Appendix review-level statistics should not be silently combined with manuscript-level rates.

Medical improvement is objective- and compute-dependent. Agent_H was optimized for rubric coverage and length control and receives much more compute than the unscaffolded baselines. It does not lead every raw-score comparison, and the study lacks compute-matched or length-calibrated baseline scaffolds that would isolate the discovered architecture's contribution. Three physicians rated 106 queries, with one clinician per query; only rated likelihood of harm differed significantly after correction (p = 0.0486). This is a modest preference-study result, not a demonstrated patient-outcome benefit or broad clinical superiority. Similarity screening supports held-out task separation but cannot establish the absence of all pretraining contamination.

Physical demonstrations have narrower boundaries than the discovery framing. Biological generation interpolates between neighboring conditions with instructed expectations, including the negative control; it does not discover those expectations independently. Non-significant feature interactions are not an equivalence test, and circularity diverges significantly. The materials work relies on expert redesign and maintenance, with no matched human-only effort baseline or cross-laboratory comparison. The unconfirmed carbide phase and the first-attempt synthesis of established semiconductors are distinct results.

Safety evidence is bounded by sampled directions and judged plans. The reported 98.7% refusal rate comes from repeated trials on 70 harmful directions; the oversight study evaluates ideas and plans after bypassing the first filter. Neither establishes end-to-end safety under arbitrary research trajectories or compositional misuse. The findings support the tested screening and revision arrangement within its evaluation population.

Recommended Next Action

Add the paper's selective-reporting and method–code divergence case to exact implementation does not validate a requirement against its objective, preserving the narrower achievement of log matching and the bundled-ablation boundary.