Ingest: Many AI analysts, one dataset

Type: kb/sources/types/ingest-report.md

Source: many-ai-analysts-one-dataset-agentic-data-science-multiverse.md Captured: 2026-08-18 From: https://www.pnas.org/doi/epdf/10.1073/pnas.2606495123

Classification

Genre: scientific-paper -- a peer-reviewed PNAS experiment with controlled model and persona conditions, three dataset–hypothesis tasks, approximately 5,000 autonomous analysis runs, and transcript-level AI auditing. Domains: agentic-analysis, research-reproducibility, prompt-sensitivity, llm-evaluation Author: Martin Bertran and Riccardo Fogliato (Amazon) and Zhiwei Steven Wu (Amazon and Carnegie Mellon). The paper discloses its experimental scaffold, tasks, prompts, code, and transcripts, but the authors also designed the system and its AI auditor; credibility rests on the reported experiment rather than independent deployment evidence.

Summary

Bertran, Fogliato, and Wu run 4,946 autonomous data-analysis sessions across three fixed dataset–hypothesis–estimand tasks, four LLMs, and five analyst personas, then use a separate Claude Sonnet 4.5 auditor to screen full traces for methodological compliance. Only 3,303 runs (67%) pass. Even among compliant runs, effect estimates, p-values, and support verdicts vary widely; changing the base model or persona shifts their distributions, with Negative-to-Strong-Confirmation-Seeking support-rate gaps of 34 to 66 percentage points across tasks. Auditing removes aggressive specification search and overclaiming disproportionately, but does not eliminate steering. The authors argue that cheap automated analysis makes selective reporting easier while also making multiverse-style uncertainty mapping practical, and recommend reporting result distributions and disclosing prompts alongside code and data.

Connections Found

This paper is the first end-to-end empirical case in the KB for LLM output deviation requires three-way diagnosis. Repeated runs inside a fixed model–persona cell expose indeterminism; materially different compliant analytic pipelines show that the unchanged task admits a plural valid set; methodological rejections supply the out-of-spec category, with pilot hallucinations and recalled published findings as concrete interpreter failures. It also supports The boundary of automation is the boundary of verification, because an auditor rejects one-third of generated analyses yet leaves substantial steering among accepted runs. Relative to Prompt Stability in Code LLMs, it moves from meaning-preserving prompt variants and code correctness to end-to-end analytic pipelines and scientific conclusions. The comparison sets a boundary: confirmation-seeking prompts change the analyst's commission, so their effects are controlled steering rather than clean bias estimates under fixed semantics. The experiment's unvaried design choices remain an orthogonal coverage limit under Learning inside a fixed decomposition inherits its mistakes, not a fourth source of output deviation.

Extractable Value

  1. Use agent populations as an end-to-end specification stress test. Give many tool-using agents the same input and partial contract, inspect the outcome distribution, then extract the decision dimensions responsible for dispersion. This extends prompt-variation diagnosis from final answers to complete work pipelines and can reveal which choices a specification left implicit. [experiment]

  2. Keep output-deviation causes separate from experimental coverage. First classify repeated-sample variation, plural spec-valid analyses, and out-of-spec outputs as indeterminism, underspecification, and interpreter failure. Then report between-condition effects and alternatives excluded by the fixed decomposition as separate attribution and coverage limits. This prevents sampling noise, semantic plurality, bias, deliberate commission changes, and untested design space from being collapsed into one “variance” number. [quick-win]

  3. Audit residual distributions, not only pass rates. A 67% compliance rate sounds like quality control, but persona-conditioned outcome shifts survive inside the accepted subset. A gate can remove invalid processes without making the retained result distribution neutral or representative; downstream evaluation should therefore compare distributions before and after filtering. [quick-win]

  4. Treat prompt-treatment semantics as part of experimental validity. Negative and Positive personas mainly alter stated prior plausibility, while Confirmation Seeking explicitly requests supporting specifications and Strong Confirmation Seeking invites p-hacking-style exploration. Pooling those as generic “prompt sensitivity” would obscure that the latter treatments change the task commission rather than only its surface expression. [quick-win]

  5. Test multiverse reporting and prompt disclosure as interventions, not just norms. The proposed transparency package is plausible and operationally specific, but this experiment does not show that disclosure prevents cherry-picking, improves reviewer decisions, or identifies the representative analysis distribution. A follow-up could compare decisions made from one selected run, an audited multiverse, and the same multiverse plus prompts and decision traces. [experiment]

Limitations (our opinion)

The causal results are narrow but useful. Analysts could condition on the full dataset, hypothesis, prespecified estimand, persona prompt, model priors, run history, and tool outputs; they could compose Python, shell, file-editing, preprocessing, modeling, and inference operations. The effective hypothesis class was whatever four selected LLMs could express inside one ReAct/Inspect scaffold. The datasets, hypotheses, estimands, persona family, model family, tool surface, run cap, output protocol, decision codebook, auditor model and prompt, and compliance criteria stayed fixed. The study therefore isolates model and persona effects inside this space; it does not establish that the space represents human analysis, all defensible AI analysis, or the best decomposition for automated science.

The AI auditor is a major unverified dependency. The snapshot reports transcript-level inspection and 13 scored dimensions, but no independent human-labelled calibration of its per-run discrimination. “Compliant” therefore means accepted by this auditor, not established methodological soundness. The residual steering result is still informative—dispersion survives the filter—but neither the pass rate nor the post-filter subset should be treated as ground truth.

Coverage is limited to three tasks, four model checkpoints from two families, one agent scaffold, and one auditor. The high-contamination soccer task may activate memorized analyses; the newer METR task and constructed ANES task reduce that concern but do not supply a human many-analyst baseline. The authors explicitly do not claim that their AI-generated multiverse is interchangeable with a human-expert multiverse.

Finally, the experiment establishes that model and persona treatments change outcomes, not that multiverse reporting or prompt disclosure corrects selective reporting. It also does not code-ground or reproduce the reported results in this ingest: public code and transcripts are named in the paper, but no repository revision was inspected or executed.

Run a code-grounded follow-up on the paper's public repository and transcripts. Classify a stratified sample of accepted and rejected runs under the three-question diagnosis, test whether the auditor's rationales discriminate plural spec-valid analyses from interpreter failures, and keep fixed-decomposition coverage as a separate audit result.