h t t p s : / / a r x i v . o r g / a b s / 2 5 0 2 . 0 9 8 5 8

Ingest: Automated Hypothesis Validation with Agentic Sequential Falsifications

Type: kb/sources/types/ingest-report.md

Classification

Scientific paper by Kexin Huang, Ying Jin, Ryan Li, Michael Y. Li, Emmanuel Candès, and Jure Leskovec, captured as the February 2025 arXiv v1 manuscript. It combines a conditional statistical theorem, an LLM implementation, benchmark comparisons, ablations, and a small expert study. Its evidence concerns automated hypothesis validation, not a complete autonomous discovery process.

Summary

POPPER takes a supplied natural-language hypothesis, proposes measurable tests, executes Python analyses, and accumulates statistical evidence until a threshold or budget ends the run. A design agent revises proposals, an LLM relevance checker screens their connection to the main hypothesis, and an execution agent returns p-values. Converting these to e-values permits sequential aggregation with Type-I error control, provided the null implications, conditional statistical validity, and stopping assumptions hold. Within a fixed design/execution/checker architecture using static datasets, the paper reports better error control and power than general code-generation and reasoning baselines, with ablations supporting particular component choices. Nine expert participants provide a narrow comparison suggesting similar validation performance and a 9.7-fold time advantage. The strongest reusable contribution is the explicit separation between proposing tests, checking their meaning, and aggregating their results; the formal guarantee does not establish that generated tests satisfy its premises or that a validated hypothesis is true.

Quotes

No source quotes have been retained yet.

Connections Found

For reconsidering the learning paradigm, POPPER is a concrete partial realization of the discovery lifecycle: it proposes consequences and tests them for a supplied conjecture, without demonstrating revision and retention of the main theory. Its tests can change while the main hypothesis and validation architecture remain fixed. The benchmark improvements therefore support this particular validation arrangement, not the superiority of that decomposition over alternative learning arrangements.

It also provides a bounded comparison with verification needing a typed target. Figure 11 shows an execution agent using evidence about expression of RAB39A to support a claim about regulation of IL-2. The lesson concerns preserving the identity of the assessed claim through execution. It complements the note's distinction between classifying a target and establishing semantic adequacy; it does not directly test the note's claim that artifact classification precedes reusable verification.

Learning Claims (our opinion)

The adaptation is within a validation episode. The design agent sees the main hypothesis, dataset metadata, prior sub-hypotheses, and their p-values. It proposes and critiques new tests; the execution agent writes code, observes results, and repairs execution errors. These operations change explicit test descriptions and programs without a reported update to the underlying LLM weights. The accessible hypothesis space for tests is therefore broader than choosing among a prewritten list, although it remains bounded by available datasets, tools, prompts, and the required p-value interface.

Relative to theory refinement, this demonstrates theory use and local revision of proposed tests more clearly than revision of the domain theory. The main hypothesis supplies the task and remains the claim being validated. The paper does not show a failed consequence causing an edited main hypothesis to be retained and reused. Its Self-Refinement prompt is consequently not evidence of reflective theory refinement: critiquing a test proposal does not by itself revise a causally connected theory of the system's own organization.

The Popperian mapping is partial. The formal procedure seeks to reject a null associated with the supplied claim; its terminal output is validated or unvalidated. That statistical acceptance rule is not equivalent to establishing truth, nor does lack of validation refute the supplied claim. The paper supplies computational machinery for one part of criticism and testing, rather than an empirical comparison of philosophical accounts of learning.

As learning inside a fixed decomposition predicts, different boundaries must be assessed separately. Test descriptions, statistical methods, and execution code can vary; the main claim, data access arrangement, component roles, and evidence-to-decision interface remain outside the episode's revision scope. The ablations vary the relevance checker, statistical combination, or coding method, and the model comparisons vary the inference model. They do not compare a learner that can revise those boundaries with one that cannot. The source thus supports adaptation without weight updates, but provides no comparative result against model training and no demonstration of persistent learning across tasks.

Extractable Value

  • Changeable tests and changeable theories are distinct capabilities. [quick-win] Use POPPER as a concrete case when assessing what a learning system can revise: generated tests and executable analyses change while the supplied main hypothesis remains fixed. This sharpens the distinction between consequence testing and theory refinement without requiring a new learning category.
  • A formal aggregator cannot repair a wrong claim-to-test relationship. [quick-win] Sections 2.3 and 3 expose the premises needed for the statistical guarantee; Figure 11 shows an implementation failure at the semantic interface. This is useful beyond the benchmark wherever an interpreted claim is translated into an executable check.
  • Conditional sequential validity is a possible assessment mechanism. [experiment] The p-to-e transformation and cumulative product support adaptive stopping when their premises hold. Applying that mechanism to KB claim evaluation would require actual valid statistical tests and defensible null implications; ordinary model verdicts do not meet those requirements merely because they are repeated.
  • Component evidence is narrower than paradigm evidence. [just-a-reference] Removing the relevance checker worsens reported Type-I error, while substituting direct code generation reduces power relative to ReAct within this setup. These comparisons support the varied components under the tested data and task arrangement, not the fixed overall architecture or a general advantage of Popperian learning.

Limitations (our opinion)

The theorem is conditional; the implementation uses fallible LLM judgments to approximate its implication premise. Section 4.2 reports that human evaluators rated fewer proposals as strongly implied than the relevance checker did. Figure 11 makes the risk concrete: a very small p-value about RAB39A expression is treated as evidence about IL-2 regulation. Numerical strength cannot supply the missing logical connection.

Conditional validity is a separate obligation. Hiding raw data from the design agent helps constrain test selection, but the execution agent still inspects data and chooses statistical procedures. The paper's theorem requires conditionally valid outputs; it does not prove that every generated analysis meets that condition. Reported error categories include misinterpreted p-values, poor experiment design, broken implications, and incorrect implementations.

Benchmark negative examples are produced through column-wise permutations. Error control under these constructed nulls does not establish equal performance on naturally false hypotheses with realistic confounding or dependence. The main comparisons use static data analysis, so laboratory execution and intervention remain extensions of the framework rather than demonstrated outcomes. General-purpose baselines also lack specialized statistical safeguards, leaving open how POPPER compares with stronger alternative statistical workflows.

The nine-person expert study is too small to establish equivalence with human scientists; absence of a significant performance difference is not an equivalence test. Its time advantage applies to the supplied validation tasks. No implementation code was inspected or executed for this ingest, and the reported experimental outcomes were not independently reproduced.

Finally, per-hypothesis Type-I error control does not control the proportion of false discoveries among accepted claims. Appendix C explicitly acknowledges this distinction and leaves evaluation of broader multiple-testing arrangements to future work. Neither the guarantee nor the benchmarks warrant treating the procedure's acceptance decision as final knowledge.

Review Theory refinement against POPPER as a boundary case, checking that its distinction between applying a theory, revising tests, and revising the theory remains explicit.