Ingest: The Alien Space of Science: Sampling Coherent but Cognitively Unavailable Directions

Type: types/ingest-report.md

Classification

The source is a methods paper with empirical evaluation (arXiv v3, marked as NeurIPS 2026). It defines a representation and two learned scorers, then compares them with frontier LLM ideation baselines, human ratings, LLM-judge tournaments, autoresearch runs and a frozen temporal backcast.

Author: Artiles, Weiss, Brinkmann, Rahwan, Schölkopf, Pal, Larochelle, Goyal and Rahaman, from Tiptree Systems, the Max Planck Institutes for Human Development and Intelligent Systems, Mila and Université de Montréal. The author set is strong for ML and computational social science. Two authors co-founded Tiptree, which sells research-discovery tools, so the authors have a commercial interest in the framing.

Summary

The paper argues that LLM ideation inherits the "community prior": prompted for novel ideas, models recombine concepts that the current research community is already positioned to produce. It targets the complementary region, directions that are coherent but cognitively unavailable. An LLM distills 16,068 LLM-research papers into short methodological "conceptual units", which are clustered into a vocabulary of 273 "idea atoms". Two small models are trained over atom sets. The coherence model is an autoregressive transformer that scores whether atoms plausibly co-occur in one paper. The availability model is a contrastive dual encoder that scores whether a community of existing authors (median of the top-m author matches) is positioned to produce the combination. Candidates are ranked by a weighted sum of z-scored coherence and unavailability (β = 0.7 for the main runs), and an LLM decodes only the selected triples into idea sketches. All comparisons hold the atom vocabulary and the decoder fixed: the LLM baselines also select triples from the same 273-atom catalogue, so the tests compare selection policies inside one decomposition, not alien sampling against open-ended LLM ideation. Within that frame, alien portfolios are more diverse than Claude and Gemini selections, including batch-prompted ones (Vendi 225 versus at most 198). Per-idea quality is tied with Claude on 10 human-rated ideas per method and leads on LLM-judged tournaments and autoresearch report rankings. In the frozen pre-2025 test, the top 0.875% of 1.14M triples recovers 2025 combinations at 35.9× chance with coherence alone. Strictly novel combinations are recovered at only 2.35× chance, rising to 3.24× when unavailability is weighted at β = 0.4. The authors call this "an early proxy rather than a final test".

Contribution assessment (our opinion)

The problem matters: if AI ideation reproduces the field's prior, it accelerates available work and adds little exploration. The main contribution, tentatively, is the conceptual separation of plausibility from community availability, made operational as two independently testable scorers. Each scorer has its own falsification check: the availability model must match unseen 2025 authors to atom combinations they never published, and the coherence model must rank future combinations. That is a stronger design than prompting a model to "be novel" and judging the output with another model. The most decisive evidence is the frozen temporal test. It shows that cheap learned scores concentrate future combinations, but most of that headline is recovery of non-novel recombinations. The novel-recovery gain from the availability term is real but modest (29 to 40 hits), so the evidence that availability steering finds productive directions is thin. Quality evidence is weaker still: ten human-rated ideas per method, with the rest resting on one LLM judge calibrated against experts on ten report pairs. The originality claim relative to Sourati and Evans' hypergraph availability and to keyword-level recombination systems is argued in the paper and partly tested (Appendix A.9). I cannot judge coverage of other prior work. My estimate is a useful framing and tool, with demonstrated value for portfolio diversity inside a bounded vocabulary, and unproven value for producing better science.

Quotes

No source quotes have been retained yet.

Connections Found

The paper is a technical anchor for the KB's synthesis-oracle cluster. Its strongest role is evidence for Automated synthesis is missing good oracles (is-evidence-for). The coherence model is a learned, cheap, first-tier filter for recombinative synthesis: it ranks millions of candidates without per-candidate LLM calls, and costlier human, LLM-judge, autoresearch and temporal checks run only on the survivors. That partly answers the note's open "tiered oracle" question, within a bounded domain (one subfield, 273 atoms) and with feasibility checked only late. The same structure is a worked instance of Candidacy evidence licenses escalation to assessment, not acceptance (is-evidence-for): the scores route atom combinations to decoding and autoresearch, and the authors make only a distributional claim, not a claim that any routed idea is correct. It is also evaluated the way A search controller is tested by what it brings to stronger evaluation prescribes (is-evidence-for): by matched comparison of what each selection policy routes into stronger evaluation.

The temporal backcast bears on Known-target discovery benchmarks show reachability, not discovery closure (is-evidence-for). Unlike GIANTS, which supplies parent papers, this backcast ranks the whole candidate space, so it exercises candidate triage. Its split between overall recovery (35.9×) and strictly novel recovery (2.35–3.24×) makes the note's caution concrete: most of what a backcast recovers was already within reach. Uses of this paper as evidence of prospective discovery should cite the novel-recovery figures, not the headline.

The paper's observations on LLM homogenization support Decorrelated reviewers still share the field's prior (is-evidence-for) from the generation side. Frontier models select from the same narrow concept region, and under iterative autoresearch Claude's runs drift from 2/10 to 8/10 interpretability-centred reports. The note's prediction cuts back at the paper, which relies on LLM judges for most of its quality evidence about deliberately low-availability ideas.

The atom representation compares with Short composable notes maximize combinatorial discovery (compares-with, axis: the unit of recombination and how combinations are chosen). Both treat discovery as recombining small self-contained units. The note's units are authored and selected by recognition; the paper's units are distilled, clustered and selected by a learned objective. "Cognitive availability" also compares with Information value is observer-relative (compares-with, axis: what makes coherent structure inaccessible). The note relativizes accessibility to one observer's knowledge and tools; the paper estimates it at community scale from realized publications.

Among sources, Geometry of Knowledge (compares-with, axis: mechanism for escaping LLM homogeneity and whether a quality filter accompanies it) broadens generation by embedding perturbation and admits it has no quality detector; alien sampling pairs its off-prior pressure with a learned coherence filter. GIANTS (compares-with, axis: how temporal provenance validates scientific ideation) is the conditional backcast that this paper's whole-space backcast contrasts with. LLMs can't jump (compares-with, axis: recombination versus premise invention) places this method on the recombinative side: by its own limitation, it cannot produce new primitives. Derived from the external source: arXiv 2603.01092 (derived-from).

Extractable Value

1. A learned first-tier synthesis filter, with a measured cost split -- The scorers (a 42.8M-parameter transformer and a dual encoder trained in hours) rank 3.35M triples in under 15 minutes, and the LLM is called only to decode selected candidates. This is a concrete existence case for the tiered oracle that Automated synthesis is missing good oracles leaves open. Its reach is bounded: it needs a corpus large enough to train scorers and a vocabulary small enough to enumerate. The operator's DayDreaming experiment is a local instance of the exhaustive design this filter would replace: it enumerates every combination from a small concept pool and evaluates every rendered essay, while its goals document defers ranking heuristics and a cheap verifier to future work. [quick-win]
2. Separate plausibility from community availability as two scores -- High-reach framing: novelty judgments conflate "could this work" with "would anyone here think of it". The paper shows the two axes can be learned and validated separately, and that random recombination remains surprisingly available in a dense field, so reaching the low-availability tail needs explicit search. For KB synthesis this suggests scoring candidate connections both for coherence and for distance from what the KB's existing notes already co-cite. [experiment]
3. Whole-space backcast with a novel/non-novel split -- The temporal test design (freeze vocabulary and scorers on pre-cutoff data, rank all candidates, report enrichment separately for strictly novel targets) is a reusable evaluation method. It distinguishes triage that rediscovers available work from triage that reaches new combinations, which Known-target discovery benchmarks show reachability, not discovery closure needs as a positive example. [quick-win]
4. Attractor collapse during iterative refinement -- Ideas not framed around interpretability drifted toward it during autoresearch for LLM-selected starting points (Claude 2/10 to 8/10), but not for alien-selected ones (0/10 to 0/10). This is an operational warning for any iterative LLM refinement loop, including KB revision loops: homogenization can happen during execution, not only at generation. The counts are small and the mechanism is the authors' interpretation. DayDreaming's draft-to-essay refinement stage is one concrete place where this drift could undo what the combination stage opened up. [just-a-reference]
5. Community-level availability via median of top-m matches -- Taking the median over the best-matching authors, rather than the single best, stops one prolific generalist from marking a combination as available. This is a small, transferable design choice for any "is this already covered" score over contributors. [just-a-reference]

Limitations (our opinion)

The tested comparison fixes the decomposition: all methods choose triples from the same 273 LLM-distilled atoms and the same Claude decoder verbalizes them. The results therefore show a better selection policy within that representation. They do not show that atom-level alien sampling beats free-form LLM ideation, retrieval-augmented systems, or researchers. They also do not test whether the atom vocabulary itself (HDBSCAN cluster granularity, the 80% coverage point, k = 3) is the right decomposition. β = 0.7 was chosen by an originality sweep judged by an LLM with the same decoder, so the selection objective and the evaluation share an LLM judge's notion of originality. Human evidence is ten ideas per method, and Claude matches alien on overall quality and beats it on coherence and promise. The LLM-judge calibration covers ten report pairs, with a wide confidence interval (54–80%). The judge is a frontier model that Decorrelated reviewers still share the field's prior predicts will under-credit original, low-availability claims, which would bias against the paper's method, though how much is unknown. The headline 35.9× recovery mostly reflects coherence-driven rediscovery; the availability term trades total recovery for a modest gain in novel hits. A simpler account fits much of the diversity result: any objective that penalizes popular atoms would spread selections across the vocabulary, and the paper's own comparison with uniform random shows that diversity alone is easy. The coherence constraint and the downstream quality checks are what separate the method from that account, and those checks are the weakest part of the evidence. The domain was chosen for near-complete author coverage and is one dense subfield. Availability is a publication-derived lower bound and cannot distinguish neglected directions from ones avoided for ethical reasons. The method recombines only existing atoms, so it says nothing about inventing new primitives.

Recommended Next Action

Update Automated synthesis is missing good oracles: add this ingest under current attempts as a learned first-tier coherence filter, and revise the "tiered oracle" open question to record what it shows (cheap learned ranking ahead of costly decoding and assessment in a bounded, enumerable vocabulary) and what remains open (weak evidence that the filtered survivors are better science, with novel-recovery gains of 2.35–3.24× chance).