h t t p s : / / a r x i v . o r g / a b s / 2 6 0 6 . 0 4 7 5 1

Ingest: FALSIFYBENCH rule-discovery games

Type: kb/sources/types/ingest-report.md

Classification

Scientific paper by Leonardo Bertolazzi, Katya Tentori, and Raffaella Bernardi, affiliated with the University of Trento and the Free University of Bozen-Bolzano. The retained observation is the full June 2026 arXiv v1 paper, including methods, prompts, statistical analysis, and example traces. Its contribution is an empirical benchmark and process analysis, grounded explicitly in Wason's task and Klayman and Ha's account of testing strategies.

Summary

FALSIFYBENCH evaluates twelve LLMs on 100 semantic rule-discovery games each. Players propose test triples or guesses, receive feedback, and retain their interaction history for up to twenty turns. Each model also supplies its own stateless oracle through separate calls. Games use five curated WordNet target categories, with initial examples drawn from narrower descendants to encourage overly specific hypotheses. Within this construction, greater positive testing correlates with lower success across models (Spearman ρ = −0.779); the best reported success rate is 75%. Turn-level analysis distinguishes testing intent from actual falsification and associates failures with partially overlapping, disjoint, or surface-linguistic hypotheses. Crucially, positive tests can refute an overly broad hypothesis: the paper's negative-testing result is specific to the dominant overly narrow case. It measures discovery behavior inside a supplied game, without testing a persistent learning architecture or a causal intervention that prescribes negative testing.

Quotes

No source quotes have been retained yet.

Connections Found

For the Popper-grounded learning question, this source supplies a bounded empirical case of explicit conjectures, self-selected probes, and feedback-driven revision. It sharpens the existing caution in the sample-efficiency conjecture: the full methods establish that the benchmark deliberately encourages hypotheses narrower than the target, so the negative-testing association concerns informative exploration under that construction. It does not test retained-theory reuse across a shift, nor establish that an epistemic commitment caused better performance.

The paper also supplies a process-level example for known-target discovery benchmarks. Hypothesis-target relations and conclusive falsifications reveal more than final recovery alone, while the authored target and semantic-equivalence oracle still determine success. The result therefore adds evidence about reaching a known target without closing the problem of judging unknown scientific contributions.

Learning Claims (our opinion)

The player learns within an episode by carrying forward its proposed hypotheses, actions, and oracle responses in conversation history. It can change a natural-language hypothesis, choose new items, or submit a guess; an incorrect guess also provides rejection feedback. Test-strategy annotations are for later analysis and do not teach the player. The oracle receives only the current judgment inputs, and a separate offline GPT-5-Mini annotator classifies hypothesis-target set relations.

This is a partial implementation-level analogue of theory refinement. Hypotheses are explicit, semantic membership supplies criticizable consequences, and traces show revision following informative tests. Natural-language hypotheses permit inspectable changes of category or scope. However, the experiment does not isolate selective repair, preservation of useful prior knowledge, or whether the declared hypothesis causally mediates the next decision. It neither measures cross-game retention nor revises the testing machinery. No weight updates are part of the reported mechanism.

The epistemic contribution is narrower than endorsing a complete Popperian learning paradigm. The source's stated lineage is Wason and Klayman and Ha, rather than a direct implementation of Popper's philosophy. Its useful distinction is between seeking an example outside the current hypothesis and obtaining evidence that contradicts that hypothesis. For a hypothesis narrower than the target, an outside example accepted by the oracle refutes the hypothesized boundary; an outside example rejected by the oracle does not. For an overly broad hypothesis, a positive test rejected by the oracle can instead refute it. Thus attempted criticism, successful error detection, and productive revision need separate measurements.

The fixed-decomposition boundary matters here. Players can propose arbitrary strings, including non-taxonomic hypotheses; they are not restricted to choosing WordNet nodes. Nevertheless, the experiment fixes semantic membership as feedback, triple testing and guessing as actions, the target categories, the turn limit, and oracle acceptance as closure. Better play inside that arrangement does not compare alternative feedback, theory representations, retention policies, or acceptance criteria. Success shows episode-level adaptation toward the supplied target, not a general advantage for this decomposition.

Extractable Value

  • [quick-win] Scope criticism by the hypothesis-target relation. The full text resolves the existing sample-efficiency note's uncertainty about benchmark construction: initial examples deliberately encourage overly narrow hypotheses. Negative testing is particularly informative there, whereas positive testing can refute overly broad hypotheses. This is a reusable qualification for claims about falsification-seeking behavior, not evidence of persistent learning.
  • [experiment] Measure criticism at separate stages. Record the declared hypothesis, selected probe, expected response, observed response, resulting contradiction, and subsequent revision. FALSIFYBENCH distinguishes intended test direction from conclusive falsification, showing how a benchmark can inspect part of the conjecture–criticism–revision pathway instead of scoring only final answers. Extending that measurement to retained-theory reuse would be a new experiment.
  • [just-a-reference] Preserve the bounded quantitative result. Across twelve models and 1,200 games under the fixed semantic-game design, positive-testing frequency negatively correlates with success and with conclusive falsification (the latter ρ = −0.937). These are useful reference observations about active testing under predominantly narrow hypotheses; neither comparison manipulates epistemic commitments or retention.

Limitations (our opinion)

The paper's language of a primary driver exceeds what the observational comparisons isolate. Models vary in training, scale, reasoning configuration, and provider settings. Testing strategy is measured during the same games whose success it predicts; model quality and evolving game difficulty can affect both. A regression with a model random effect and an oracle-error covariate does not substitute for a matched strategy intervention.

Oracle reliability remains a consequential measurement boundary. Each model plays against itself through separate calls; human checks cover only one Test and one Guess turn per game. The resulting binary error flag cannot establish that errors elsewhere in the trajectory are inconsequential. The reported lack of a credible independent oracle-error association should not be generalized to perfect-oracle equivalence. Hypothesis-target relation labels additionally depend on an offline LLM, and surface-feature classifications use regex heuristics. These readouts are useful proxies rather than independently established semantic ground truth for every turn.

The five curated, English-centric semantic categories and deliberately narrow initial examples make the test favorable to upward category revision. Target recovery is not explanatory progress in an open scientific domain. The benchmark does not compare negative testing with a matched information-gain policy, or show that a philosophical stance explains performance beyond useful probe selection. It also supplies no controlled persistence, transfer, or forgetting test. The full paper was analyzed, but its released code was not inspected or executed here; reported outcomes have not been independently reproduced.

Update the FALSIFYBENCH discussion in the sample-efficiency conjecture using this full-text observation, replacing uncertainty about construction with the explicit narrow-hypothesis design while preserving the distinction from causal strategy interventions and retained-theory reuse.