h t t p s : / / a r x i v . o r g / a b s / 2 6 0 6 . 0 6 4 5 4

Ingest: Scaffold, Not Vocabulary? A Popperian Code-Generation Skill

Type: kb/sources/types/ingest-report.md

Classification

Scientific preprint by Mehmet İşcan, affiliated with PythaLab, Yıldız Technical University; the captured paper is arXiv v1, dated 4 June 2026. It reports controlled prompt ablations, executable correctness tests, and a judge audit. The author reports preregistration through repository commits and release of prompts, outputs, and analysis code. Those release and chronology claims were not independently checked in this ingest.

Summary

The paper compares a full Popperian coding prompt with vanilla prompting, a labels-only scaffold, and a length-matched best-practices placebo using HumanEval+ execution tests. On Claude Sonnet 4.6 through Claude Code, all four conditions score 95.1–96.9% across 163 tasks, leaving little headroom. On Qwen2.5-Coder-0.5B, full and labels-only prompts each reach 56.7% best-of-eight correctness, versus 34.8% vanilla and 54.3% placebo across 164 tasks. The small-model controls exist only at best-of-eight, and their confidence intervals do not establish equivalence. Thus the added procedural bundle has no detected advantage in these comparisons; the experiment does not isolate terminology or establish that scaffold structure alone causes the gains. A same-model selector does not beat random choice, while a reproducible rubric judge awards substance points to an empty vocabulary sentinel. The reusable contribution is a control design for testing methodology-flavored prompts, not a test of Popperian epistemology or persistent learning.

Quotes

No source quotes have been retained yet.

Connections Found

For reconsidering the learning paradigm, this source is a boundary case: it evaluates supplying a fixed procedure, rather than an agent revising and retaining its own learning machinery. It supplies concrete evidence for an experiment identifying only its actual contrast. Full versus labels-only varies procedure, density, and length together; low-tier labels and placebo controls do not resolve single-sample benefits. The best-of-eight gain therefore cannot establish that Popperian reasoning causes improvement, nor can non-separation refute that broader methodology.

The judge sentinel is a counterpoint to reliability dimensions as oracle-hardening stages: intra-rater ICC of 0.959 coexists with a 10/20 substance score for an output containing labels but no code or procedure. Repetition establishes consistency without establishing that the instrument measures the intended property. This distinction matters when Commonplace uses a repeatable model judgment as evidence for learning.

Learning Claims (our opinion)

The proposed skill treats programs as conjectures, asks for severe tests and revision, discourages ad hoc rescue through a description-length rule, and declares test survival provisional. These are instructions for an epistemic process. The evaluated realization supplies the prompt to fixed generators, extracts generated functions, and grades them with a fixed external test suite. At the low tier, eight candidates are generated independently and a separate same-model rubric call selects among them. The paper's correctness pipeline does not demonstrate a repeated trajectory in which execution failures drive program repair, skill revision, or test revision. Its proposed successor based on generating and executing counterexamples remains future work.

Relative to theory refinement, a generated program could be an addressable tentative theory when its parts have testable consequences and are revised against cases. This study does not establish that operation merely by calling programs conjectures: its main observations concern prompt-conditioned production and candidate selection. It also supplies no retained revised theory whose later use can be compared. Fixed model weights would not preclude learning through external artifacts, but no persistent artifact-learning effect is measured here.

The effective update space is consequently important. The experimenter changes prompt packages across conditions; the generator can vary code, and the self-judge can choose one of eight candidates. Benchmark specifications, external tests, extraction rules, sampling budget, and selection interface stay outside that process. These controls make a local comparison interpretable but leave alternative task representations, feedback loops, and revisable testing arrangements untested. The full-versus-labels contrast tests the supplied package within that fixed arrangement, not which decomposition is best for learning.

Reflection remains a separate question. A rubric judging its own model's outputs is self-evaluation, not evidence of revising a causally connected representation of the learner's own organization. The author interprets the experiment itself as Popperian testing of a Popperian skill; that methodological self-application likewise does not establish autonomous reflective theory refinement. The source strengthens the demand for discriminating tests of an implemented process. It neither requires a change to Commonplace's definition of refinement nor supplies outcome evidence for Commonplace's broader learning paradigm.

Extractable Value

  • Separate epistemic discipline from its prompted realization [quick-win]. A skill can instruct conjecture, criticism, and provisional acceptance without the experiment establishing execution-guided repair, retained learning, or reflection. This source gives a concrete comparison case for keeping those claims distinct when assessing the paradigm.
  • Use matched controls at the claimed operating point [experiment]. Full, labels-only, and length-matched generic prompts plus an independent outcome measure provide a reusable design. Here the small-model best-of-eight gain does not identify the source of the single-sample gain because labels-only and placebo were not evaluated at that operating point. Any Commonplace adaptation needs controls on its actual revision-and-reuse loop.
  • Test judge validity separately from repeatability [quick-win]. A content-free sentinel received substantial substance credit from a highly repeatable judge. A corresponding probe could test whether Commonplace reviewers reward expected terminology without the required mechanism or evidence.
  • Measure selection against a matched baseline [experiment]. The small-model self-judge achieved 25.6% correctness against a 24.9% analytic random expectation and 26.8% seeded random draw from the same candidates. A richer critique rubric alone did not demonstrate useful selection signal in that setting.

Limitations (our opinion)

The title and several passages favor a stronger structure-versus-content story than the comparisons establish. Full versus labels-only is a bundle contrast; the placebo contains generic best practices rather than inert text. At the low tier, the full-minus-labels 95% interval is −6.1 to +6.1 percentage points and full-minus-placebo is −4.3 to +9.2. Both exceed the declared ±5-point equivalence margin. Non-separation leaves meaningful effects open. The manuscript also defers its quantitative prompt-token-count table, limiting inspection of length matching from the paper alone.

The high tier has a ceiling and an incompletely observed Claude Code harness even in its vanilla condition. The low tier uses one very small model and one eight-sample budget. Serving path, task count, sampling, and treatment of one benchmark problem differ between tiers, so their difference is not a controlled capability comparison. A small model can have headroom while lacking the capability to execute a proposed procedure; its failure is not a decisive screen for stronger models.

Only the four main prompts receive the execution evaluation; intermediate procedure/severity additions and the verificationist control receive exploratory rubric evaluation. That stage uses three authored cases, has moderate cross-judge agreement, and lacks some early per-cell data. Neither its component scores nor the fixed-order selection-index concentration identify general causes. Position bias needs an order-randomized comparison, and this one failed selector does not prove that every useful verifier must be stronger than its generator.

Executable tests avoid the rubric's vocabulary reward but retain specification, extraction, coverage, and test-bug assumptions. The paper itself calls augmented tests an engineering analogue of statistical severity, not a full implementation with an explicit error model and detection probability. No code was inspected or executed for this ingest, and no reported result was reproduced. Claims about persistent theory learning, revised tests, or reflective machinery remain outside the experiment.

Add a bounded comparison case to An experiment identifies only the contrast it runs, using this study to distinguish a fixed Popperian prompt ablation from a test of execution-guided, retained theory refinement.