h t t p s : / / a r x i v . o r g / a b s / 2 6 0 7 . 1 9 5 9 2

Ingest: Knowledge-Centric Self-Improvement

Type: kb/sources/types/ingest-report.md

Classification

ArXiv preprint with a defined curation protocol, benchmark setup, agent-centric and prompt-optimization baselines, cross-LLM tests, and held-out transfer experiments. Author: Xuefei (Julie) Wang, Lauren Hyoseo Yoon, Chengrui Qu, Amanda Zichang Wang, Atharva Sehgal, Eric Mazumdar, and Yisong Yue (Caltech); the source is a current research preprint with public code at recursive-knowledge/KSI, not a peer-reviewed final publication.

Summary

The paper proposes knowledge-centric self-improvement: keep task-solving agents generic and disposable, and make a shared curated knowledge base the persistent object of improvement. Each attempt contributes structured evidence to a task-level forum; a cross-task forum tests which claims transfer beyond their originating task; and distillation retains scoped, actionable, evidence-grounded guidance for later agents. Across Polyglot, SWE-bench Pro, ARC-AGI-1, ARC-AGI-2, and Terminal-Bench 2, the authors report higher solve rates and lower costs than several agent-centric or prompt-optimization baselines using the same general task-solving models. A separate held-out transfer experiment freezes the distilled artifact and gives it to recipient agents with no new forum or distillation; positive transfer across Polyglot and ARC-AGI-1, including across LLM families, is the paper's strongest evidence that the external artifact—not only a persistent agent or adaptation run—carries reusable value.

Quotes

  • Source extract (verbatim): To isolate persistent curated knowledge as the improvement mechanism, we keep agents generic, stateless, and disposable. Each agent starts with a fresh context, receives the relevant distilled knowledge bundle, attempts one task with standard tool use, and contributes its experience back to the shared knowledge base. Agents do not carry persistent private memory, role specialization, task-specific architecture, or custom orchestration. The only object that changes is the curated knowledge base.
  • Source location: §1, “Introduction,” controlled-design paragraph
  • Source extract (verbatim): The knowledge base stores three artifacts: a typed attempt table, forum posts, and distilled bundles.
  • Source location: §3.1, “System Overview,” artifact inventory
  • Source extract (verbatim): The attempt table records each attempt and outcome.
  • Source location: §3.1, “System Overview,” artifact roles
  • Source extract (verbatim): Distilled bundles compress the surviving claims into reusable guidance for future agents.

  • Source location: §3.1, “System Overview,” artifact roles

  • Source extract (verbatim): We instantiate this design with a deliberately simple three-stage curation protocol, organized as threaded discussions in which agents post claims, cite one another, and reply with supporting or challenging evidence. After task execution, each agent posts to a task-level forum, turning its attempt into evidence-grounded local claims about what worked, what failed, and which constraints or hypotheses mattered, which peer agents then support or challenge. A cross-task forum then brings these claims into open discussion across tasks: agents respond to peers’ posts, and recurring principles, disagreements, and failure modes surface through this exchange.

  • Source location: §1, “Introduction,” protocol overview
  • Source extract (verbatim): The cross-task forum decides which local observations should survive beyond the task that produced them. Agents discuss task-level posts from the current generation and propose claims that recur across tasks, such as common error types, verification strategies, invariants, environment assumptions, or decomposition patterns. To prevent generic advice from accumulating, each cross-task claim must be grounded in concrete evidence from one or more attempts, and later posts take an explicit stance toward prior claims: agree, disagree, or synthesize.
  • Source location: §3, Stage 2, “Cross-task forum”
  • Source extract (verbatim): Stage 3: Distillation. The third stage turns the accumulated evidence into the artifact consumed by future agents. Distillation produces per-task bundles and a cross-task bundle. Both bundle types use the same typed fields, namely transferable insights, confirmed constraints, rejected hypotheses, pitfalls, checks, and next steps.
  • Source location: §3, Stage 3, “Distillation”
  • Source extract (verbatim): Distillation is therefore designed as a selection step that is tailored to the new task rather than a generic summarization step: the distiller LLM is instructed to keep claims that are actionable, evidence-grounded, and scoped, and to drop vague advice that does not name the condition under which it applies.

  • Source location: §3, Stage 3, “Distillation”

  • Source extract (verbatim): The two forum stages take the familiar form of an online discussion board: each task has its own thread, a shared thread spans all tasks in the generation, and agents post evidence-grounded claims, cite earlier posts by id, and reply with supporting or challenging evidence.

  • Source location: §3, protocol overview after Figure 2
  • Source extract (verbatim): Each agent receives the typed attempt table for its assigned task, the per-task distilled insights for that task, and the cross-task distilled insights from the global knowledge base.

  • Source location: §4.1, “Agent-Centric Baselines,” benchmark protocol

  • Source extract (verbatim): ARC-AGI-1 and ARC-AGI-2 [ 7, 8 ] evaluate abstract visual reasoning by requiring exact-grid reconstruction from a few input-output examples, and we sample 50 tasks from each benchmark using the official exact-match scoring protocol. Terminal-Bench 2 [ 19] evaluates agents in 89 real terminal environments with containerized tasks, human-written reference solutions, and verification tests.

  • Source location: §4.1, “Agent-Centric Baselines,” benchmarks and protocol
  • Source extract (verbatim): For swebench_pro, our framework captures the resulting workspace diff as the candidate patch and evaluates it with the SWE-bench Pro harness. For polyglot, our framework evaluates the edited exercise workspace using the language-specific test command and Docker-based evaluator.
  • Source location: Appendix F, “Tool Surfaces by Benchmark,” coding benchmarks
  • Source extract (verbatim): Results. Table 1 shows that agents using our knowledge curation protocol achieve the highest solve rates among the listed Haiku-based methods on ARC-AGI-1, ARC-AGI-2, Polyglot, and SWE-bench Pro, while also using the lowest reported costs. On Terminal-Bench 2, one of the hardest agentic benchmarks, our simple agent framework equipped with knowledge-centric improvement achieves competitive performance against multiple strong agentic coding systems.

  • Source location: §4.1, “Agent-Centric Baselines,” results

  • Source extract (verbatim): The curated knowledge base is itself an output of the protocol. We finally ask whether it carries standalone value, that is, whether the artifact, separated from the procedure that produced it, improves performance on unseen tasks.

  • Source location: §4.4, “Held-Out Knowledge Transfer,” opening
  • Source extract (verbatim): task solving, discussion, and distillation produce a structured knowledge asset containing cross-task heuristics, constraints, pitfalls, and validation strategies. We then freeze that generation-10 asset and transfer it to a disjoint evaluation split. The recipient run is zero-shot and task-execution-only: it uses no new forum discussion and no recipient-side distillation. A task-conditioned adapter converts the shared donor asset into a short memo tailored to the current task.
  • Source location: §4.4, “Held-Out Knowledge Transfer,” setup
  • Source extract (verbatim): Results. Table 4 shows that transferred knowledge improves zero-shot performance on both Polyglot and ARC-AGI-1 in every donor–recipient pairing. Across all cells the GPT-authored bundle is the stronger donor, yet cross-family transfer remains positive in both directions. Because the recipient runs no new forum or distillation, these gains come from the frozen donor bundle at inference time, indicating that the bundle carries donor-agnostic structure rather than donor-specific habits.

  • Source location: §4.4, “Held-Out Knowledge Transfer,” results

  • Source extract (verbatim): A single persistent agent must absorb many local lessons, some of which are task-specific, redundant, or mutually inconsistent; as the agent grows, useful behavior can be diluted by conflicting updates, and each new adaptation might degrade performance on the previous tasks.

  • Source location: §1, “Introduction,” agent-centric motivation (the snapshot preserves two PDF code-fence artifacts inside the sentence)

Connections Found

This is the strongest empirical source currently connected to Automating KB learning is an open problem: it demonstrates a bounded, benchmarkable slice of extraction, discussion, distillation, and reuse while leaving open-ended KB judgment unsolved. Its forum and distillation gates give Choosing what to learn requires both validity and learning-value gates a concrete implementation, and its scoped transfer claims instantiate Abstract an experience into a lesson only when you can state where the lesson stops. The external artifact changes later solve behaviour, supporting Continual learning requires governing behaviour-changing writes, not just storing content and Retained system-definition artifacts enable persistent deployment-time adaptation. It also complements the taxonomy in The What & When of Self-Evolving Agents and provides an outcome-level test for Claw learning loops must improve action capacity, not just retrieval. The causal faithfulness paper Large Language Model Agents Are Not Always Faithful Self-Evolvers remains an important counterpoint: this paper shows outcome gains from a curation protocol, but does not fully establish which individual retained claims caused each decision.

The self-improving-systems cluster can also profile the protocol directly, which makes it the ontology's strongest external instance. Under a boundary containing the disposable agents, forums, and distiller, it passes the self-improving-system membership test — the distilled bundles are behavior-determining organization consumed as guidance by later agents, and changes to them respond to solve outcomes and peer challenge — through a proposal-selection architecture whose forum and distillation stages can reject a claim before retention. On the pathway profile: retention is cumulative across generations, since each generation's attempts and distillation consume prior bundles and cited posts; addressability is the demonstrated strength — Appendix C traces a claim cited by id, challenged, split into two scoped claims, and its falsified branch retained as a rejection, exercising computationally the operations reflection buys addressability names; reflective coverage stops at the retained claims, with agents, prompts, and the curation protocol itself fixed outside the loop; actor allocation is computational at every improvement function; and warranted autonomy rests on strong benchmark oracles for the pass/fail evidence but only LLM debate for transfer-worthiness. The held-out transfer experiment exercises addressable retention's transfer operation end-to-end, turning a payoff the cluster deliberately leaves empirical into a first measured positive case.

Two structural claims classify the protocol as well. For moving the interpretation–enforcement boundary requires cross-form coverage, it is a boundary-stable comparison rather than a positive crossing: every retained improvement stays in natural-language, fixed external benchmarks supply the symbolic verification, and no responsibility transfers into symbolic enforcement owned by the loop. The protocol therefore shows that improvement need not cross forms; its untested settings — open-ended knowledge work, ambiguous goals, and slow feedback — remain limits on that result, not evidence that the boundary would move there. For Topology, isolation, and verification form a causal chain for reliable agent scaling it instantiates all three mechanisms: stateless agents supply isolation by construction, the curation ladder is the topology, and forum challenge plus that same external verification supplies the filtering — with one instructive wrinkle: mutable state does cross agent boundaries through the shared knowledge base, but only via writes that survive challenge and distillation, so the isolation requirement is met by gating shared state through verification rather than forbidding it.

Extractable Value

  1. External knowledge can be the persistent improvement substrate. The protocol keeps agents stateless across tasks while retaining typed attempts, forum posts, and distilled bundles; this is a concrete positive case for the KB's claim that deploy-time learning can live in inspectable artifacts rather than model weights or a continuously specialized agent. [quick-win]

  2. The curation loop has three distinct abstraction and selection stages. Task-level discussion preserves local evidence and disagreement, cross-task discussion tests recurrence and transfer, and distillation writes the reusable artifact. This gives automated KB learning an explicit architecture rather than a generic “summarize the trace” operation. [deep-dive]

  3. The main Haiku 4.5 results are strong across multiple task families. The paper reports 86.7% on ARC-AGI-1 at $76, 82.7% on ARC-AGI-2 at $80, 68.0% on Polyglot at $126, and 64.0% on SWE-bench Pro at $208; on Terminal-Bench 2 it reports 43.8% ± 3.4, above the listed 37.6% Meta-Harness comparator. These figures are evidence that the protocol can improve end-task outcomes, not merely memory retrieval. [just-a-reference]

  4. Knowledge curation beats matched-budget prompt optimization in the reported comparisons. On ARC-AGI-1 the paper reports 86.7% versus GEPA's 44% and OpenEvolve's 54%; on Polyglot it reports 68.0% versus 36% and 46%. The comparison isolates a useful design distinction: refining a reusable knowledge artifact can outperform refining a reusable solver prompt under the tested budgets. [just-a-reference]

  5. Frozen knowledge transfers beyond the producing run. On 20 held-out difficult tasks, the frozen generation-10 artifact improves zero-shot solve rates on both Polyglot and ARC-AGI-1 in every donor–recipient pairing, including cross-family transfer between GPT-5.4-mini and Haiku 4.5. This is the paper's clearest evidence for portability, while the reported ARC cross-family variance means the magnitude should be treated as indicative rather than precise. [experiment]

  6. Promotion requires scoped claims, not undifferentiated summaries. The paper's typed posts distinguish transferable insights, confirmed constraints, rejected hypotheses, and failure modes; distillation is instructed to retain the condition under which advice applies. This operationalizes the KB's boundary and validity-gate concerns and offers a schema to inspect in future automated curation work. [quick-win]

Limitations (our opinion)

The source is a v1 preprint and the snapshot is a user-supplied PDF-to-Markdown conversion. The conversion preserves much of the paper but contains split words, layout fragments, extracted tables, code-style blocks, and mojibake; fine-grained quotations and exact table interpretation should be checked against the original PDF or arXiv HTML.

The empirical scope is controlled but narrow: five benchmark families, 50-task improvement pools, 10 generations, and two LLM families. The held-out transfer set contains 20 tasks selected from cases that both recipient models initially failed, so it tests transfer on difficult tasks rather than ordinary population performance. The paper does not show that the protocol works for open-ended knowledge work, ambiguous goals, slow feedback, multi-party preferences, or long-horizon tasks with substantial subtask decomposition.

The “knowledge-only” isolation is useful but not complete. Agents still receive benchmark-specific prompts, tools, task-conditioned adapters, and a fixed execution scaffold; in the transfer experiment the adapter determines how much shared knowledge reaches the recipient. The forum and distillation phases also spend model calls and tokens, so the lower reported task-solving cost should not be interpreted as zero-cost curation or as a complete lifecycle cost analysis.

Baseline comparisons require care. Some baselines are single runs while the paper's entries average three seeds, and the Terminal-Bench Meta-Harness comparator combines a Claude Opus 4.6 proposer with a Haiku 4.5 base agent. The paper demonstrates that the protocol works under its evaluation design, but it does not isolate the contribution of each forum stage, the typed schema, the distiller, the task-conditioned adapter, or the removal of already-solved tasks through a full ablation matrix.

Finally, solve-rate gains establish end-to-end utility but not causal faithfulness of individual retained claims. The paper shows that frozen bundles help recipients, while Large Language Model Agents Are Not Always Faithful Self-Evolvers shows why a separate intervention is needed to establish whether condensed experience is actually used rather than merely present.

Update Automating KB learning is an open problem with a bounded positive case for knowledge-centric curation: add the task-forum → cross-task-forum → distillation protocol, the held-out transfer result, and the paper's scoped-claim schema, while preserving the note's distinction between benchmarked curation and general automated KB judgment.