h t t p s : / / a r x i v . o r g / p d f / 2 6 0 3 . 1 8 7 4 3

Ingest: Memento-Skills: Let Agents Design Agents

Type: kb/sources/types/ingest-report.md

Classification

An arXiv preprint presenting a formal framing, implemented system, controlled ablation, router evaluation, and held-out benchmark results. Author: A 17-person academic and engineering team led by Huichi Zhou, with affiliations including University College London, Jilin University, HKUST (Guangzhou), and the Yangtze River Delta AI Lab; the paper links public code, but this ingest evaluates the paper rather than independently auditing the implementation.

Summary

Memento-Skills treats a folder containing a declarative SKILL.md, prompts, and executable code as mutable external memory for a frozen LLM. Its read-write loop routes a task to a skill, executes it, judges the result, records utility, attributes a failure to one skill, and either rewrites that skill or creates a new one; a generated unit test gates mutations. A separately trained contrastive router targets execution usefulness rather than semantic similarity. Against a read-write ablation that disables skill optimization, the full system gains 13.7 percentage points on GAIA test accuracy (66.0% versus 52.3%) and 20.8 points on HLE (38.7% versus 17.9%). The paper's most useful boundary is as important as its headline: transfer is limited on heterogeneous GAIA tasks and much stronger within HLE's recurring subject domains, so accumulating skills helps when future tasks actually revisit the library's behavioral structure.

Quotes

  • Source extract (verbatim): To test whether offline retrieval gains translate into real execution improvements, we measure two end-to-end metrics: route hit rate (whether the router’s top-1 choice is an appropriate skill for the task) and judge success rate (whether the full trajectory actually solves the task). Fig. 9 (right) reveals that Memento-Qwen lifts route hit rate from 0.29 (BM25) and 0.53 (Qwen3) to 0.58, and judge success rate from 0.50 and 0.79 to 0.80.

  • Source location: p. 12, "Router Evaluation," Figure 9 results paragraph

  • Source extract (verbatim): To isolate the contribution of the self-evolving mechanism, we compare Memento-Skills (the full system) against a Read-Write ablation that retains the same read–write reflective learning loop—skill retrieval, LLM execution, and feedback collection— but disables all skill-level optimisation: no failure attribution, no skill rewriting, and no skill discovery. All the experiments in this paper use the Gemini-3.1-Flash as the underlying LLM.

  • Source location: p. 15, "Experimental Settings," Baselines paragraph
  • Source extract (verbatim): On the unseen test set, the full Memento-Skills system achieves 66.0% overall accuracy, compared with 52.3% for the Read-Write ablation, confirming that the skill optimisation pipeline contributes a 13.7 percentage-point gain beyond what retrieval and execution alone can provide.
  • Source location: p. 15, "Results of GAIA"
  • Source extract (verbatim): On the test set, Memento-Skills achieves 38.7% overall, more than doubling the Read-Write baseline (17.9%).

  • Source location: p. 17, "Results of HLE"

  • Source extract (verbatim): The system is built on a memory-based reinforcement learning framework with stateful prompts, where reusable skills (stored as structured markdown files) serve as persistent, evolving memory. These skills encode both behaviour and context, enabling the agent to carry forward knowledge across interactions.

  • Source location: p. 1, abstract
  • Source extract (verbatim): In the read phase, a behaviour-trainable skill router selects the most relevant skill conditioned on the current stateful prompt; in the write phase, the agent updates and expands its skill library based on new experience. This closed-loop design enables continual learning without updating LLM parameters, as all adaptation is realised through the evolution of externalised skills and prompts.
  • Source location: p. 1, abstract
  • Source extract (verbatim): Crucially, the memory is not limited to episodic traces but consists of reusable skills, each containing a declarative specification (SKILL.md) together with helper scripts and prompts. Because the write operation rewrites the prompt or program that will be executed next, each write step directly improves the policy embodied in the skill.
  • Source location: p. 9, "The Skill-Level Read–Write Loop"
  • Source extract (verbatim): Behaviour-aligned skill router. We train a contrastive retrieval model via single-step offline RL, casting skill routing as a KL-regularised Boltzmann policy that optimises for execution success rather than semantic similarity.

  • Source location: p. 8, Contributions, item 2

  • Source extract (verbatim): To prevent regression, all mutations are guarded by an automatic unit-test gate, a synthetic test case is generated, executed through the updated skill, and scored by the judge [21].

  • Source location: p. 10, section 2.1 "The Skill-Level Read–Write Loop", failure-attribution and skill-rewriting paragraph
  • Source extract (verbatim): 17: if Ut (c† ) < δ and n(c† ) ≥ nmin : 18: c′ ← DiscoverSkill(c† , xt , tracet ); St+1 ← St ∪ {c′ } 19: else: {optimise existing skill in-place} 20: St+1 ← OptimiseSkill(c† , xt , tracet , St ) 21: if UnitTestGate: validate St+1 (c† ); rollback on failure
  • Source location: p. 10, section 2.1, Algorithm "Read–Write Reflective Learning", steps 17–21

Connections Found

This paper is a strong empirical anchor for the readable-artifact loop: the retained policy is a mutable natural-language-plus-symbolic artifact around a frozen model, rather than merely a log or prompt. It also bears directly on retrieval misses breaking local reflective paths, because behavior-trained routing improves route hits and final execution over lexical and generic embedding retrieval, and on evaluating memory by effects, because the paper distinguishes retrieval quality from end-to-end task success. Its closest comparisons are SkillOpt, which validation-selects edits to one natural-language skill; SkillRL, which co-trains model weights with a natural-language skill bank; Trajectory-Informed Memory Generation, which retrieves distilled tips rather than executable folders; and Voyager, which promotes successful code but lacks Memento-Skills' failure-driven rewriting and behavior-trained router. The self-evolver faithfulness study supplies the key counterweight: selecting and injecting condensed experience does not by itself prove that the agent causally uses it.

Extractable Value

  1. Executable multi-file skills are a concrete mixed-form memory unit -- The learned object bundles declarative scope, prompts, and code, so one retained unit can carry instruction, context, and executable behavior while remaining inspectable and rewritable. This extends the KB beyond natural-language-only skill optimization and code-only promotion cases. [quick-win]
  2. Skill routing should optimize behavioral utility, not semantic resemblance -- On 140 synthetic queries, behavior-trained retrieval raises Recall@1 from 0.54 for the generic embedding baseline to 0.60; on real trajectories it raises route-hit rate from 0.53 to 0.58 and judge success from 0.79 to 0.80. The modest end-to-end delta over dense retrieval is the more honest result, but the paired measurements operationalize two different breaks in the retrieval wire. [experiment]
  3. Domain alignment bounds the return from skill accumulation -- Learned GAIA skills often did not fire on held-out questions because the task structures were too heterogeneous, whereas HLE's recurring subject domains supported reuse and much larger gains. Library growth is therefore not itself coverage; the future task distribution must revisit behaviorally similar regions. [quick-win]
  4. Failure attribution makes the mutation unit selectively addressable -- The system does not rewrite the whole agent after every miss. It selects one responsible skill, patches that folder while trying to preserve generality, and escalates to restructuring or new-skill discovery only after sufficient low utility. This is a reusable proposal-selection shape for mixed-form artifact loops. [experiment]
  5. Routing quality and causal uptake require different tests -- Route-hit and trajectory-success measurements establish that selection quality matters, but they do not show whether a selected skill's content caused a particular action. Combining this paper's router ablations with memory-perturbation faithfulness tests would close a missing evaluation layer. [deep-dive]
  6. The fast loop can include a small parametric component without updating the LLM -- The agent's large language model stays frozen, but the retrieval policy is trained as a contrastive embedding model. This is a useful boundary case for the KB's representational-form account: readable skill mutation and parametric routing can coevolve while remaining distinct adaptation surfaces. [just-a-reference]

Limitations (our opinion)

The evidence is narrower than the paper's "agent-designing agent" framing. Both benchmarks provide clear answers and repeated evaluation, all reported agent experiments use Gemini-3.1-Flash, and the main baseline retains the same read-write loop while disabling skill optimization; the paper does not compare against the strongest neighboring self-evolving-skill methods or isolate every component of failure attribution, rewriting, discovery, and unit-test rollback. HLE's within-subject reuse may reflect benchmark taxonomy and repeated solution motifs rather than general cross-task abstraction, while GAIA already shows how quickly transfer weakens when those motifs do not recur.

The claim of continual learning "without parameter updates" applies to the underlying LLM, not to the whole system: the behavior-aligned router is itself trained parametrically on synthetic query-skill pairs drawn from a curated public skill catalog. The formal convergence result is inherited from Memento 2 for KL-regularized retrieval policy iteration under bounded rewards and discounting; it does not by itself prove convergence, safety, or non-regression of the paper's practical file-rewriting and library-expansion operations. A generated unit test for the observed failure has a small validation radius but can overfit the triggering example, and the paper does not evaluate cross-skill regressions, adversarial skill mutation, provenance, contradiction handling, retirement, or million-skill routing. Finally, aggregate accuracy and route-hit metrics show useful effects but not causal faithfulness to individual retrieved skills, the distinction emphasized by Evaluate Memory By Effects, Not By Existence.

Update A retrieval miss is a local reflective-path failure with Memento-Skills as evidenced-by: add the router's generic-embedding versus behavior-trained route-hit and end-to-end results, then state the boundary that route selection tests the discovery wire while a separate perturbation test is still needed to establish causal uptake after retrieval.

Relevant Notes: