Use Trace-Derived Extraction As Meta-Learning

Type: kb/types/note.md · Status: current · Tags: agent-memory, context-engineering, learning-theory

Trace-derived extraction is the parallel path for memory that was not captured while understanding was live, or that only becomes visible across later traces. Session logs contain latent memory-creation opportunities, but those opportunities differ by oracle strength.

Corrections are strongest because the log contains both a negative and positive signal. Silent failures are weaker: the task appears completed, but the trace shows errors, retries, fallback paths, warning output, or weakened guarantees. Preferences are distributed over many accept/reject events. Procedures show up as recurring action sequences. Discoveries and broad syntheses have the weakest immediate oracle; their value often appears only through later reuse.

Without an explicit signal-quality distinction, automated or semi-automated extraction can give weak-signal discoveries, preferences, or syntheses the same apparent authority as corrected errors. That creates trust and lifecycle failures: low-confidence memories look durable, reviewers cannot tell which candidates need stronger evidence, and activation mechanisms may spend context on lessons that were never well grounded.

Artifact Learning And Weight Learning

This requirement mainly describes readable and symbolic memory artifacts because they can be inspected, diffed, promoted, and rolled back. Systems such as AgeMem show a different path: traces train a policy for Add/Update/Delete/Retrieve/Summary/Filter actions. That path belongs where the oracle is strong enough to justify learned memory-management policy; it should not be smuggled in as ordinary artifact promotion.

Memory Evolution

Extraction needs an evolution operation, not only creation. New memory may update, split, merge, re-tag, or contextualize nearby old memory. The comparative review flags A-MEM's evolution step because new notes update neighboring notes' context and tags, while Hindsight and Cludebot show CRUD and dream-cycle variants. The requirement is not that every system automate this immediately; it is that the architecture leave room for old memory to be revised by new evidence instead of only appending candidates.

Methods

  • Narrow, schema-constrained extraction prompts for one signal type at a time.
  • Classifiers or simple rules for explicit events: user correction, command failure, retry, fallback, approval, rejection, or repeated tool sequence.
  • Batch analysis over many sessions for preferences, procedures, and recurring failure patterns.
  • Manual observation inboxes that let agents record noticed improvement opportunities without interrupting the current task.
  • Human or agent review queues for weak-oracle candidates such as discoveries, broad design principles, or high-impact policy changes.
  • Confidence, source pointers, and candidate status fields so extracted items do not masquerade as durable knowledge.
  • Evolution proposals that update tags, context summaries, links, nearby notes, or existing observations when new evidence changes how older memory should be read.

Evaluation Questions

  • Does extraction distinguish strong corrections from weak discoveries?
  • Are weak-oracle candidates prevented from gaining durable authority by default?
  • Can new evidence update nearby old memory rather than only appending new records?
  • Is weight or policy learning limited to domains with sufficiently strong feedback?

Relevant Notes: