h t t p s : / / a r x i v . o r g / a b s / 2 6 0 9 . 0 3 9 1 5

Ingest: RuleMem: Active Rule Memory for Long-Term Conversational Agents

Type: kb/sources/types/ingest-report.md

Classification

Scientific paper by Xingyuan Zeng and colleagues, retained as arXiv v1 dated 3 September 2026. The authors describe their own system and report benchmark comparisons, ablations, sensitivity analysis, and illustrative cases. These are author-reported experiments; this ingest does not independently reproduce them.

Summary

RuleMem extracts temporal facts from conversations, mines paths through them, and induces reusable natural-language rules with typed placeholders. It filters candidate rules using Rule Perplexity Consistency (RPC): a score combining how much a rule body increases the model likelihood of its conclusion and how much retrieved evidence further increases it. At query time, similarity to rule conclusions selects rules whose premises guide fact retrieval; an LLM filters entity bindings and generates answers from the rules and evidence. Within this fixed fact/rule representation and prompt-based inference pipeline, the paper reports LoCoMo average accuracy of 78.05, versus 65.90 without RPC and 43.43 without rule abstraction and explicit reasoning. Its useful contribution is a concrete way to use retained abstractions for both finding evidence and interpreting it. The results support those tested interventions, without establishing formal deduction, reliable causal rules, or superiority over alternative memory decompositions. A reported failure shows a rule overriding explicit contrary evidence.

Quotes

No source quotes have been retained yet.

Connections Found

RuleMem supplies bounded empirical support for access burden and transformation burden as distinct query dimensions. Its diagnostics separate supporting-fact recall from wrong answers after supporting facts have been retrieved, and its interventions separately add rule-guided recall and rules in the answer prompt. Within the tested fact/rule pipeline, this supports diagnosing retrieval and inference separately; it does not measure their full costs or show that the chosen representation is optimal.

It also provides a useful comparison for trace-extracted memory earning authority per operation. Rule extraction, admission, and later application are distinct operations. RPC improves aggregate performance against admitting every induced rule, but the interview failure demonstrates that admission does not establish applicability to each future case. A likelihood-based admission signal and a warranted generalization remain different achievements.

Learning Claims (our opinion)

The learning mechanism changes external memory: an LLM groups similar reasoning paths, replaces concrete entities with typed placeholders, and retains rules that pass RPC. Path reconstruction has access to extracted facts and corresponding raw dialogue snippets. At use time, query-to-head similarity activates a rule, body-to-fact similarity retrieves evidence, and LLM filtering supplies semantic variable binding. No model-weight update is described. The resulting rules can change both what evidence enters the answer context and how the model interprets it.

In Commonplace terms, this is induction and subsequent use of an explicit, criticizable rule set. The rule bodies and conclusions offer locations that could be revised. However, the described mechanism does not establish theory refinement: it does not show downstream errors being localized to existing rules, followed by selective repair and checks that useful prior behavior survives. RPC is a candidate-admission operation, not demonstrated revision from counterexamples. The paper's closed-loop description should therefore not be read as evidence of recurrent error-driven theory repair.

Its effective update space is bounded by temporal fact extraction, graph-path mining, conjunctive natural-language bodies and single conclusions, similarity-based activation, and prompt-based inference. The model has expressive latitude inside those stages, including access to raw dialogue during path reconstruction, so missing information cannot simply be inferred from the quadruple representation. But the experiments do not compare learning those boundaries against keeping them fixed. As learning inside a fixed decomposition inherits its mistakes explains, gains inside an update space do not vindicate choices outside it. Removing RPC varies admission filtering; removing rules and explicit reasoning changes several mechanisms together. Neither contrast isolates the Horn-clause template as the cause of improvement.

The contribution to our account is thus a concrete consumption path for induced prose rules and evidence that the tested admission filter helps that path. It leaves open how such rules should be corrected when explicit facts contradict them. Its Horn-clause notation does not fix mechanically computed consequences: variable binding and inference remain model interpretations.

Extractable Value

  • Separate evidence-access and answer-generation diagnostics [quick-win]. Add the reported comparison to the existing access/transformation note: the paper reports average supporting-fact recall increasing from 0.56 to 0.79 with Guided Recall, and average post-retrieval reasoning failures falling from 120.4 to 105.9 with Explicit Reasoning. These are author-reported diagnostics within the supplied memory and prompting setup, not cost estimates or a comparison of alternative decompositions. The diagnostic distinction transfers more readily than the exact gains.
  • Use rule premises as evidence-search cues [experiment]. The head-to-query and body-to-fact sequence offers a testable retrieval design when the query and necessary evidence are semantically distant. A KB experiment could compare it with direct retrieval while holding fact extraction, retrieval budget, and answer generation fixed. The source supports this as a candidate mechanism, not a general advantage for all rules or domains.
  • Keep admission confidence separate from applicability [quick-win]. The unfiltered-rule ablation and the interview counterexample jointly illustrate why a useful admission score cannot license unconditional consultation. RPC tests changes in model likelihood; it does not test entailment or determine which evidence should override an admitted rule.

Limitations (our opinion)

The results compare compound systems and selected removals inside RuleMem. Baselines generally use official implementations with default parameters, while RuleMem's admission threshold and signal mixture are empirically chosen. The retained paper does not establish matched compute, matched retrieval budgets, or a clean alternative-representation comparison. Consequently, the headline improvement over the average of 14 baselines is not the margin over the strongest baseline, nor evidence that abstract rules are necessary. Table 1 gives average accuracy of 78.05 for RuleMem and 72.66 for Zep, a 5.39-point difference; the abstract's 27.47-point figure uses the baseline average.

RPC measures conditional likelihood, which can reward familiar or fact-associated conclusions without proving a rule's causal validity or scope. The framework figure infers residence from employment and employer location, a relationship that need not hold. The interview failure is stronger evidence of the practical risk: the model answers that an interview is required despite an explicit statement that it is unnecessary. Aggregate gains do not remove this failure mode.

The successful case is internally inconsistent: the question asks about Caroline, while the recalled facts and final answer concern Melanie. It therefore cannot serve as clean evidence that the semantic-binding filter worked. Its abbreviated dialogue also does not independently establish every detail in the retrieved injury facts. The case should not be silently repaired into a successful demonstration.

The capture is marked full-source, but the retained document ends after references while referring to absent appendices for prompts, extended configurations, LongMemEval_s* results, and cross-model results. Those claims cannot be checked at their advertised detail from this observation. PDF text extraction also interleaves chart labels; the diagnostic averages above are the paper's prose reports, not independently recomputed chart statistics. No implementation was inspected or executed, and the retained results do not establish uncertainty across repeated runs, long-term rule maintenance, or transfer to Commonplace's methodology-writing tasks.

Update Access burden and transformation burden are distinct query dimensions with RuleMem's separate supporting-fact recall and post-retrieval error diagnostics, explicitly limiting the evidence to its tested fact/rule pipeline.