EAL-Bench synthesis
Type: agentic-system-analyses/types/agentic-system-synthesis.md
Bounded synthesis
An experiment workflow, not an authorization service
EAL-Bench tests whether persistent agent memory preserves changing authorization boundaries and whether erroneous memory affects downstream tool decisions. This analysis covers its whole shipped evaluation workflow at commit 51648690bc52d7a9c7ac080a2c67a784fe9d56cb, with applicability cutoff 2026-10-15. The sole source, SRC-1, supplies implementation and doctrine/design evidence. No target code or provider calls were executed, and no inspected execution trace or causal experiment establishes operation or benefit. Whole-system scope does not mean exhaustive inspection of every study or maintenance path. The missing primary historical archives also prevent independent reconstruction of published runs. See the boundary and Source register and EPI-CLM-benchmark-test.
The ordinary workflow is host-owned experimental orchestration. An operator selects domain, study, cases and model targets. Code builds a plan, orders writer updates, selects executor inputs and persists the results. Models propose memory contents and terminal responses; they do not own the overall next-step policy. The inspected benchmark runner scores native tool calls rather than placing purchases, transferring money or performing security operations. Its answer reference is the fictional domain's encoded authorization ledger, not the memory under test. Thus an unauthorized benchmark action is a measured model decision, not evidence of a real transaction. See RT-RTE-study-execution and RT-RTE-executor-score.
Formation: admitting a profile without certifying its meaning
In the ordinary LangMem path, a writer receives domain guidance, selected history updates and one existing profile. It proposes typed authorization state or free text. The wrapper enforces profile identity, payload structure, visible source identifiers and capacity when capacity enforcement is enabled. Model proposal and software admission are separate; launching the experiment does not constitute human approval of each remembered grant. A visible citation can still be interpreted as authority it does not confer. The resulting accepted status is protocol admission, not evidence-consuming acceptance of the authorization claim as true. See RT-RTE-writer-admission.
Recovery preserves this distinction. An unsuccessful first attempt can receive one structural repair opportunity when two attempts are configured. Terminal failure preserves prior accepted state; where none exists, an empty artifact can support downstream evidence formation. A completed run or available memory artifact therefore does not prove successful substantive writing. The representative Procurement workflow traces this ordinary generated-final branch from launch through repair, freezing, execution and persisted delivery without inserting unrelated mitigation stages.
Accepted payloads carry lineage in a separate metadata envelope. Attempts retain rejected arguments and diagnostics. Freezing copies the payload and explicitly omits source history; provenance identifiers do not automatically supply source text or reasons to the executor. The same typed JSON has fixed symbolic consequences at a parser or evaluator but acquires authorization significance through model interpretation at the executor. See MEM-OBJ-memory-artifact, MEM-OBJ-memory-attempt and RT-BAP-memory-evidence.
Alternative formation and selection paths have different guarantees. A reviewer selects one whole blinded candidate without editing or merging; oracle-only selection instead minimizes encoded field error, which need not be zero. The optional source-authority gate removes records unless every citation is visible and authority-capable, but does not check entailment, scope or lifecycle correctness. Memory's annotation establishes paired original/gated replay wiring, not observed mitigation benefit. Event sourcing admits structurally valid lifecycle deltas into a cloned reducer and checks reduced-state capacity before returning a candidate. Its cumulative event log and compact current state are separate products: recovery replays the former, while subsequent writer context receives compact state and the new block. See RT-RTE-candidate-selection, RT-RTE-authority-gate, RT-RTE-event-admission, MEM-OBJ-event-log and MEM-OBJ-reduced-state.
Consumption, measurement and delivery
For an ordinary executor job, code selects full history, memory or empty evidence, then builds a fresh invocation. The memory path serializes selected payload beside the current request under a separate domain system prompt. Writer reasoning and audit history are not inherited. Explicit messages and challenge preparation are alternatives, so this frame is not universal. The model's terminal response is subsequently scored against independent domain semantics. Trial validity and authorization classification are measurements after generation, not a transaction veto. Public scoring and the shipped Inspect adapter preserve this measurement boundary; a separate generic tool loop actually invokes caller-supplied Python handlers without the benchmark scorer as a guard. See RT-RTE-executor-score, RT-BAP-memory-evidence and RT-RTE-alternate-consumers.
The engine delivers a run-directory path with applicable memories, attempts, states, evidence, trials, contexts and manifest data. An explicit pre-execution checkpoint pause is intermediate delivery, not completed executor work. Supported resume validates hashes and reconstructed identities, reuses completed calls and schedules missing ones without regenerating the writer; reviewer-backed checkpoints are rejected. Saved replay instead starts new executor work from a completed source run. Event recovery separately reconstructs and checks reducer state. None establishes exactly-once provider execution or trusted filesystem isolation. See RT-OBJ-executor-plan, RT-RTE-checkpoint-resume, MEM-RTE-saved-replay and MEM-RTE-event-recovery.
For the research scenario, the discriminating contribution is separation of remembered-content error from response error. Preservation and formation assessment compare memory against encoded case semantics. Free-text scoring additionally requires an eligible typed annotation: matching content hash, accepted status, parseability and agreement control score eligibility, not interpretation fidelity. Deliberately widened memories test sensitivity but cannot establish that a writer produced the error. See EPI-RTE-memory-assessment, EPI-RTE-annotation-admission and EPI-RTE-variant-construction.
The implemented end-to-end attribution predicate is stricter than unauthorized action alone: false authority must form for the request, the written-memory arm must take the denied action, and the oracle-exact arm must not take it. Resource, surface and executor matching constrain the comparison. Caller-supplied rows do not authenticate formation timing, and omitting the optional expected population can hide dropped written arms. This supports a bounded reporting criterion, not independent causal identification of a model mechanism or transfer beyond the tested contrast. See RT-CLM-memory-attribution and EPI-RTE-failure-attribution.
Feedback: a retained update pathway, not demonstrated improvement
The strongest supported learning-related contribution is an implemented automatic route from recorded executor action/payload and benchmark outcome, through synthesized feedback and a seeded writer update, into retained memory supplied to subsequent requests. This qualifies as wired trace-fed artifact revision without requiring weight changes. Its scope is bounded: neutral controls omit action/outcome, rejected updates retain the seed, the final request has no further update, and optional cumulative action logs change the supplied context. No inspected run establishes that this delivery changed behavior or improved future capacity. See RT-RTE-closed-feedback and RT-BAP-profile-update.
The independent properties remain separate. The runtime supplies no affirmative assessment that formulated theories meet theory-builder conditions 1–4. Localized authorization state is available, but delivery alone does not establish content-dependent consumption, outcome labels do not establish formulated content-directed criticism, and retained feedback alone does not establish iteration on such criticism. These are unestablished theory-building conclusions, not repository-wide absences. Criticism of a consumed theory improving future capacity is likewise not demonstrated. Reflection is not established by the feedback route's subject matter alone. Computational proposal and admission are wired, but do not establish autonomous theory-building across every internal operation. See RT-RTE-candidate-selection, RT-RTE-closed-feedback and the epistemic bounded conclusion.
For self-improvement, the closed-loop horizon is later requests within the configured chain, and the relevant objective is authorization-preserving behavior. The retained update can enter the system's own behavior-determining input state rather than remaining an external report. However, no observed comparison closes the link from that update to subsequent behavior causally depending on it; exercised self-improvement and successful improvement remain unestablished. The code establishes a feedback capacity, not a guaranteed improvement-directed or beneficial update. Hosted writer, executor and reviewer interfaces request inference; provider-internal parameter changes and exact weight identity remain uninspected. See RT-RTE-closed-feedback, RT-CMP-writer-model, RT-CMP-executor-model and RT-CMP-reviewer-model.
These conclusions would change with inspectable matched executions linking feedback, accepted payloads and successor behavior; independent interventions would be needed for finer component attribution. Source-to-candidate comparisons and semantic admission checks would strengthen authorization-warrant claims, while validated annotation/source comparisons would strengthen prose scoring. Original run archives would enable historical rescoring. Completing the alternate writer and consumer inventory would refine context and authority comparisons, not erase the positive baseline mechanisms. No new target-source reads were performed for this synthesis.
Limitations
| Limitation | Affected SRC IDs and record citations | Inspected boundary | Conclusion prevented | Evidence that would resolve it |
|---|---|---|---|---|
| Alternate writer and context inventory is partial; report verification identifies BM25-selected earlier messages, periodic empty-profile source-history rebuilds and hybrid prose notes omitted from separate analyst inventories. Hybrid notes reach executors but are stripped for record fidelity. | SRC-1; report verification limits, MEM-OBJ-memory-artifact | Verification's reads of experiments/authorization_memory/writing_methods.py, experiments/authorization_memory/hybrid_memory.py, writer-variant wiring and LangMem rebuild handling |
Exhaustive working-context, read-back and mixed-form artifact-check comparisons; a universal compact-profile/default-context account | Accepted route/part records tracing selection, rebuilding, hybrid-note admission and each later consumer |
| Event-recovery authority is locally under-enumerated in the partial profile: replay inconsistency enforces rejection as well as supplying validation. | SRC-1; profile verification limits, MEM-RTE-event-recovery, MEM-OBJ-event-log | Inspected event restoration and saved/replayed record comparison | Complete consumer-by-consumer authority comparison for event recovery; not the separately supported existence of enforcement | Explicit classification of the retained log's rejection consequence at restoration |
| No inspected execution or causal experiment establishes activation or benefit; primary historical raw archives are missing. | SRC-1; Source register, RT-RTE-closed-feedback, EPI-RTE-failure-attribution | Source-grounded workflow and evaluator code, not original trial/memory/context archives | Reproducing primary results, observed behavioral influence, improved capacity, exercised self-improvement or causal component effects | Original linked artifacts and matched intervention traces with population, chronology and comparison conditions |
| Admission and annotation eligibility do not establish semantic fidelity or independently correct domain truth. | SRC-1; RT-RTE-writer-admission, RT-RTE-authority-gate, RT-RTE-event-admission, EPI-RTE-annotation-admission, EPI-OBJ-case-ledger | Structural/profile checks, source-capability filter, reducer rules and annotation consistency; case oracle correctness not independently validated | Certified remembered authorization, faithful prose extraction, or real organizational permission guarantees | Source-entailment/scope/lifecycle checks, validated annotation comparisons and independent oracle validation |
| Public attribution consumes supplied rows; chronology and complete population are not intrinsically authenticated. | SRC-1; EPI-RTE-failure-attribution | Matched per-request attribution implementation, including optional expected rows | Unqualified causal identification or assurance that omitted written arms cannot bias reports | Frozen pre-execution formation evidence, complete expected populations and authenticated matched outcomes |
| Remaining pressure/challenge, hybrid, domain repair/maintenance and analysis paths are not fully traced; generic caller control and initial provenance remain unresolved. | SRC-1; memory integration issues, epistemic coverage, EPI-OBJ-annotation-state | Positive baseline, gate, event, feedback, replay and Procurement annotation paths; not every shipped alternative | Universal admission/budget/human-control classification, complete trace-source inventory, or whole-system negatives about criticism and semantic checking | Producer-to-consumer records for the named alternatives and original-input provenance |
| External model/dependency internals, exact weights and deployed host grants are not frozen; recovery is not universal or exactly-once. | SRC-1; RT-CMP-writer-model, RT-CMP-executor-model, RT-CMP-reviewer-model, EPI-CMP-annotation-model, RT-RTE-alternate-consumers, RT-RTE-checkpoint-resume | Hosted inference clients, scoring/handler distinction and local checkpoint checks | Immutable model identity, provider-internal learning conclusions, deployed authorization isolation or exactly-once external effects | Version-attested provider evidence, dependency contracts, explicit host authority boundaries and interruption/effect traces |