EAL-Bench boundary

Type: agentic-system-analyses/types/agentic-system-boundary.md

Boundary and evidence

EAL-Bench is an in-scope workflow for evaluating whether model-written persistent memory changes authorization and whether an executor acts on those changes. Its intended use is research evaluation, not deployment as an organization's authorization service. The boundary is the whole shipped system, including experiment drivers, memory admission and persistence, subsequent consumption, domain semantics, mitigation paths, reusable evaluation interfaces and result analysis. It is not restricted to the public scoring API. SRC-1, README.md, domains/README.md, pyproject.toml and docs/reusable_api.md establish this use and packaging.

A memory writer compresses an organizational history into persistent memory; an executor later receives that memory and a new request, then chooses whether to act through a native tool. --- README.md

The classification depends on model calls issued by the shipped workflow, not merely on an external consumer using a library. The main driver dispatches domain study plans; the shared engine calls the writer, freezes a pre-execution checkpoint and calls executors over prepared evidence. It then persists memories, attempts, states, evidence, trials and model contexts. The executor batch path submits model-visible tools and scores returned calls. These responsibilities are internal to EAL-Bench (SRC-1, experiments/run.py, experiments/authorization_memory/study_engine.py, experiments/authorization_memory/pipeline.py).

"""Run and persist a domain-built study through shared infrastructure.""" --- experiments/authorization_memory/study_engine.py

Memory production and admission are included. The LangMem wrapper assembles instructions from domain policy, maintains a single profile, parses candidates, checks visible provenance identifiers and capacity, and distinguishes accepted, unchanged and invalid candidates. It freezes the accepted artifact into executor evidence. This is a bounded benchmark memory protocol, not evidence of a general-purpose retrieval service (SRC-1, experiments/authorization_memory/langmem_writer.py).

"Maintain exactly one persistent profile containing current authorization state.", "Update the existing profile from the supplied information; do not create another profile.", --- experiments/authorization_memory/langmem_writer.py

The included consumption paths extend beyond one immediate replay. Checkpoint resume reconstructs saved execution state and schedules missing executor calls. Pressure and mitigation studies reuse saved writer runs. The shipped closed-loop driver starts from frozen base memories, executes requests, constructs action or neutral write-back blocks, seeds subsequent writer updates with the current artifact, and uses the resulting memory for later requests. Thus write-back is not assigned to an excluded host (SRC-1, experiments/authorization_memory/resume.py, experiments/mitigations/README.md, experiments/closed_loop.py).

1. The paper's incremental chains, then the open loop on their frozen memories.

--- experiments/closed_loop.py

The reusable interface supplies trial construction, preservation assessment, propagation comparisons, end-to-end attribution, export and scoring. Its scorer delegates to the internal runner's scorer. The Inspect adapter is a shipped integration, while Inspect itself and arbitrary downstream applications remain external. An external caller's model routing or storage is not attributed to EAL-Bench. The repository's own experiment runner remains included even where the public API lets a caller supply responses (SRC-1, src/eal_bench/eval/__main__.py, src/eal_bench/eval/scoring.py, pyproject.toml, docs/reusable_api.md).

The adapter supplies the dataset, the tools and the scorer. The model and its configuration stay yours. --- docs/reusable_api.md

Frozen evidence and applicability

The evidence is SRC-1 at full commit 51648690bc52d7a9c7ac080a2c67a784fe9d56cb, with applicability cutoff 2026-10-15. The checkout HEAD matches the acquired revision and its working tree was clean at inspection. The overall tier is code-grounded: entry-point dispatch, writer admission, persistence, executor consumption, resume, closed-loop write-back, mitigation cores and public scoring delegation were inspected as source text. No target code, tests, examples, provider calls or experiments were executed. Code inspection supports implemented paths, not successful live operation or benchmark validity.

External dependencies include LangMem and its LangChain integrations for the writer, OpenAI-compatible provider clients, configured model services, Pydantic, JSON patching and token counting; Inspect is optional. Their implementations, provider state, credentials and actual model behavior are not frozen here. The supplied repository includes dependency declarations and route-client code, not those external systems' internals. This prevents conclusions about external transport guarantees, model reliability or deployed authorization enforcement (SRC-1, pyproject.toml, src/eal_bench/llm/client.py, docs/reusable_api.md). The linked arXiv paper and linked external websites are not additional sources.

Historical results require a separate limit. The primary result guide says original trial, memory and provider-context JSONL files are not distributed in the normal clone, and the paper exporter can use saved counts while reporting absent originals. Some diagnostic JSONL files are tracked under results/diagnosis/; their presence does not supply the primary run archives. No raw archive was acquired. This prevents independent reconstruction and rescoring of the primary published runs or their bootstrap inputs from this boundary alone. Repository reports remain attributed reported operation unless later inspection establishes an actual trace or intervention with adequate conditions (SRC-1, results/README.md, analysis/paper_results.py).

Missing raw files prevent independent scoring and reconstruction of bootstrap inputs. New model runs are stochastic replications and cannot replace original records. --- results/README.md

Top-level coverage

Every tracked top-level entry is classified below. “Included” declares functional scope, not that every file has already been read. Initial inspection is stated explicitly; an available but unread file is not an access gap. Later jobs may inspect additional paths at this same registered commit. No shipped state-producing or state-consuming path is deliberately excluded.

Source register