jaz-evals boundary

Type: agentic-system-analyses/types/agentic-system-boundary.md

Boundary and evidence

The target is the jaz-evals reproduction package, not the separately distributed JAZ runtime. Its intended use is to configure and run agent-method/benchmark pairings, grade attempts, retain traces and metrics, and rebuild comparative results. It is in scope as a model-dependent workflow: shipped harnesses invoke agents, and the package includes instructions and code for carrying improved context between tasks. The README calls JAZ “the agent framework under test,” rather than identifying this package as that framework (SRC-1, README.md).

The frozen boundary is commit 2ffe7b5e6792a76fccb26f9ce188c88fb95ffa72, with an applicability cutoff of 2026-10-15. The evidence tier is code-grounded for the shipped orchestration and adapter code. Source text was inspected read-only; no target, tests, examples, containers or provider calls were executed. Documentation about published runs is attributed reporting, not an observed run or a causal experiment in this evidence set.

The boundary kind is complete artifact, partial loop. All parent-repository paths at this commit remain eligible evidence, including shipped callers, prompts, evaluators, patches and reporting consumers. However, the benchmark submodules are uninitialized, and the installed JAZ, Letta and smolagents implementations are not part of the frozen source. These are material parts of execution, state maintenance and grading. This boundary therefore does not claim whole-system coverage merely because the parent repository is frozen. Submodule gitlinks pin other repositories but do not supply their contents as registered evidence.

Shipped entry points and responsibilities

The installed jaz-evals entry point calls cli.main, which loads a configuration and calls run_evaluation. The runner builds the environment and selected harness, supplies an AgentEnv, grades after the harness returns, writes attempt records, and aggregates results. These are responsibilities of this target, not an excluded enclosing application (SRC-1, pyproject.toml, src/jaz_evals/cli.py, src/jaz_evals/eval_harness.py, src/jaz_evals/registry.py). The inspected implementation includes:

report = harness.run_task(AgentEnv(env)) --- src/jaz_evals/eval_harness.py

The registered methods are JAZ, JAZ-per-task, ACE, Letta and smolagents. Their shipped adapters respectively call JAZ, drive fresh task sessions, evolve an ACE playbook, create and message a Letta agent, or build and run a smolagents chain. These are not passive dataset utilities (SRC-1, src/jaz_evals/harnesses/__init__.py, src/jaz_evals/harnesses/jaz_harness.py, src/jaz_evals/harnesses/jaz_per_task.py, src/jaz_evals/harnesses/ace.py, src/jaz_evals/harnesses/letta_harness.py, src/jaz_evals/harnesses/smolagents_harness.py). For example, the JAZ adapter issues:

jaz.invoke(self._root_overrides(), self._build_root_hooks(), root_guard, *inputs) --- src/jaz_evals/harnesses/jaz_harness.py

The AppWorld JAZ configuration selects a shipped improvement prompt. Its declared loop is relevant evidence about intended context production and later consumption, but not proof that a model followed it (SRC-1, configs/appworld_jaz.yaml, prompts/ttsi/jaz.md):

Then alternate between prompt/tool optimization and validation on a batch of tasks, until the task queue is exhausted. --- prompts/ttsi/jaz.md

The ACE implementation independently supplies an explicit task queue and adaptation path, including loading initial playbook state, passing it to sessions, adapting after task completion and writing snapshots (SRC-1, src/jaz_evals/harnesses/ace.py). No general-purpose durable knowledge system is inferred from this evaluation-local state.

Post-run consumers are also inside the artifact boundary. jaz-evals-analyze discovers retained attempt artifacts and computes diagnostics; the two log-viewer entry points render method-specific traces. The shipped official-AppWorld shell driver launches external AppWorld runs and is included as this package's driver, even though it bypasses run_evaluation. Its downstream runtime and grader remain unavailable dependencies, not responsibilities attributed to jaz-evals (SRC-1, pyproject.toml, src/jaz_evals/analyze_cli.py, scripts/run_appworld_official_react.sh).

Coverage and limits

This inventory distinguishes initial inspection from the eligible source boundary. A file not yet inspected is not an access gap.

External execution dependencies include JAZ (jaz-lang==0.2.0a4 in the local group), smolagents, a Letta server/client with optional Turbopuffer-backed search, OpenAI model services, benchmark forks and task data, and ACE's sentence-transformer encoder. Their declarations and adapter seams are evidence; their internals and live behavior are not. This prevents conclusions about actual context compaction/retrieval semantics inside those runtimes, provider stability, full benchmark validity or reproducibility of historical numbers. The README also distinguishes the installable JAZ release from the development checkouts used for historical runs (SRC-1, pyproject.toml, README.md). No linked paper, package, external trace archive or dependency repository is added to the frozen source set.

Source register