jaz-evals synthesis
Type: agentic-system-analyses/types/agentic-system-synthesis.md
Bounded synthesis
Configure an experiment, not an enclosing runtime
jaz-evals is an evaluation workflow around separately distributed agent runtimes. It configures method/benchmark pairings, lets agents act, grades completed work, retains traces and metrics, and supplies comparative reporting. Some methods also carry revised context into later tasks. This account covers the parent repository at commit 2ffe7b5e6792a76fccb26f9ce188c88fb95ffa72, with a 2026-10-15 cutoff. The boundary is complete artifact, partial loop: benchmark submodules and installed runtime implementations are unavailable. Evidence is inspected implementation, shipped instructions and attributed historical reporting, not executed target code, supplied run traces or causal experiments. See the boundary and evidence and Source register.
An operator selects a YAML experiment. The central Python runner creates attempt identities, constructs environment and method adapters, supplies an agent-facing tool view, then grades and writes results. The five registered methods differ in who owns the next step: JAZ assigns queue policy to a root model and execution to external JAZ; per-task JAZ and ACE sequence tasks in Python; Letta delegates agent steps to a remote server; smolagents supplies a managed-agent chain. The official AppWorld shell driver bypasses the central runner entirely. These are distinct control paths, not interchangeable instances of one permission, budget or recovery envelope. See RT-RTE-evaluation-attempt, RT-RTE-jaz-queue, RT-RTE-fresh-session, RT-RTE-ace-adaptation, RT-RTE-letta-bridge, RT-RTE-smolagents-chain and RT-RTE-official-baseline.
Delegate work, submit it, then interpret feedback
The representative AppWorld JAZ workflow connects a stronger root model, smaller solver delegates and a shipped prompt for test-time self-improvement. The root opens a task, manually forwards single-task instructions and current prompt/helpers, and requests an answer plus history. Shared API tools propagate to children; queue tools stay with the root. A child context is not a transaction: its API effects act on the live benchmark world, while diagnostic history must explicitly cross back. The prompt requests one solver call per task, diagnosis over initial batches, minimal revisions, later validation and discarding ineffective changes. That discipline is model-interpreted policy, not a separately enforced acceptance gate. See RT-RTE-jaz-queue, RT-CLM-ttsi-experiment and the workflow account.
The root's completion call, rather than solver return, submits and grades the task. AppWorld exposes assertion outcomes and potentially expected-versus-actual details. Its code retries a non-scalar submission once as None only for a particular text-matched shape error; grading proceeds despite submission failure. A grader exception becomes a failed result, and task advancement occurs regardless of success. This preserves continuation, not correctness or rollback of earlier API effects. StuLife has a materially different contract: it retains correctness internally but returns progress only, leaving ACE without evaluation feedback through that channel. A benchmark verdict does not validate a transferable explanation of why a solver succeeded. See WF-RTE-task-submission, EPI-RTE-appworld-check, EPI-RTE-appworld-disposition and EPI-RTE-stulife-feedback.
Carry changed context without treating it as verified knowledge
ACE provides the clearest implemented cross-task update. A fresh solver receives the current playbook; its recorded trajectory and returned feedback feed a reflector and curator; structurally admitted ADD operations change advice; optional similarity refinement rewrites it; subsequent solver and adaptation prompts receive all current advice after count removal. The live string carries the update. Per-task and final snapshots support explicit later seeding, not automatic mid-run recovery. Admission is computational, without per-bullet human approval. Initial seed authorship and approval remain unresolved. See RT-RTE-ace-adaptation and MEM-OBJ-playbook.
This is a wired trace-fed durable advice route, not merely log retention. Its operational authority is nevertheless limited: parsers enforce reply shape and allowed operations, while solver-facing advice is explicitly advisory. Structural admission neither tests advice truth nor accepts generality against an evidential criterion. No generated instance establishes whether particular output is a new truth-apt claim, reshaped content or an imperative policy. Retention and later prompt supply therefore do not establish epistemic acceptance or integration after such acceptance. See EPI-RTE-curator-check, EPI-RTE-curator-admission and RT-BAP-playbook-guidance.
The evidence available to adaptation is also selected: the trajectory input keeps its first 400,000 characters and drops the tail; task excerpts are shorter. Diagnostic sidecars preserve call bodies for inspection, but later solvers do not reload them as per-bullet reasons. Helpful/harmful counters are not measured reliability and are stripped from ordinary model-facing copies. Optional dedup groups advice by embedding similarity, not correctness. Its public merge-free branch keeps the leader; the harness instead supplies model merging. Groups can overlap, so a failed merge does not guarantee all that group's members survive another successful merge in the same pass. Neither parsing nor similarity establishes semantic preservation. See MEM-OBJ-adaptation-artifacts, MEM-OBJ-playbook-counts, MEM-RTE-playbook-dedup and the report verification limit.
Continue in a new context, then deliver inspectable artifacts
Long-horizon JAZ and smolagents guidance affords explicit history handoff and requested substring search into retained experience. Progress summaries are separate from raw history. In smolagents, the requested wrapper displays only a suffix while retaining the full list if constructed as instructed; the warning callback does not force delegation or verify faithful recording. Fresh per-task frames and the optional CodeAct binding filter instead control current visibility; they do not themselves create learned guidance or prove all experience inaccessible. See MEM-RTE-jaz-history, MEM-RTE-smolagents-history, RT-RTE-fresh-session and RT-RTE-codeact-filter.
Letta's adapter exposes block initialization, task-message selection and a summary-production seam, but not complete archive, search, compaction-installation or editing semantics. Exported host diagnostics are not restorable server memory; teardown destroys ephemeral server state. See MEM-OBJ-memory-blocks, MEM-OBJ-conversation, MEM-OBJ-compaction-summary and MEM-OBJ-archival-memory.
Final delivery distinguishes task submission, attempt termination and run-level artifact completion. Ordinary harness failures can leave partial graded work, but construction, cleanup, final grading and process failures can prevent records. AppWorld's inherited true completion predicate means JAZ's normal-return validator does not enforce queue exhaustion; scoring still uses the entire selected queue. Retained traces/results feed an explicit operator-requested analysis invocation, not an automatic solver update. See RT-RTE-evaluation-attempt, RT-RTE-jaz-queue and MEM-RTE-artifact-analysis.
What the improvement pathways establish
The strongest supported contribution is an implemented pathway from ACE solver experience to retained advice delivered to later fresh sessions, alongside JAZ's afforded model-directed prompt/helper revision. Within these bounded curricula, context can change without weight updates. Whether criticism of consumed theories improved future capacity remains uninspected for both: changed inputs and persistence do not establish semantic dependence or benefit. Their independent assessments are as follows, supported by RT-RTE-ace-adaptation and RT-RTE-jaz-queue:
| Property | ACE workflow | JAZ root-plus-solvers workflow |
|---|---|---|
| Theory-builder condition 1: localized content | afforded through whole advice bullets | afforded through whole prompt/helper units |
| Condition 2: decisions depend on content | uninspected; delivery is wired | uninspected; delivery is not dependence |
| Condition 3: content-directed criticism | claimed by diagnosis guidance; no generated case | claimed by hypothesis/test/discard guidance; no generated case |
| Condition 4: result shapes another round | wired advice delivery into later rounds; semantic use unobserved | afforded later-batch revision reuse |
| Reflection | afforded trajectory self-representation and guidance-mediated action; actual fidelity and both causal directions unobserved | afforded history-based self-representation and editable guidance; causal exercise unobserved |
| Reflective theory builder | uninspected | uninspected |
| Autonomous theory builder | uninspected as a full property despite computational proposal/admission | uninspected as a full property despite computationally assigned roles |
| Self-improvement | wired dispositional, trace-responsive change to its own later solver guidance | afforded dispositional, improvement-directed prompt/helper change |
The self-improvement boundary is each coordinating workflow plus its solvers, the horizon is later tasks within an attempt, and the independent objective is benchmark task success. Neither route establishes exercised self-improvement: no particular evidence-shaped update and subsequent causally dependent operation are supplied, especially the fourth causal link from changed organization to later action. Successful improvement is separately unestablished. These limits do not negate the wired update mechanism or its narrower trace-learning classification.
For a bounded study of context adaptation, the package exposes consequential seams: different feedback channels, explicit carried advice and fresh solver frames. For a claim of reproducible improvement or warranted transferable explanations, the present evidence is insufficient. Linked before/after advice, source trajectories, actual admission/rejection decisions and later behavior would test activation and exercised change. Controlled comparisons varying the proposed revision independently of task order, shared state and other changes would test benefit at the intervention's actual granularity. Frozen runtime/grader implementations and matching historical artifacts would change the assessment of end-to-end execution and reproducibility. No product ranking or system-wide epistemic verdict follows from the current boundary.
Limitations
| limitation | affected SRC IDs and record citations | inspected boundary | conclusion prevented | evidence that would resolve it |
|---|---|---|---|---|
| External execution and grading loop is partial | SRC-1; RT-RTE-evaluation-attempt, EPI-RTE-appworld-check, RT-RTE-official-baseline | Frozen parent adapters and native-driver launch code; benchmark submodules/runtime internals unavailable | Whole-loop guarantees, native grader validity and historical reproduction | Registered frozen dependency implementations, task data and matching execution artifacts |
| No supplied observed runs or causal experiments | SRC-1; RT-RTE-jaz-queue, RT-RTE-ace-adaptation, MEM-RTE-smolagents-history | Source and instructions only | Actual compliance, semantic dependence, improved future capacity and exercised or successful self-improvement | Provenance-linked updates, critic decisions and later actions; controlled comparisons for benefit and attribution |
| Dedup failure recovery is local, not pass-wide losslessness | SRC-1; MEM-RTE-playbook-dedup, EPI-RTE-dedup-replacement, report verification | Similarity grouping and pass-wide absorbed-set reconstruction | Guarantee that members of an unparseable group survive the entire pass; semantic preservation remains separately unverified | Disjoint grouping or explicit pass-wide preservation checks; source/output comparisons for meaning preservation |
| Initial content control is unresolved | SRC-1; MEM-OBJ-playbook, MEM-OBJ-memory-blocks | Seed-file and configured-block supply seams | Original authorship or per-content human approval; complete write-agency account | Provenance and explicit admission decisions for supplied content |
| Letta retained state and later consumers remain opaque | SRC-1; MEM-OBJ-conversation, MEM-OBJ-memory-blocks, MEM-OBJ-compaction-summary, MEM-OBJ-archival-memory | Adapter inputs, summary return and diagnostic export | Complete storage, authority, retrieval, curation, checkpoint recovery or trace-learning classification | Frozen server selection, persistence, edit-control and later-injection implementations with linked artifacts |
| Publication and diagnostic consumer coverage is incomplete | SRC-1; MEM-RTE-artifact-analysis, RT-RTE-official-baseline | Analysis dispatch and selected artifact-consumer seams; viewers/plot/release transformations not fully audited | Exhaustive transformation inventory, system-complete negatives and reconstructed historical numerical comparisons | Full material consumer inspection and archived inputs sufficient to reproduce outputs |
| Control and model fixity are adapter-specific | SRC-1; RT-CLM-tool-boundary, RT-RTE-letta-bridge, RT-RTE-ace-adaptation, RT-CMP-lite-llm, RT-CMP-sentence-transformer | Tool wrappers, bridge, direct adaptation calls and model construction seams | Hostile-code isolation, universal budget enforcement or immutable weight identity | Inspected runtime/deployment contracts and authenticated boundaries; model weight revisions and coverage of direct-call budgets |
The profile verification declares no additional limits (profile verification). No unresolved cross-report conflict was identified in the supplied reconciliation. This synthesis adds no source inspection or evidence.