jaz-evals exact-publication audit

Decision

Integrate with recorded limits. The inspected publication supports a bounded implementation account, not demonstrated improvement or historical reproduction. The important deduplication recovery exception is already explicit in the public synthesis. No consequential blocker was established within this audit's selected scope. Parent review, byte recheck and integration remain separate decisions.

Identity and authority

  • Exact run: AAS-2026-10-15-jaz-evals-17ab95f93794-01.
  • Auditor: independent openai/gpt-6.1-sol, medium; runtime environment reported PI_PROVIDER=openai, PI_MODEL=gpt-6.1-sol, PI_REASONING_LEVEL=medium.
  • Prepared worktree: /home/zby/llm/commonplace/.commonplace/worktrees/jaz-evals-17ab95f93794 (called W below).
  • Command prefix: W/.venv/bin, expanded to the absolute path in executed commands.
  • Method HEAD: a568c6dc489ad546374536b642c3889ce23149a8; independently verified against Git.
  • Frozen source: https://github.com/jaz-lang/jaz-evals, W/related-systems/jaz-lang--jaz-evals, HEAD 2ffe7b5e6792a76fccb26f9ce188c88fb95ffa72. Source status was clean using GIT_OPTIONAL_LOCKS=0.
  • Publication: W/kb/agentic-system-analyses/retained/jaz-evals (P below).
  • Run evidence root: W/kb/agentic-system-analyses/state/AAS-2026-10-15-jaz-evals-17ab95f93794-01 (R below).
  • Trace: /media/zby/2b5e98b6-baf0-4e97-90d2-03321ea23372/outputs/batch-13/jaz-evals.output (T below).

Read the complete commission, both audit procedures, repository instructions, work collection contract, workshop framing and exact inventory row 38. W's status showed only the untracked retained publication. No sources, publications, method or run state were changed; no delegation, acquisition, recovery or integration was performed. The only authored file is this report.

Byte identity

Independently hashed all 13 publication files and checked exact filenames against row 38's complete published-sha256 map: all matched, with no extra files. R/effects/publish.json is completed, names this exact run/source/destination, and its new map agrees with the inventory. R/state/attempts/000033-publish.json is completed.

The assembled ARTIFACT.yaml hash is f8e9ba3ff3836e26ae130a2f3ab525c14ef214a44dd6d15c1cdf54cfc654e85b, not the inventory's separate type-only working declaration hash. Key public bytes are overview fb25e920c1a2dc75645e49a9b36f1fbbfa8a033e62c5dc78921c84156dc51ef7 and synthesis 2e2e8ef1826758fd304bf9cddb23d844987f01b6dd539e9f18879ae64e5ed264. This is byte/provenance evidence, not proof of worker compliance.

Method and historical comparison

Used a bounded discrepancy-led inspection: read the public overview, synthesis and boundary fully; compared the complete historical lead; searched relevant runtime, memory, reconciliation and verification passages; then checked selected consequential mechanisms directly in the frozen source and selected actual worker messages.

Historical lead: kb/agentic-systems/reviews-archive/jaz-evals.md, run AAS-2026-09-26-jaz-evals-01, declared source revision 83dc51ebbd02c9299890b6db93ddc773f07b74b9. This is a different declared revision, not assumed equivalent to the current checkout. Its claims guided checks, not acceptance. No old result archive, old source checkout or external paper/runtime was acquired. There is no origin incumbent in the assigned intake.

Consequential findings

  1. Deduplication failure is not whole-pass preservation; the public qualification is present. Historical wording about adaptation failures preserving the playbook could encourage an overbroad recovery reading. Current memory.md MEM-RTE-playbook-dedup and reconciliation still contain local “group intact” language. Frozen src/jaz_evals/harnesses/ace_dedup.py, similar_groups and deduplicate (especially lines 104–119 and 215–257), explicitly permit overlapping groups and use one pass-wide absorbed set. A failed group's member may be removed by another successful merge. P/report-verification.md#limits identifies this precise exception. P/synthesis.md, “Carry changed context” and its dedicated limitations row, explicitly reject pass-wide losslessness. This is a retained local wording limit, not an audit-only repair. No guarantee of semantic preservation or correctness is asserted by the synthesis.

  2. ACE scheduling and failure continuation survived the narrower account. Historical coverage named the 42-task adaptation cutoff and direct adaptation calls outside the solver budget. Frozen configs/appworld_ace_codeact_seed42_full.yaml lines 36–42 retain freeze_after: 42 and enabled deduplication. src/jaz_evals/harnesses/ace.py lines 476–551 test the freeze predicate, complete the task before adaptation, assign the returned playbook only on success, log adaptation exceptions and continue with the previous caller state. P/runtime.md RT-RTE-ace-adaptation retains the freeze and budget separation. The synthesis does not repeat the numeric setting, but makes no score comparison or equal-schedule attribution requiring it. The central implemented route is retained without claiming that advice was true or beneficial.

  3. Historical numerical comparisons are withheld rather than silently strengthened. The old review reported scores, statistical non-separation and bundled interventions. The new boundary explicitly says table numerical contents were not inspected; synthesis expressly withholds historical numerical reconstruction, causal attribution and rankings. Runtime RT-RTE-codeact-filter retains the combined history/input-binding intervention, consistent with inspected src/jaz_evals/hooks.py enter/send capture and message-rewrite code. Omitting the old numeric table is not a consequential coverage regression for this deliberately non-comparative conclusion. No old score or statistical statement was independently verified here.

  4. Reproduction and external-runtime limits remain public. Frozen README.md lines 156–164 say the installable jaz-lang==0.2.0a4 is not the literal historical code: development checkouts included uncommitted changes and missing commits. Boundary coverage explicitly distinguishes the installable release from historical checkouts. Synthesis opens with the unavailable runtime/submodule boundary and expressly rejects historical reproduction and whole-loop guarantees. Its limitations and closing evidence requirements prevent reading this as a reproduced result. The overview description also disclaims exercised improvement. Missing target runs, actual compliance, semantic dependence and causal benefit are stated in the public synthesis, not supplied by this audit.

Selected operational evidence

Parsed only completed tool_execution_end subagent results in T for the selected worker histories. Matched assistant tool calls to toolResult.toolCallId within the same worker's messages; did not count stream snapshots or proposed calls as execution. Line numbers below are physical JSONL lines; nested message indices are zero-based.

  • Runtime self-check and repair: T line 980, worker 000005-runtime, call call_aiFub3gszxeObfrniJTxRb8V (ID before the API suffix), result message 69: executed commonplace-run validate-output returned exit 1: runtime.md: quote-anchored citation at line 227: quotation not found ... (src/jaz_evals/harnesses/ace.py at the recorded commit). Subsequent paired edit call_1CKbbwrNJ1Ipl5U2FSYTZMdO, result 73, replaced the quote block successfully. Further paired edits repaired wording/anchors, including six blocks at result 79. Paired validation call_j3cSNW0broNwZAPeiDEPNTBQ, result 87, returned “Member content check passes; invocation residue not checked. This is not job acceptance or claim support.” Thus the initial failure is preserved, and a later pass is actually observed, not inferred from publication.
  • Verifier independently inspected the recovery mechanism: T line 2370, worker 000022-report-verification, messages 51→53 pair call_nysiDicsIrjOwWedB5k9GAVB for a successful read of frozen ace_dedup.py lines 1–300. Messages 58→59 pair call_vn1PXEZRk8iFiUFTDgc1sm0j for ace.py lines 443–702. Messages 90→91 pair call_9SZsvizlHfIe5PgCNxlYMB1P wrote verification with the overlap limitation. Messages 94→95 pair call_sl1IsPyzVekAbci5EqNpIHWm executed the member content check and returned the same qualified pass. This supplies actual independent source-inspection and output/check evidence for the consequential exception.
  • Synthesis check: T line 3394, worker 000028-synthesis, paired call call_tF05gbAx4MdTF245WIaRpg2u, result message 70, executed validate-output and returned the qualified member-content pass. Independently reading the retained synthesis confirms that the exception reached public prose.

All retained judgment JSON outcomes inspected were accepted; no retained semantic refusal was found. That is engine evidence, not a claim that all optional searches or required worker reads were completed. Selected histories establish a concrete quote failure, repairs, source reads and checks; they do not certify exhaustive compliance or absence of every scope deviation.

A present-day read of R/state/handouts/000022-report-verification/prompt.md failed with ENOENT: no such file or directory; recursive inspection found zero handout contents. No recovery was attempted. The trace retains the successful original prompt-read call/result and worker history, so the missing current handout does not erase that execution evidence. It limits direct filesystem reconstruction of the original invocation workspace.

Validation

Executed from W with the recorded absolute prefix:

/home/zby/llm/commonplace/.commonplace/worktrees/jaz-evals-17ab95f93794/.venv/bin/commonplace-validate --full kb/agentic-system-analyses/retained/jaz-evals
/home/zby/llm/commonplace/.commonplace/worktrees/jaz-evals-17ab95f93794/.venv/bin/commonplace-validate --full /home/zby/llm/commonplace/kb/work/agentic-system-reruns/audits/next-publications/AAS-2026-10-15-jaz-evals-17ab95f93794-01.md

Publication: PASS (clean), one directory artifact analysed; detailed output checks all 12 Markdown members and artifact roles/relations. Zero failures, zero warnings, INFO none; no validation-excluded subtrees. No source-unavailable notice was emitted. This does not substitute for semantic verification or certify every quotation by independent audit.

Report: PASS (clean), implicit text; no frontmatter or structural requirements. WARN none, FAIL none, INFO none. This check does not impose an audit-specific semantic schema.

Limits and return conditions

This is not a second full analysis, exhaustive source/quotation audit, runtime execution, statistical review or complete worker-obligation audit. No target code was run. External implementations, benchmark submodule contents, original evaluation traces and causal experiments remain outside the inspected evidence. Full handout reconstruction is unavailable as noted above.

No unblock action is required for the selected disposition. Parent acceptance must preserve the existing public qualifications and recheck the exact bytes before integration. A proposal to publish rankings, reproduced scores, exercised improvement, universal budget/isolation guarantees or lossless dedup recovery would require separate authorized evidence/revision and renewed review; the present audit does not authorize those claims. Any broader source or ownership choice returns to the parent rather than expanding this audit.