Napkin amendment-effects audit

Outcome

The quotation and validator amendments were delivered and visibly adopted. All 18 worker validator invocations were direct shell calls; both failures remained visible and recovered. The old revision-attribution failures did not recur. One incorrect quotation range still required repair. Absolute handout generation also worked, but worker adherence did not: all 14 analytical workers shortened supplied paths, and one reconstructed artifact path caused a visible validation failure. No masked failed shell command was found in this run.

The accepted result preserves the supported account of local file retention, requested retrieval and host-interpreted curation without demonstrated benefit. It now qualifies the distill create-example mismatch and separates saved reports from current-run aggregation. Documented automatic delivery is better represented but still incompletely classified. Semantic review again repaired the exposure-evaluator account, and also repaired default CLI score delivery. These are successful safeguards, not flawless first drafts. Publication bytes match acceptance. The run took 75m 31.9s, versus 49m 22.2s for the preceding same-source run.

This is observational evidence of amendment uptake and remaining failure mechanisms, not causal evidence of improved quality or speed. All three amendments changed together; source inspection, repair work, model behavior and concurrent service conditions also varied.

Audit date: 2026-10-11. Commission: Audit an analysis, narrowed by the operator to this report only. The coordinator owns the workshop README and cross-run synthesis. Frozen sources, method, analytical outputs, traces and logical run state were not edited. No analysis rerun, target execution, integration, commit or delegation was performed.

Identity and evidence conventions

run-id: AAS-2026-10-11-napkin-28752ba411ea-01
method-commit: 48453a8110bd36bc51fbcad3e9b2b394b694f603
source-revision: 7582d6a46f5a11995956e60a59c41a5b242109f1
source-identity: https://github.com/Michaelliv/napkin
worker-profile:
  profile: pi-sol
  harness: pi
  launch-model: gpt-6.1-sol
  model: openai/gpt-6.1-sol
  reasoning-effort: medium
reporter-profile:
  harness: pi
  model: openai/gpt-6-astra
  reasoning-effort: medium
  basis: PI_CODING_AGENT, PI_PROVIDER, PI_MODEL, PI_REASONING_LEVEL

Settings beyond those exposed fields, including sampling and provider scheduling, are unknown. Worker identity is supported by the manifest, all 14 model-attempt identity fields and embedded environment outputs, not inferred from the launch alias.

Local locators:

  • W/: /home/zby/llm/commonplace/.commonplace/worktrees/napkin-28752ba411ea/.
  • R/: W/kb/agentic-system-analyses/state/AAS-2026-10-11-napkin-28752ba411ea-01/.
  • A/: W/kb/agentic-system-analyses/retained/napkin/.
  • S/: W/related-systems/Michaelliv--napkin/.
  • T: /home/zby/.pi/agent/sessions/--home-zby-llm-commonplace--/2026-10-11T15-32-12-661Z_01a12b97-d2f5-7274-939c-cb3181407ed7.jsonl.

L means a 1-based JSONL line, r a zero-based message.details.results index, and m a zero-based embedded messages index. Omitted r is zero. All original analytical traces are embedded in T; the diagnostic at L7 is excluded. Handout directories were cleaned up, but delivered prompts, input reads, commands, results and authored outputs survive in the embedded messages. Version blobs and attempt records survive under R/state/.

Same-source comparators

Run method-commit Profile Comparison basis
AAS-2026-10-10-napkin-b831630d0978-01 6ff7c828e94dd66abdf5af00c89db6420bc6a1ad pi-sol / openai/gpt-6.1-sol / medium First audit, subsequently independently byte-checked by the repeat audit
AAS-2026-10-11-napkin-1b60c5d21780-01 0275b8abb307039d0bc975b5699e5b86fe6ca1d5 Same Previous repeat audit
Current run 48453a8110bd36bc51fbcad3e9b2b394b694f603 Same New full analytical failure scan and focused result audit

All three source checkouts independently resolve to 7582d6a46f5a11995956e60a59c41a5b242109f1 with empty source Git status. The prior worktree names are napkin-b831630d0978 and napkin-1b60c5d21780 under the same parent. The previous audit supplies exact prior trace paths and old finding anchors. This audit re-extracted prior validator commands and rechecked all three publications; other prior trace conclusions are attributed to that audit rather than presented as a second complete scan.

Exact method boundary and delivery

T L15 preparation, R/state/attempts/000001-opening.json and the publication receipt agree on the current method commit. Between the preceding repeat and this run, the diff under the analysis collection and artifact-run implementation changes only:

  1. d823ef6216ee0d7d3e49bc7347d1d9e75022b926: path-only quotation examples in jobs-engine/follow-worker-rules.md, with checked ranges or longer quotations for ambiguity.
  2. 4eed3876d26e41a4d0100a50e308237facbdf78e: separate direct validator invocation in jobs-engine/prompt-section.md, explaining trailing-print status masking.
  3. b38982da28b66bdbf0ccaf987cf8f9ae0edc6882: absolute generated filesystem paths in src/commonplace/artifactrun/handouts.py.

Other repository commits include tests, source/workshop material and the previous audits; this is not a whole-checkout three-file-only comparison. All three runs have plan SHA-256 9062207fdc5fecf8ffd5e697924c52228125fd7c0a73d3e7c0ebcfa03cfac6a2. The later workflow-lens contract is not part of this method and was not applied to this historical set.

Current committed bytes match the worktree's three changed files. Worker-rules SHA-256 is 33a4dbe4a6ee7d2e5feb3f02e1c84c0d0ecd7c5abac71a68c763069682b949c6; prompt-section is a2fa1f6574b7c54ab2ac9d0e4d640c25c9af95a649be963e6075e2f7c47b7b6b; handouts implementation is 449d49b8119b0c45af24fbbb84fa7fbdff1300662b331f70aa9d72e8efc8797d. Boundary and report-verifier attempt pins name the new worker-rules hash. The report-verifier instruction and member type remain the previous repeat's 41163de951b29115cfad318fa1f3baf7b36a94ee730dbb93b2d91397145490b2 and 6ea2cb1ca1f41e26bbe835f6d20ac30f74627d962454b0e304de98308761432e.

All 14 embedded prompts contain the direct-invocation advice and absolute filesystem assignments. All workers read the worker rules. T L21 m3 is a complete concrete delivery example; L21 m4 and m6 load its instruction and input batch. Presence in Git alone is therefore not the adoption evidence.

Amendment opportunities, adoption and recovery

T1 — Path-only quotations: adopted; range accuracy still matters

The first authored primary outputs containing quotations are boundary (T L21 m60), runtime (L25 m56), memory (L29 r0 m79), epistemic (L29 r1 m76) and first report verification (L39 m130). Across these five first drafts, all 96 attribution lines contain only a backticked source path, optionally a range. No explicit revision/SRC decoration was added. This count measures attribution formatting, not independent claims, unique passages or successful grounding. One of the 96 carries a range.

That range fails. Memory's src/core/daily.ts:96-99 excludes part of its quoted conditional. T L29 r0 m82 reports quotation-not-found with exit 1. The worker rereads lines 87–111 at m83–84, expands the quote to include the distinctive appendDaily declaration and drops the range at m85. Validation passes at m88. The claim is retained, not weakened to satisfy the checker. The failure-to-pass window is 26.591 seconds.

The final four source-quoting members contain 99/99 path-only attributions, all without ranges: boundary 5, runtime 22, memory 25, epistemic 47. These are final occurrences, not additive to the 96 first-draft occurrences. No attribution/revision-resolution failure appears among the 18 validator invocations. The earlier runs each had three quotation-related failed validations among 17 invocations; in the preceding repeat, two boundary failures and one epistemic failure concerned attribution syntax. This new observation is consistent with the amendment addressing that narrow mechanism. It does not establish a lower semantic error rate, and the remaining range failure demonstrates why checked uniqueness still matters.

T2 — Direct validation: complete invocation-style uptake

18/18 worker validator calls invoke the supplied worktree executable directly in their own shell call, without a trailing status print or pipeline. Seventeen use the supplied command correctly; one has a mistyped artifact argument. Results: 16 passes, two failures, in two workers; 14/14 workers eventually pass. Both failed tool results have isError: true and explicit nonzero exit text. All validator result anchors are retained below so the denominator can be reconstructed:

Worker trace Validator result message indices
L21 m63 pass
L25 m59 pass
L29 r0 m82 fail, m88 pass
L29 r1 m79 pass, m85 fail, m87 pass
L33 m45 pass
L39 m133 pass
L43 m43 pass
L47 m54 pass
L51 m57 pass
L55 m134 pass
L59 m55 pass, m61 pass
L63 m65 pass
L67 m64 pass
L71 m55 pass

The preceding repeat used 4/17 bare direct calls and 13 wrappers; 12 wrappers explicitly returned the captured status, while one trailing-print wrapper was risky but actually passed. The original run used 3/17 direct calls and 14 wrappers; 11 returned captured status and three were risky but actually passed. Here “direct” excludes even a correctly status-preserving wrapper. Those safe old wrappers are not failures. The original run's demonstrated masked failure was a separate source-inventory command, not one of these validators.

The new run has zero demonstrated masked failures among 48 Bash calls. Four Bash calls report nonzero status: two validators and two source-path probes. Boundary's missing .pi path check at L21 m38–39 supports the documented unavailable integration; it is an expected negative, not a failed source acquisition. Epistemic's inventory at L29 r1 m23–24 guesses nonexistent src/core/query.ts and src/core/graph.ts; the final wc exposes exit 1. The worker subsequently reads the real Base and graph paths. Unlike the original runtime inventory error, this error was not hidden by a final successful print.

Some commands prepend identity printing before cd and an && chain. That is not evidence that a consequential source command failed invisibly: the source chain's status remains the final status. No failed pipeline or trailing-print masked validator was observed. Absence of an observed masked failure is not a guarantee about every possible shell failure.

T3 — Absolute handouts: generation succeeds, worker compliance remains poor

All 14 prompts supply absolute paths for inputs, outputs and the validator's artifact. Nevertheless, 14/14 workers make at least one relative spelling of a supplied filesystem path. This is now a demonstrated rule/behavior mismatch, unlike the earlier runs where generated output paths themselves were relative.

A reproducible call-level denominator matches each executed read/write/edit path, after prefixing the worker cwd when relative, against absolute paths present in that worker's delivered prompt or launch task. It excludes discovered target-source paths, scratch children, Bash arguments and the two incomplete provider-emitted calls that never received tool results:

Operation on exactly supplied paths Absolute spelling Relative spelling Total
Read 13 266 279
Write 23 9 32
Edit 3 3 6

Most absolute reads are the initial prompt reads. Profile generation even reads its initial prompt relatively (T L59 m2). For example, the absolute boundary assignments at L21 m3 are shortened at m4/m6; relative runtime output writes occur at L25 m52/m54/m56. These paths resolved in the correct fixed cwd. The audit found no wrong-file read or unauthorized write from those relative spellings. This count is spelling adherence, not a count of 278 independent harmful incidents.

There is one actual reconstructed-path consequence. After a first pass and semantic self-edits, epistemic invokes validation at T L29 r1 m84 with artifact /home/zby/llm/commonplace/.commonplace/worktrees/napkin-28752ba411ea-01/artifact, omitting the real run subtree. The supplied artifact path was correct. The validator reports missing ARTIFACT.yaml, exit 2, at m85. The worker restores the full supplied path at m86 and passes at m87, 7.302 seconds after the failure. This is a recovered worker transcription/reconstruction error despite correct delivery, not an absolute-handout generator defect or a masked check.

The amended generator removes ambiguity about which path should be used; it does not force reuse of that path. These traces do not test worker behavior after a cwd change, because analytical launches all use the prepared worktree cwd.

T4 — Semantic repair, provider recovery and bounded reading

The first report verifier raises three addressed blockers at T L39 m130: runtime CLI score delivery, epistemic exposure wiring and reconciliation's dependent exposure statement. It also identifies the distill create-example mismatch as a publishable local limit. Runtime repair at L43 m35 corrects score delivery and incorporates the path-example qualification; epistemic repair at L47 m47 corrects the caller/loader account and dependent fields; reconciliation is revised at L51 m52. Final report verification at L55 m131 accepts the bounded result. These repairs preserve the IDs and unrelated analysis, rather than changing the target or erasing disputed records.

This reintroduces an exposure correction cycle absent from the preceding repeat, although the final qualification agrees across all three runs. It also catches a different selection/delivery overstatement. Three blocker addressees are not three independent runs or necessarily three independent root defects.

Three embedded assistant messages end with OpenAI Responses stream ended before a terminal response event: runtime repair L43 m37, profile generation L59 m49 and synthesis L67 m23. All recover inside their original successful workers. The profile's incomplete write and synthesis's empty-argument read have no corresponding tool result and are excluded from executed-call counts. Their subsequent complete calls are L59 m50 and L67 m24. All 14 outer results finish with exit 0, normal stop and empty stderr; those envelopes alone would conceal these recoveries.

No actual Bash delivery-truncation marker was found. Workers chunk large supplied reports and recover substantive continuations. Synthesis leaves advertised one-line continuations for memory, epistemic and profile at L67 m29/m37/m45; comparison with the accepted blobs identifies empty EOF remainders, not omitted substantive lines. Source excerpts are intentionally bounded and are not treated as failures merely because more source lines exist. This is not a certificate of complete source coverage or comprehension.

No explicit analytical writes outside assigned outputs, identities or scratch were found. The profile executes its own scratch generator, which writes only its assigned profile and answers; it does not execute Napkin. Correction workers copy supplied previous outputs into their own outputs. No analytical target execution, network acquisition, nested delegation, publication command or prior-audit read was found. Coordinator publication instructions were read at T L34–35 before the remaining verification/publication sequence.

Accepted result: focused source checks and old findings

The historical frozen record and profile contracts govern this audit. Their distinction between retained-file consumption and current-run state, and between claimed and wired mechanisms, matters to the findings below. This is not a new comprehensive characterization of Napkin. Direct checks use the commissioned frozen source rather than creating or changing source-ingest records.

C1 — Distill path mismatch is now explicitly retained

The old N1/C1 finding is supported by S/skills/distill/SKILL.md:79-86, CLI forwarding at S/src/main.ts:230-241 and S/src/commands/crud.ts:54-62, and full-path selection at S/src/core/crud.ts:46-55. A supplied folder is not combined with the title. The new runtime's RT-RTE-note-write and RT-RTE-session-distill, final report-verification Limits and synthesis all state that constraint. The trace shows independent discovery during verification and incorporation during runtime repair.

This resolves the main previously omitted create-example qualification. The related name-only overwrite/nested-basename issue is not separately described; that narrower residual does not negate the newly retained warning. No executed distillation failure is claimed. The verifier's limit disposition is reasonable because the reports do not infer successful literal skill execution from the example.

C2 — Saved results no longer inherit current-run aggregation authority

S/bench/longmemeval-eval.ts:628-632,661-694 and S/bench/locomo-eval.ts:439-445,454-495 separately append result objects to allResults, persist logs and aggregate the current arrays. They do not establish reloading those logs.

Unlike the preceding repeat's residual wired-report finding, A/memory.md:164-178 expressly names current result arrays and limits report reuse. A/memory-profile.md:800-809 emits no positive authority for later benchmark-report consumers; :1206-1213 leaves later report consumption not determinable. The storage and encoding positives do not upgrade consumer evidence. Old N2/C2 is resolved in the inspected result. A future external reader remains possible, not disproved.

C3 — Documented delivery is partly recovered, still underclassified

A/memory.md:315 now records the timer-driven separate model reading new entries, root context and templates. A/memory-profile.md:1186-1198 emits claimed push for always-loaded host context and retains partial direction coverage rather than the preceding repeat's known pull-only scope. This is substantive improvement in representation, not proof of implementation.

However, :1199-1205 leaves the timed-model chain wholly not determinable, and :1231-1254 withholds every push-selection finding because actual scheduling/selection is unavailable. S/docs/distill.md:11-24,44,75,79-82 explicitly documents the timer/new-entry condition, prompt assembly and configurable template selection (empty means all templates). Missing integration prevents a wired assessment, not retention of that claimed automatic supply chain. The new analyst record already contains more of the needed facts than the preceding repeat did.

Disposition: old N3/C3 is partly addressed, not fully closed. Recommend a bounded local clarification of the documented chain at claimed strength, preserving unknown implementation and checkpoint details. Exact selector classification should follow the described operation rather than treating a filename alone as identifier push. Do not force an unsupported precise value merely to replace uncertainty. This is not grounds to infer observed automatic delivery or overturn the central pull-retrieval account.

C4 — Smaller old qualifications

  • Trash search (old N4/C4): S/src/utils/files.ts:28-54 skips encountered trash directories but starts directly at a selected folder; the search loader accepts that folder. A/memory.md:301 still discusses ordinary recursive discovery and exact-path reading without the separate --path .trash search case. The default-search reading is defensible; an explicit qualification remains useful, not a newly established universal-access falsehood.
  • Empty root context (old N5/C5): A/runtime.md:16 again says bare-vault construction creates empty NAPKIN.md. S/src/utils/vault.ts:91-94 creates it only if absent. This is a low-priority missing condition; it does not support a claim that existing context is erased. The preceding repeat's modal wording was less categorical.
  • Successful-result denominators (old N6/C6): A/memory.md:329 now explicitly says failed questions are excluded from scored result arrays. The independently checked means above consume those arrays. The epistemic member is less explicit about means than the preceding repeat, but the accepted set retains the material exclusion, so this is not disappearance of the qualification. Actual published-score bias remains unknown without attempts and outcomes.

C5 — Correct final evaluator and delivery qualifications

S/bench/overview-exposure.ts:20-22 passes two strings, whereas S/src/core/search.ts:160-165 expects and destructures a vault object. A/epistemic.md:515-559 correctly separates present metric code from functioning corpus acquisition. The repaired runtime also correctly distinguishes SDK scores from default CLI JSON/human payload: S/src/commands/search.ts:39-59 strips scores unless opted in; S/src/main.ts:218 registers the option. The final synthesis carries both distinctions.

A further focused check supports the scoring limitation: S/bench/longmemeval-eval.ts:196-209 can accept a negated answer containing the reference before requesting a judge; :233-260 uses the same selected model, disables extensions and falls back to token F1 on subprocess failure. The report correctly treats such countercases as static restrictions on the score's meaning, not measured failure incidence.

Profile verification retains a local Canvas-removal inventory limit and synthesis carries it. This audit checked that propagation, but did not independently audit all Canvas operations. A carried partial-inventory limit is not automatically a historical completeness-contract failure.

Publication integrity and validation

Independently hashed all 12 published files in each of the three runs: eleven Markdown members and ARTIFACT.yaml. Every Markdown member matches its manifest entry, publication receipt, corresponding content-addressed version blob and working artifact bytes. Each published manifest matches its receipt and version blob. No mismatch was found. The working artifact's minimal manifest is not the assembled publication manifest and was not falsely compared as if it were.

Current exact receipt: R/state/attempts/000040-publish.json, output version bd25ee4b19c850e86d3215f13898b9b8abf616b51fddc2fa32fb909030dd4afb. Published manifest hash: e47a270f43f394ebf11976b28b3583ed33d58f6490c192bf1c699962e9102c05.

From W, ran its frozen .venv/bin/commonplace-analysis inspect R and .venv/bin/commonplace-validate kb/agentic-system-analyses/retained/napkin with the actual run path substituted. Inspection reports complete, publication-job-completed, publishable, no open/failed/exhausted attempts and no canonical-peer drift. Historical stale acceptances remain, as expected after replacement; they are not current refusals. Inspection explicitly does not freshly verify effect-journal completion, so the separate byte check matters. The pre-existing lock was present before inspection; no run advance or acceptance command was used.

Frozen validation output: VALIDATION SUCCESS, Files analysed: 1, Text files: 0, zero failures and warnings. One here is the set target, not an assertion that only one publication member was hashed. Form, quotation occurrence and byte equality do not certify semantic truth.

Timing and costs

Wall duration is coordinator start-result timestamp through the advance result reporting publishable after publication: T L17 15:33:23.758Z to L73 16:48:55.621Z, October 11. Worker span is last minus first timestamp in each embedded message array, including model/tool wait but excluding some launch/return overhead. Memory and epistemic overlap. Summed spans are neither CPU time nor wall time.

Measure Original Previous repeat Current
Wall duration 2,317.702 s 2,962.171 s 4,531.863 s
Summed worker spans 2,420.020 s 3,198.127 s 4,701.458 s
Analytical workers 13 10 14
Embedded messages 836 713 1,036
Executed tool calls 505 437 637
Reads / Bash / writes / edits 419 / 47 / 33 / 6 364 / 40 / 27 / 6 549 / 48 / 33 / 7
Direct target-source read calls 172 177 270
Harness model turns 305 256 371
Input tokens 991,770 898,558 1,394,871
Cache-read tokens 12,145,408 11,779,072 18,157,184
Output tokens 93,797 91,959 119,182
Harness-reported worker cost $4.1360508 $3.8946132 $5.7972804

Current executed calls require a matching tool-result ID. Counting all emitted tool-call blocks instead gives 639 (550 reads, 34 writes), because the two interrupted provider emissions never executed. Source-read counts include absolute and worktree-relative spellings containing related-systems/, not only absolute paths. They measure calls, not unique lines or understanding. Prior totals come from the previous audit; current usage is summed from the 14 outer worker usage mappings, not inferred billing.

Attempt T anchor Span, seconds
000003-boundary L21 217.655
000005-runtime L25 339.677
000007-memory L29 r0 507.080
000008-epistemic L29 r1 829.009
000014-reconciliation L33 190.771
000017-report-verification L39 436.270
000019-runtime repair L43 137.008
000021-epistemic repair L47 194.894
000026-reconciliation repeat L51 192.803
000029-report-verification repeat L55 359.125
000031-memory-profile L59 604.343
000033-profile-verification L63 238.468
000035-synthesis L67 276.653
000037-synthesis-verification L71 177.702

Wall time is 53.0% longer and summed spans 47.0% longer than the preceding repeat. The four post-refusal repair/review workers contribute 883.830 summed seconds. That is successful safeguarding work, not isolated time spent on one defect; it excludes the first verifier that detected the defects. The two validator failure-to-pass windows total 33.893 seconds, but are already included in worker spans and must not be added again. Provider errors also occurred; their independent contribution to duration was not isolated.

Concurrent Graphiti and Letta Code sessions began at 2026-10-11T15-31-16-802Z_01a12b96-f8c2-745a-b8d3-80b534c425e7.jsonl and 2026-10-11T15-31-44-641Z_01a12b97-6580-74ff-ac08-71cf6ff28392.jsonl in T's session directory, with analytical workers overlapping this run. No queue, load or latency telemetry separates contention from model variability. More source reading, a larger profile, additional repairs and service recovery all accompany the longer run. Neither eliminated attribution retries nor direct validators establish an overall speed benefit in this uncontrolled batch.

Disposition and limitations

Retain the three amendments as successfully delivered changes with different observed uptake. Quotation formatting and direct validation show the intended behavior; correct absolute delivery does not yet imply worker reuse. This case supports no new general instruction solely from its occurrence. In particular, it does not justify reverting absolute paths because a worker reconstructed one incorrectly.

For a separately authorized result correction, prioritize C3's evidence-strength distinction; keep the new C1/C2 corrections. C4's default-search and absent-file wording are inexpensive precision improvements, not central invalidations. C3 would be rebutted by an authorized exclusion of the documented mechanism or evidence that the cited prompt-assembly description does not concern retained parts; missing implementation alone is not that rebuttal. Adding a general exhaustive checklist would impose reading and reporting cost without evidence here of superior detection.

All 14 new analytical traces were scanned for commands/results, validation, errors/recovery, path spellings, mutation destinations and reading continuations. Source correctness checks concentrated on previous findings, the new repair cycle, dependent profile/synthesis statements and selected scoring paths. This is not an independent full reanalysis of every route, quote, formula, dependency or benchmark. Deployed distillation, exact external consumers, actual score bias, hidden activity and causal amendment effects remain unknown. Separate events within this run are not independent replications.

Only this report is authored. Origin-checkout commonplace-validate kb/work/analysis-audit-evidence/reports/napkin-28752ba411ea-amendment-effects-audit.md returned VALIDATION SUCCESS: one text file, zero failures and zero warnings. That check governs the report's form and links, not these judgments. Broader graph discovery through cp-skill-connect is a possible separately authorized next step, not performed within this report-only commission.