Letta Code 9d060e7dab7e amendment-effects audit
Judgment
The three amendments reached the workers, but adoption was uneven. Path-only quotations and direct validation were used successfully. All seven failed validation calls propagated failure; none was masked by a trailing status printer. Absolute handouts did not prevent workers from reconstructing paths: five validator calls used a shortened, nonexistent artifact path, and Memory briefly wrote an empty identity file outside its assigned handout before deleting it and writing the correct file.
The accepted result passes frozen-method validation and independent publication-byte checks. Focused source checks support the repaired scheduling and reflection claims and the retained committed/live-memory distinction. No new substantive correction to the accepted claims was established in this sample. The result is not a complete replacement for prior coverage: the old explicit hook ERROR/BLOCK distinction and Workflow security account are still not equivalently retained.
Completion required eighteen analytical workers, versus twelve in each prior completed run. Preparation-to-publication took 89.67 minutes, versus 64.38 for the immediate predecessor. Two report-repair cycles, followed by a profile repair, explain substantial additional work. This uncontrolled batch establishes neither causal improvement from the amendments nor a method-caused slowdown.
Identity, authority and evidence
audit-date: 2026-10-11
run-id: AAS-2026-10-11-letta-code-9d060e7dab7e-01
method-commit: 48453a8110bd36bc51fbcad3e9b2b394b694f603
source-revision: 271f119dcf37d58649a18e6c733386b3c710a456
source-identity: https://github.com/letta-ai/letta-code
worker-profile: pi-sol
worker-runtime: {harness: pi, model: openai/gpt-6.1-sol, reasoning-effort: medium}
reporter-profile: {harness: pi, model: openai/gpt-6-astra, reasoning-effort: medium}
Reporter settings come from this audit's exposed PI_PROVIDER, PI_MODEL and PI_REASONING_LEVEL. Other provider settings, immutable deployed model versions, cache state and sampling parameters are unexposed. All eighteen analytical result envelopes name gpt-6.1-sol/medium and exit zero. The publication manifest and populated attempt worker_model/worker_effort fields agree; nullable generic model fields are not contrary evidence.
This is the commissioned result-and-trace audit under Audit an analysis and the workshop commission. The work collection permits this frontmatter-free evidence report. The writing and validation skills and frozen collection, set, verification and profile contracts informed it. Direct frozen-source inspection was expressly commissioned; this is evidence about the run, not a new external-system ingest. No ancillary source record, brief or index was authorized. The coordinator owns navigation and cross-run synthesis.
Locators:
W:/home/zby/llm/commonplace/.commonplace/worktrees/letta-code-9d060e7dab7e.R:W/kb/agentic-system-analyses/state/AAS-2026-10-11-letta-code-9d060e7dab7e-01.A:W/kb/agentic-system-analyses/retained/letta-code.S:W/related-systems/letta-ai--letta-code.T:/home/zby/.pi/agent/sessions/--home-zby-llm-commonplace--/2026-10-11T15-31-44-641Z_01a12b97-6580-74ff-ac08-71cf6ff28392.jsonl.P: the corresponding worktree, state and retained paths forletta-code-f54d2479889e;O: those forletta-code-88afa3263646.
T L<n>/R<n>/m<n> means one-based JSONL line, zero-based message.details.results index, and zero-based embedded message index. Bare worker line references below mean R0; L29 also has R1, Epistemic. R/state/versions/<hash> retains historical output bytes; the named attempt's outputs maps its products to those hashes. These locators survive handout cleanup, but local worktree paths are not run identities.
I scanned all 18 analytical transcripts, including every repair and re-verification: 1,278 messages and 745 tool results, comprising 608 reads, 88 shell calls, 41 writes and 8 edits. This is a failure-, amendment- and authority-oriented scan, not independent semantic verification of all 608 reads. The diagnostic and later coordinator inspection are excluded. I read the accepted synthesis and verifications, selected profile/record content and historical refusals, and independently checked selected source mechanisms. Prior reports supplied historical timing and failure measurements; they are not independent runs or a blind comparison.
A1 — Exact comparison boundary and amendment delivery
Run suffix; full run IDs use AAS-2026-10-11-<suffix>-01 |
Method commit | Evidence status |
|---|---|---|
letta-code-88afa3263646 |
0265fff4dd8a9d00fa2aba78495b2c95b2abbad6 |
Completed; first audit |
letta-code-f54d2479889e |
0275b8abb307039d0bc975b5699e5b86fe6ca1d5 |
Completed; repeat audit |
letta-code-73f448c65be3 |
b38982da28b66bdbf0ccaf987cf8f9ae0edc6882 |
Preparation/start only in inspected state: opening/acquisition completed, Boundary open, no publication |
letta-code-9d060e7dab7e |
48453a8110bd36bc51fbcad3e9b2b394b694f603 |
Completed audited run |
All use the same source revision above and pi-sol launch profile. The incomplete 73f preparation records dirty-origin exclusion and has no reported analytical worker model; do not count it as another completed amendment trial or invent its abandonment cause. W preparation records a clean origin. W HEAD matches preparation; S HEAD matches the boundary and source working status is clean. Prior completed identities and profiles are corroborated by their run/manifest records and previous audits, not re-estimated from elapsed time.
The P-to-W method/code diff is exactly three files: jobs-engine/follow-worker-rules.md, jobs-engine/prompt-section.md under the analysis instruction, and src/commonplace/artifactrun/handouts.py. They implement:
d823ef6216ee0d7d3e49bc7347d1d9e75022b926: path-only quotation attribution; the frozen boundary supplies source revision; checked ranges remain available for ambiguity.4eed3876d26e41a4d0100a50e308237facbdf78e: run the supplied validator directly in a separate shell call; explicitly warn that a trailing status printer can mask failure.b38982da28b66bdbf0ccaf987cf8f9ae0edc6882: absolute handout paths and run-derived artifact/workspace substitutions, without converting source-relative citations.
No new analyst lens or completeness obligation belongs to W. In particular, the later workflow-lens contract was not used to judge this historical set.
Delivery is visible, not inferred merely from ancestry. Attempt 000017-report-verification pins worker rules 33a4dbe4a6ee7d2e5feb3f02e1c84c0d0ecd7c5abac71a68c763069682b949c6 and prompt section a2fa1f6574b7c54ab2ac9d0e4d640c25c9af95a649be963e6075e2f7c47b7b6b; independent hashes match W's files. Profile and profile-repair attempts pin the same bytes. Each of the eighteen embedded prompt results at m3 contains the direct-call instruction and absolute filesystem bindings. Examples at L29/R1/m3, L39/m3 and L67/m3 supply the complete correct --artifact path subsequently mistyped by those workers. The old reports' relative-output observation was a handout/rule mismatch; these new shortened paths are not supplied that way.
A2 — Quotation amendment: adoption without eliminating ambiguity
All nine source-authorized output attempts use path-only attributions: Boundary, initial Runtime/Memory/Epistemic, Runtime and Epistemic repairs, and three report verifications. Their content-addressed output versions contain respectively 3, 16, 32, 22, 19, 26, 6, 7 and 3 attribution lines: 134/134 are a single code span containing a source-relative path, optionally a range. This denominator counts repeated quotations in successive output versions, not unique passages, independent opportunities or first drafts. The published five quotation-bearing members contain 83/83 such lines: boundary 3, runtime 19, memory 32, epistemic 26, report verification 3.
There are zero attribution-format failures among 26 worker validator invocations, but two ambiguous-passage failures:
| Worker/result | Diagnostic and repair |
|---|---|
Runtime T L25/m69 |
Provider-model resolution passage occurs twice in src/backend/dev/pi-stream-adapter.ts; attribution range repair at m70–71; pass m73. |
Epistemic T L29/R1/m62 |
Compaction-store call occurs twice in src/backend/local/local-backend.ts; checked range repair and scratch account at m63–66; pass m68. |
Both failures propagate as isError: true. They test a remaining necessary uniqueness check, not failure of source revision resolution. Boundary and Memory pass their first validators; Memory later adds explicit root-selection support and refines descriptions of its inspected coverage, then passes again. Its intervening edit mismatch is separately counted below.
The P audit reports eight failed content checks, including five attribution-format failures; O reports seven failed checks. W has only two content failures, plus five invocation-path failures. These are different categories. The observations are consistent with the intended attribution simplification, but scope, selected quotes and workers changed. They do not establish a causal reduction or a population error rate.
A3 — Validator status amendment: complete direct-call adoption in this scan
All 26/26 worker validator calls are standalone direct commands, with no trailing printer, pipeline or status-replacing wrapper. Seven fail and nineteen pass, across eighteen workers; all seven failures have isError: true. Every worker eventually passes. Thus 0/7 demonstrated failed validations are masked, and 0/26 calls use the previously problematic wrapper. The denominator includes the second successful Memory check and five calls that fail before content validation.
Besides the two quotation failures, five calls shorten --artifact to the nonexistent .../worktrees/letta-code-9d060e7dab7e-01/artifact:
| Attempt | Failed result | Corrected successful result |
|---|---|---|
| 000008 Epistemic | L29/R1/m60 | m62 reaches content checking but fails ambiguity; m68 finally passes |
| 000017 Report verification | L39/m118 | m120 |
| 000026 Report re-verification | L51/m98 | m100 |
| 000037 Profile | L67/m50 | m52 |
| 000043 Profile re-verification | L79/m66 | m68 |
All five errors say the artifact's ARTIFACT.yaml is missing and exit 2. The supplied commands were correct. This is 5/26 validator calls, affecting 5/18 workers, not five broken handouts. The repaired invocations use the full run-state artifact path. There is no demonstrated acceptance bypass. Epistemic's scratch repair account at L29/R1/m65 incorrectly says the mandated command was executed verbatim and differed from the artifact binding. The actual prompt at m3 contains the correct full path in both places; the executed call at m59 shortened it. Preserve this as an inaccurate worker recovery explanation, not evidence of a producer defect.
The broader tool scan finds nine failed search shell calls, all propagated: Runtime L25/m17,m23,m39,m47; Memory L29/R0/m61,m63,m77; report verification L39/m85 and L51/m63. Missing guessed source paths account for several; Memory m63 is a no-match status. Memory's edit L29/R0/m102 also fails an exact-text match and succeeds after correction at m104. In total, 17 tool results expose errors: seven validators, nine searches, one edit. All eighteen worker envelopes nevertheless finish successfully. No embedded assistant stopReason: error or provider-stream failure appears in these eighteen transcripts; P had one recovered stream interruption.
No demonstrated masked shell failure was found among the 88 shell results. That is not proof every component succeeded. Boundary L21/m25, Runtime L25/m24 and Epistemic L29/R1/m32,m46 still use head/tail pipelines without pipefail; Epistemic m25,m32,m46 also separate searches with semicolons. Their final zero cannot establish every upstream status. Count these as risky constructions, not invented hidden failures. The narrower validation amendment was followed even where broader shell discipline remained imperfect.
A4 — Absolute paths: delivery succeeds, worker copying does not always follow
Among 49 write/edit tool calls, 37 use absolute paths and 12 use relative paths. The twelve occur in six workers:
- Memory: L29/R0/m89.
- Epistemic: L29/R1/m55,m63,m65.
- Reconciliation repair: L47/m44,m46.
- Report re-verification: L51/m93,m95.
- Profile: L67/m45.
- Profile re-verification: L79/m59,m61,m63.
This denominator excludes shell cp, generated Python writes and reads. Most relative spellings still resolve to the correct assigned handout in the launch cwd. That is a spelling-rule deviation, not automatically a misdirected write. It differs from P, where supplied output paths themselves were relative.
One misdirected write is established. Memory L29/R0/m89 writes empty content to kb/agentic-system-analyses/state/AAS-2026-10-11-letta-code-9d060e7dab7e-01/worker-identity.json, outside its assigned state/handouts/000007-memory/ directory. Tool result m90 confirms the write. It deletes that file with an absolute rm at m91–92, then writes the correct identity at m93–94. The supplied m3 binding was already correct and absolute. This is a recovered write-scope violation, not merely a risky wrapper or a changed retained member. The trace does not establish that the wrong path had pre-existing valuable bytes, so no data-loss claim follows. Cleanup does not retroactively authorize the write.
Together with the five malformed artifact arguments, this limits the path amendment's demonstrated benefit: it fixes producer delivery, not worker transcription. It does not justify reversing absolute handouts. A possible separately authorized test would compare exact command copying with reconstructed paths; expected benefit is fewer path mistakes, cost is extra tooling or constraints, and a lack of mistakes under ordinary copied commands would weigh against adding complexity. This report authorizes no such test or workflow edit.
Boundary's genuinely truncated tree listing at L21/m13 recovered its omitted prefix through reads at m14 and m16 (lines 1–650 and 651–1289). Later strings such as “Output truncated” inside source are not automatically harness truncation events. The scan did not establish another unauthorized mutation, source execution or nested launch. It is not a complete proof of every source-read or mandatory-input obligation.
A5 — Accepted result, repairs and focused source checks
Semantic review repaired three distinct problems
Four refused judgments represent three substantive problems, not four independent detections:
- False shell/file serialization assurance. Initial report verifier output
16f4c4253d497a1ea93e284633426d6b10f8aee57976fbce14ec5731f439c6c7, applied at judgment000018-apply-report-verification-1.json, refuses Runtime. Runtime repair 000019 and reconciliation 000023 are followed by verifier 000026. Independent source check:S/src/agent/approval-execution.ts:77–120,399–490assigns shell calls__global__and Edit/Write normalized file keys, then runs groups concurrently, sequentially only within a group. The source's “serialize with everything” comment overstates its implementation. Accepted runtime and synthesis correctly preserve possible Bash/Edit overlap. P described per-resource serialization and the global category but did not explicitly assert this new false guarantee; do not retroactively label P contradictory from that wording alone. - Implemented reflection review misclassified as doctrine-only. Verifier 000026 output
8b4789fbc6771dc231f20273427ef8b39a001dba55bfa526261f9e0c8070f8dagenerates two refused judgments,000027-apply-report-verification-1.jsonand-2.json, addressed to Epistemic and Reconciliation. Epistemic 000028 and reconciliation 000032 repair it; verifier 000035 accepts. Source selection atsrc/agent/subagents/index.ts:423–438, child persona delivery atsrc/agent/subagents/manager.ts:182–185, transcript launch atsrc/cli/helpers/reflection-launcher.ts:883, and review criteria atsrc/agent/subagents/builtin/reflection-v2.md:156–174support a conditional implemented checking request. They do not show correct performance, accepted truth or benefit. The accepted synthesis keeps that distinction rather than erasing the model-mediated route because fulfillment is unobserved. - Unsupported schedule identifier-selection value. Profile 000037 version
606a541ca7b29a6b6225c5693affa8dbaf6ba80de0389c905422277404f36c27is refused by verifier 000039, versionc2e4e4cf09edb499049a39eb93faaec4d9cc5a362d975bb72c708a772cbb5aea, applied in judgment000040-apply-profile-verification-1.json. Repair 000041 withdraws the schedule unit'sidentifierfinding while retaining coarse selection; re-verification 000043 accepts.src/cron/scheduler.ts:398–413rechecks an already selected task's status and wraps that task's text for enqueue. The local check does not select delivered content by identity match. The frozen profile type expressly requires such a match. Other checkpoint/cursor identifier findings are not invalidated by this local repair.
The second issue existed before the first verification but was detected on re-verification. This is useful recovery with repeated review cost, not proof of complete initial detection. Profile repair recurs in all three completed runs, but for different values: O's unsupported reflection promotion, P's skill admission classification, W's schedule selector. They must not be pooled as the same substantive defect.
Stable conclusions and coverage changes
| Accepted account or old finding | Focused check and disposition |
|---|---|
| Committed projection differs from requested live retrieval | src/backend/local/system-prompt-compilation.ts:81–199 enumerates HEAD and reads HEAD:<path>; src/tools/impl/memory.ts:306–405 reads checkout content and enforces path/root/skill restrictions. Accepted MEM-RTE-prompt-projection, MEM-RTE-deferred-read and synthesis preserve the distinction. |
| Root archive naming is not withdrawal | Compiler root selection excludes MEMORY.md from the ordinary loop, not ARCHIVE.md; the retained memory account quotes the root-path predicate. This remains a code-based deduction, not a witnessed archival event. |
| Mod feedback and conditional promotion | src/mods/learning-harness.ts:2310–2375 carries prior attempts/history and copies the selected candidate only when passing and a destination is requested. Accepted RT-RTE-mod-selection retains P's important executable-adaptation account while withholding live exercise and benefit. |
| Old hook ERROR/BLOCK distinction | Independently rechecked src/hooks/executor.ts:435–457 and src/tools/manager.ts:2192–2220: BLOCK stops, ERROR is tracked while processing continues, and the inspected manager returns early for blocked, not merely errored. W discusses vetoes and input rewriting but lacks O's equally explicit error-disposition finding. This is lost coverage, not a demonstrated contrary fail-closed claim. |
| Old Workflow and secret-redaction coverage | W restores an explicit EPI-RTE-secret-scrub record, which P lacked as an equivalent separate finding. W still does not restore O's separate Workflow VM/non-security-boundary account. These are retained-artifact comparisons; this audit did not reanalyse Workflow or all secret-substitution branches. |
| Specialized prompt mutation | Accepted report verification explicitly retains the REST compaction-prompt helper as a local unresolved inventory limit; synthesis carries it. This is not misreported as a universal Git admission route. Helper behavior was not independently rechecked here. |
The accepted report-verifier account includes omission questions, source-based dispositions and a stopping explanation. It does not assert a complete route inventory. Whole-system scope in W and P does not acquire Cloud/provider internals or erase uninspected shipped alternatives. No workflow-lens absence was counted as a historical defect. Scope gains, losses and successful repairs do not make the result uniformly better or worse.
A6 — Publication integrity and deterministic validation
Independent hashes establish:
- 12/12
R/effects/publish.json.newentries match retained bytes, including the manifest. - 11/11 manifest Markdown members match their declared hashes.
- 98/98 content-addressed version files hash to their filenames.
- There are 50 completed attempts: 18 model and 32 code. The 36 judgments comprise 32 accepted and 4 refused; refusals remain in history rather than disappearing after success.
Published manifest SHA-256 is f2c1cf0335a86bcdaf3ff6d6d6c8d428c5d64d4a734c9773b18b5374d38e3c2b; repaired profile is 33a5a9a165c76acd28176535464d6d3fc93eeac75e2db19afecf05d199724714. These are local integrity checks, not independent provenance authentication or semantic certification. Prior audits report 12/12 publication, 11/11 member and 81/81 version agreement for each completed comparator; those old hash totals were not recalculated in this audit.
From W's cwd, using its frozen .venv/bin/commonplace-validate:
VALIDATION SUCCESS
Target: kb/agentic-system-analyses/retained/letta-code
Scope: kb/agentic-system-analyses/retained/letta-code
Files analysed: 1
Text files: 0
Diagnostics: 0 failures across 0 subjects; 0 warnings across 0 subjects
The command treats the directory artifact as one analysed subject; it does not mean only one Markdown member exists. No current-origin completeness rules were applied to that set. No engine advance, rerun or integration was performed. The original coordinator read its publication prerequisite at T L34–35, before publication at L88–89.
Timing: wall time, worker spans and repairs
Worker spans use the minimum-to-maximum numeric embedded message timestamp, including model/tool waits but excluding full launch overhead. They are not provider compute or billed time.
| Attempt/role | Trace | Seconds |
|---|---|---|
| 000003 Boundary | L21 | 289.307 |
| 000005 Runtime | L25 | 408.482 |
| 000007 Memory | L29/R0 | 719.498 |
| 000008 Epistemic | L29/R1 | 523.040 |
| 000014 Reconciliation | L33 | 216.098 |
| 000017 Report verification | L39 | 446.944 |
| 000019 Runtime repair | L43 | 129.338 |
| 000023 Reconciliation repair | L47 | 195.451 |
| 000026 Report re-verification | L51 | 413.061 |
| 000028 Epistemic repair | L55 | 164.203 |
| 000032 Reconciliation repair | L59 | 158.261 |
| 000035 Report re-verification | L63 | 329.582 |
| 000037 Profile | L67 | 390.863 |
| 000039 Profile verification | L71 | 207.799 |
| 000041 Profile repair | L75 | 164.028 |
| 000043 Profile re-verification | L79 | 206.545 |
| 000045 Synthesis | L83 | 275.707 |
| 000047 Synthesis verification | L87 | 192.628 |
| Summed analytical spans | 5,430.835 (90.51 min) |
Preparation call to publication result is 5,379.906 seconds (89.67 min), T L14–89, 15:33:18.313Z–17:02:58.219Z. Only Memory and Epistemic overlap. Summed worker spans therefore differ from wall time, which also includes code/coordinator work. Later inspection, preparation of the incomplete 73f run, the diagnostic and this audit are excluded.
| Completed run | Wall minutes | Summed worker minutes | Analytical workers |
|---|---|---|---|
| O | 51.76 | 54.58 | 12 |
| P | 64.38 | 68.18 | 12 |
| W | 89.67 | 90.51 | 18 |
Against P, W adds 1,517.233 wall seconds and 1,340.072 summed worker seconds. The six additional report repair/reconciliation/re-verification workers total 1,389.896 seconds, with a launch-to-return window of 1,525.803 wall seconds at L42–63. These six explain most of the gross difference, but are not an isolated causal estimate: the other workers differ in duration, exploration and output.
Profile repair plus fresh verification costs 370.573 summed seconds and 415.592 wall seconds (L74–79), versus P's 361.794 and 384.711. Initial verification is useful detection work, not all repair waste. The runtime ambiguous-quote failure-to-pass window is 20.329 seconds; Epistemic's ambiguity-to-pass window is 29.186 seconds. These are local windows, not total quotation effort. Path failures and their recovery are also already inside worker totals; none should be added again to wall time.
Harness totals are 497 turns and 7.455566 reported cost units, versus P's 355 and 5.1998064 and O's 317 and 4.5389664. They exclude coordinator/diagnostic/auditor activity and are not invoices. Concurrent batch load, caches, latency, stochastic inspection choices, report breadth and different refusal chains remain uncontrolled. Fewer quotation-format failures did not translate into lower total run time, and the data cannot isolate what time would have been without the amendments.
Dispositions and limitations
- Retain the bounded accepted result. No new accepted-claim correction was established by these focused checks. Preserve earlier findings separately rather than assume replacement coverage. Further contradictory source evidence could change this disposition; additional coverage alone is not a historical publication defect.
- Retain the quotation and direct-validation amendments provisionally. Delivery and adoption are visible, the direct failure channel worked seven times, and range checking still caught ambiguity. This supports the mechanisms' intended operation, not a causal quality or efficiency claim. Do not remove uniqueness checks merely because format failures disappeared.
- Separate handout delivery from worker compliance. Absolute bindings were correct; five reconstructed artifact arguments and one misplaced identity write show residual execution errors. Do not blame the producer for those observed paths or dismiss the recovered write-scope violation because final bytes passed.
- Preserve semantic review and traces. Review corrected a misleading concurrency guarantee, an underclaimed implemented evaluator and an unsupported profile value. Their benefit is visible; repeated verification also has measured cost. One run cannot determine optimal reviewer count or justify another gate.
- Do not pool failures indiscriminately. Seven validator failures include five invocation failures; four refusals concern three substantive problems; risky pipelines are not demonstrated masked failures; a recovered mutation is still a mutation. These categories overlap in time and are not independent-run counts.
This audit is not exhaustive source review, reclassification of every profile unit, a source-to-record completeness certificate, a security audit or a target execution. It does not establish provider/Cloud guarantees, deployed sandbox behavior, summary fidelity, actual semantic acceptance or learning benefit. The source checks are deliberately focused on consequential accepted qualifications and prior findings. Prior audits inform comparison and make this audit non-blind. Missing observations are unknown, not passes. Broader graph discovery via cp-skill-connect is a possible separately authorized next step, not performed here.
Report verification and preservation
Only this report was authored. Frozen sources, method, publication and run state were left unchanged. No agents were launched, commits made, reruns performed or integration attempted. Read-only evidence calculations used the frozen environment where dependencies were needed; frozen validation ran from W. Origin validation of this report returned:
VALIDATION SUCCESS
Target: kb/work/analysis-audit-evidence/reports/letta-code-9d060e7dab7e-amendment-effects-audit.md
Scope: kb/work/analysis-audit-evidence/reports/letta-code-9d060e7dab7e-amendment-effects-audit.md
Files analysed: 1
Text files: 1
Diagnostics: 0 failures across 0 subjects; 0 warnings across 0 subjects