Dynamic Cheatsheet agentic-system analysis
Type: agentic-systems/types/agentic-system-analysis-overview.md
Boundary and evidence
The target is Dynamic Cheatsheet as shipped: a prompt-driven, inference-time memory method and its benchmark driver. It generates an answer from a current question plus retained state, then (for cumulative variants) asks a model to update that state. Retrieval variants build question-specific context from prior inputs and outputs, with optional curator synthesis. The shipped benchmark runner is included because it supplies, carries forward, persists, reloads, and evaluates the stateful loop. The provider client is included only as the model-call and optional code-execution adapter wired into that loop; provider service internals and the underlying language models are external dependencies. The analysis covers the repository at commit 5cfe3c37e8e52b1d858d0f3df46e7f17c50991b9; cutoff is 2026-10-03. Evidence is code-grounded, with README and prompts establishing declared design and usage. Checked-in result files are artifacts, not independently verified execution traces in this boundary assessment.
| Top-level path | Coverage | Role and exclusion consequence |
|---|---|---|
.gitignore |
Included | Repository packaging/configuration; no direct memory-loop role. |
README.md |
Included | Declares intended method, variants, prompts, and benchmark use. |
dynamic_cheatsheet/ |
Included | Core method implementation, model/provider adapter wiring, answer and cheatsheet extraction, code execution, and evaluation helpers. |
text_generation/ |
Included | Unified provider client called by the core; provider SDKs and remote model services remain external. Excluding this adapter would prevent tracing model-call dispatch and optional hosted code-interpreter wiring. |
prompts/ |
Included | Shipped generator and curator instructions define answer production and memory maintenance. |
run_benchmark.py |
Included | Shipped caller loads prompts/data, initializes or resumes cheatsheet, supplies retrieval history, calls the method, evaluates answers, and writes JSONL after each item. Excluding it would prevent conclusions about the shipped persistence/reload and evaluation loop. |
data/ |
Included | Locally retained benchmark datasets and metadata consumed by the runner; dataset provenance and any upstream-host behavior are outside this code-grounded boundary. |
embeddings/ |
Included | Precomputed retrieval vectors and their associated inputs consumed by retrieval variants; embedding generation procedure is not established by these files. |
results/ |
Included as artifacts | Checked-in model/task result files illustrate stored output formats, but do not establish reproducible operation or causal effects. |
EvaluatingResults.ipynb |
Included | Shipped evaluation/results consumer; notebook outputs and any external service provenance require separate scrutiny before treating them as observed operation. |
ExampleUsage.ipynb |
Included | Shipped usage path invoking the method; helps establish caller wiring beyond the CLI. |
figures/ |
Included as documentation artifacts | Illustrations and reported aggregate performance graphics; they do not alone establish experimental conditions or causal attribution. |
LICENSE |
Included | Distribution context; no memory-loop behavior. |
The state lifecycle visible in code is caller-managed. The benchmark runner initializes a cheatsheet from a file or (empty), replaces it with each call's final_cheatsheet, writes output records, and can resume by loading prior JSONL and recovering the last cheatsheet. The core method itself accepts state as an argument and returns updated state; it does not own a separate durable store. Retrieval consumers use supplied corpus, embeddings, and prior generator outputs. The notebook consumers and result artifacts are in the inspected repository boundary, while their outputs are not treated here as independently verified runs. External dependencies include selected provider APIs/models, provider SDKs, optional hosted code interpreter, Python packages, and upstream benchmark data where fetched. The runner also offers local subprocess execution of generated code; that is a shipped capability, not an external memory service.
Source register
| source ID | kind | identity/location | revision or capture | evidence layer | inspected scope | citation anchors | access gaps and conclusion prevented |
|---|---|---|---|---|---|---|---|
| SRC-1 | git | https://github.com/suzgunmirac/dynamic-cheatsheet |
5cfe3c37e8e52b1d858d0f3df46e7f17c50991b9 |
implementation | Core library, runner, provider adapter, and utility modules inspected for entry points, state flow, model calls, persistence/reload, retrieval, and evaluation. Additional files at this same commit remain in scope for later jobs. | run_benchmark.py; dynamic_cheatsheet/language_model.py; dynamic_cheatsheet/utils/extractor.py; dynamic_cheatsheet/utils/evaluation.py; dynamic_cheatsheet/utils/execute_code.py; text_generation/simple_unified_client.py |
No repository access gap for boundary classification. This source does not establish provider-side behavior or causal effects. |
| SRC-2 | git | https://github.com/suzgunmirac/dynamic-cheatsheet |
5cfe3c37e8e52b1d858d0f3df46e7f17c50991b9 |
doctrine/design | README, shipped prompts, and notebooks inspected for declared method, usage, curation policy, intended evaluation, and caller wiring. | README.md; prompts/generator_prompt.txt; prompts/curator_prompt_for_dc_cumulative.txt; prompts/curator_prompt_for_dc_retrieval_synthesis.txt; EvaluatingResults.ipynb; ExampleUsage.ipynb |
Prompt claims of verification and README performance claims do not establish executed behavior, experimental provenance, or causal effects. |
| SRC-3 | git | https://github.com/suzgunmirac/dynamic-cheatsheet |
5cfe3c37e8e52b1d858d0f3df46e7f17c50991b9 |
reported operation | Checked-in results and figures inspected only as reported artifacts; benchmark datasets and embeddings inspected as inputs. | results/; figures/; data/; embeddings/ |
The artifacts alone do not establish their complete provenance, execution conditions, or causal attribution; those conclusions are prevented without supporting run records and experimental design evidence. |
Amended or superseded records: none; reconciliation.
Bounded synthesis
The frozen repository implements a caller-managed, inference-time memory loop: the benchmark runner supplies a question and retained context to a selected model, adopts returned context where the approach produces it, and saves result rows for continuation (runtime.md, RT-RTE-1; memory.md, MEM-RTE-1). The client does not own a durable store. JSONL rows retain answers and returned cheatsheets, and continuation reloads the latest cheatsheet and prior outputs (runtime.md, RT-RTE-1; memory.md, MEM-OBJ-1, MEM-OBJ-2).
The approach determines what context reaches a later answer. Cumulative variants pass the standing cheatsheet and use a curator proposal to replace it; full-history appends prior question/output pairs; retrieval ranks prior pairs by cosine similarity over supplied embeddings; retrieval synthesis adds a model-written context view (runtime.md, RT-RTE-1, RT-RTE-2; memory.md, MEM-RTE-1, MEM-RTE-2, MEM-RTE-3, MEM-RTE-4). The source establishes prompt delivery and persistence, not that a model used the material or that selected examples were relevant. Embedding provenance is unavailable (memory.md, MEM-OBJ-3, MEM-RTE-3).
For cumulative and synthesis updates, tag extraction accepts proposed text or uses a fallback. The curator prompts request correctness and reusable strategies, but no content-level verifier or human approval gates adoption (runtime.md, RT-RTE-1; memory.md, MEM-RTE-1, MEM-RTE-4; epistemic.md, RT-CLM-1). The benchmark scores answers against task targets after adopting the updated context; that score is not sent back to curate or revise memory (epistemic.md, EPI-RTE-1; reconciliation.md). Thus the implementation wires trace-fed text updates, but the evidence does not show correct retained claims or improved future capacity (memory.md, MEM-RTE-1, MEM-RTE-4).
At the declared whole-system boundary, self-improvement over an observed horizon is not established. The routes wire interaction-fed text updates and later delivery, but the evidence does not show that evidence bearing on an improvement objective shaped an update and that a subsequent operation depended on that change (runtime.md, RT-RTE-1, RT-RTE-2; memory.md, MEM-RTE-1, MEM-RTE-4; epistemic.md, RT-RTE-1, RT-RTE-2).
The inspected code provides two distinct code-execution paths: model-marked Python can run as a local subprocess, while an option delegates execution to a provider tool. The local path has no sandbox in the inspected function; provider isolation is outside the evidence (runtime.md, RT-RTE-3, RT-RTE-4; epistemic.md, RT-RTE-3). The available records establish implementation wiring only. They do not establish an executed deployment, model outputs, activation, causal performance effects, or realized memory benefit (runtime.md, RT-RTE-1, RT-RTE-2, RT-RTE-3, RT-RTE-4; epistemic.md, EPI-RTE-1).
The supplied evidence does not establish a formulated theory-building route or content-directed criticism and iteration meeting theory-builder conditions 1–4. Reflection and autonomy are also unsupported at this boundary. These are limits of the inspected records, not claims about inaccessible provider processing (runtime.md, RT-RTE-1; epistemic.md, RT-RTE-1, EPI-RTE-1).
Limitations
| limitation | affected source, record, or route IDs | inspected boundary | conclusion prevented | evidence that would resolve it |
|---|---|---|---|---|
Unresolved conflict: EPI-RTE-1 and dynamic_cheatsheet/utils/sonnet_eval.py leave unclear whether the standalone evaluator helper participates in a shipped epistemic route. |
EPI-RTE-1, SRC-1 | Whole-system repository at commit 5cfe3c37e8e52b1d858d0f3df46e7f17c50991b9; helper exists but no caller is evidenced, and the records do not delimit evaluator coverage against it. |
Completeness of evaluator coverage, including whether the helper supplies a material check. | Evidence establishing whether and how the helper is called in a shipped route, plus an updated coverage account. |
| No execution trace or independently supported run artifact is supplied; checked-in results and figures do not establish their execution conditions or provenance. | SRC-1, SRC-2, SRC-3; RT-RTE-1, RT-RTE-2, RT-RTE-3, RT-RTE-4; EPI-RTE-1 | Frozen repository and supplied code-grounded, design, and reported-artifact evidence. | Actual operation, model outputs, memory activation, correctness, and causal or realized benefit. | Reproducible run records with inputs, provider/model identity, outputs, execution conditions, and suitable comparisons for causal claims. |
| Provider internals, exact model revisions, provider-side retention and execution controls, and deployment permissions are outside the inspected boundary. | SRC-1; RT-CMP-1; RT-RTE-4; RT-RTE-3 | Repository adapter and runner code; external providers and deployed environment are inaccessible. | Exact model behavior, service retention, provider-tool isolation, and deployed host-control guarantees. | Versioned provider and deployment specifications plus inspectable execution and retention evidence. |
| The origin of initialized cheatsheets, checked-in datasets and retrieval embeddings is not established. | SRC-1, SRC-3; MEM-OBJ-1, MEM-OBJ-2, MEM-OBJ-3; MEM-RTE-3 | Supplied initialization path, repository data and vectors, and their inspected consumers. | Complete lineage and the correctness or relevance of supplied targets, initial content, and embedding rankings. | Provenance records for source data, initialization content, and embedding generation, with validation of their intended use. |
| The inspected routes wire interaction-fed text updates and later delivery, but the evidence does not establish evidence-responsive self-change over an observed horizon. | RT-RTE-1, RT-RTE-2; MEM-RTE-1, MEM-RTE-4; EPI-RTE-1 | Whole-system repository and supplied code-grounded/design records; no inspectable run trace. | Whether Dynamic Cheatsheet self-improves at this boundary and horizon. | An inspectable trace linking evidence bearing on a stated improvement objective to an operative text change and a subsequent operation that depends on it. |
Verification and blockers
Record verification
The frozen Git checkout is at the boundary's commit 5cfe3c37e8e52b1d858d0f3df46e7f17c50991b9. Its top-level tree agrees with the whole-system boundary table: the shipped runner, core package, provider adapter, prompts, data, embeddings, result artifacts, notebooks, figures, README, license and packaging file are accounted for. The boundary separates implementation, design and reported-artifact evidence and does not treat checked-in results as observed runs. SRC-1, SRC-2 and SRC-3 resolve to the one registered repository and commit; cited repository paths exist in that tree. No source identity or boundary defect was found.
The records resolve to the declared members: runtime records RT-CMP-1, RT-OBJ-1, RT-OBJ-2, RT-RTE-1, RT-RTE-2, RT-RTE-3, RT-RTE-4, RT-CLM-1, RT-BAP-1 and RT-BAP-2; memory records MEM-OBJ-1, MEM-OBJ-2, MEM-OBJ-3, MEM-RTE-1, MEM-RTE-2, MEM-RTE-3 and MEM-RTE-4; epistemic records EPI-OBJ-1, EPI-OBJ-2 and EPI-RTE-1. The records' source references resolve to the supplied register. Their evidence statuses distinguish wired implementation from uninspected operation and do not promote prompt claims, checked-in artifacts or scoring results to observed operation or causal evidence.
The Part of: relations are syntactically valid and resolve: MEM-OBJ-2 and MEM-OBJ-3 are declared parts of RT-OBJ-2; MEM-RTE-1 is a declared part of RT-RTE-1; MEM-RTE-2, MEM-RTE-3 and MEM-RTE-4 are declared parts of RT-RTE-2. The distinctions in representational form, consumer and operation support retaining the containers. No supersession or split is needed. EPI-OBJ-1, EPI-OBJ-2 and EPI-RTE-1 have distinct answer, target and scoring referents. RT-RTE-3 with RT-BAP-1 and RT-RTE-4 with RT-BAP-2 distinguish local and provider-side execution.
I checked the whole memory-comparison profile against all member records, not just its cited memory subset. The scope covers accumulated or changed cheatsheet and example material, its vector access structure, writes, persistence/reload, and later prompt consumers; static prompts and task targets/scores are excluded from memory with an adequate rationale because the score runs after the update and does not feed admission or selection. The profile's files and in-memory values are supported by MEM-OBJ-1, MEM-OBJ-2, MEM-OBJ-3, MEM-RTE-1, MEM-RTE-2, MEM-RTE-3 and MEM-RTE-4. Natural-language and parametric forms, partial imported/trace-extracted lineage, knowledge/ranking/routing/learning authority, automatic and manual write agency, consolidate/dedup/evolve/synthesize, pull and push, coarse and inferred-embedding push signals, trace-learning yes, and session-logs/tool-traces are supported at the stated wired or afforded strength. The profile retains the unknown initialization and embedding provenance, limits trace-learning to the curation routes, and conditions tool-trace support on execution feedback entering retained outputs.
The checked routes and dispositions are RT-RTE-1 and MEM-RTE-1 (cumulative update and continuation); RT-RTE-2, MEM-RTE-2, MEM-RTE-3 and MEM-RTE-4 (append-all, embedding selection and synthesis); RT-RTE-3 with RT-BAP-1 (local code execution); RT-RTE-4 with RT-BAP-2 (provider code execution); and EPI-RTE-1 (answer scoring). Their separate effects and consumers are represented without collapsing scoring into memory admission, selection into curation, or either execution boundary into the other. The profile does not omit an additional supported memory value due to EPI-RTE-1: its post-answer boolean has no consumer in the write or selection routes. Reconciliation's identity and amendment dispositions are consistent with these records; it makes no unsupported amendment or supersession.
The epistemic coverage table omits a material path from its account: dynamic_cheatsheet/utils/sonnet_eval.py exists in the frozen tree and implements a sonnet constraint checker. I inspected the file and searched the frozen tree for references. It has no shipped caller outside its own corpus-check/self-test code; evaluation.py has only a commented import, and the shipped benchmark and results notebook call the separate functions in dynamic_cheatsheet/utils/evaluation.py. This supports neither an assertion that the sonnet checker participates in a shipped benchmark route nor a system-wide conclusion that it cannot be invoked as a checker. Reconciliation records this bounded issue as Unresolved conflict: affecting EPI-RTE-1, with evidence path SRC-1 dynamic_cheatsheet/utils/sonnet_eval.py and the prevented conclusion: completeness of evaluator coverage and whether this helper supplies a material check. That marker names the affected record, evidence and prevented conclusion, so this fixed-member coverage limit is a checked conflict for synthesis, not a repairable blocker. The whole-system boundary remains supported; the uncertainty limits evaluator completeness only.
The reconciliation also records the independent convergence on scoring timing for RT-RTE-1 and EPI-RTE-1, scoped to inspected implementation and not to observed operation or memory correctness. No unresolved conflict or source gap beyond the evaluator-coverage conflict above was found. The supplied record-set check is none.
Synthesis verification
The Description accurately summarizes the established mechanism and evidence limit: retained text or selected examples are delivered through wired routes, while provider behavior and memory benefit are not established.
The Bounded synthesis is supported by the cited records. The runner carries and persists the cheatsheet and prior interaction rows (RT-RTE-1, MEM-RTE-1, MEM-OBJ-1, MEM-OBJ-2). Cumulative, append-all, top-k retrieval, and retrieval-synthesis context paths are distinguished correctly (RT-RTE-1, RT-RTE-2, MEM-RTE-1, MEM-RTE-2, MEM-RTE-3, MEM-RTE-4, MEM-OBJ-3). Supplied embedding provenance is unavailable (MEM-OBJ-3, MEM-RTE-3). The synthesis correctly distinguishes curator instructions from a content-level admission check, and answer scoring from memory admission (RT-CLM-1, RT-RTE-1, EPI-RTE-1). Its self-improvement, theory-builder, reflection, and autonomy statements are bounded as unestablished rather than absent (RT-RTE-1, RT-RTE-2, EPI-RTE-1). The local subprocess and provider execution paths are kept distinct, with execution and isolation limits stated (RT-RTE-3, RT-RTE-4, RT-CMP-1).
The Limitations section states the inspected boundary, prevented conclusions, and resolving evidence for the principal operation, provider, provenance, and self-improvement limits. It carries the reconciliation's unresolved conflict, naming EPI-RTE-1, SRC-1, and dynamic_cheatsheet/utils/sonnet_eval.py, and limits the conclusion about evaluator coverage. The synthesis reads without relying on member prose; its links identify the supporting reports and records. All record references are explicit full IDs.
Deterministic validation
commonplace-validate kb/agentic-systems/reports/state/AAS-2026-10-03-dynamic-cheatsheet-02/output --full runs after the manifest is written, over every member, the manifest and the set's cross-member checks; code publishes only when it passes, and publication runs it again.
Blockers
none