Dynamic Cheatsheet synthesis
Type: agentic-system-analyses/types/agentic-system-synthesis.md
Bounded synthesis
From a question to an evolving reference
Dynamic Cheatsheet implements inference-time memory by carrying reference text and earlier model responses into later problem solving. The strongest supported contribution is an automatic chain from problem-solving traces through a model-proposed replacement reference to subsequent model context. This chain can persist across benchmark questions and resumed runs without an explicit local weight update. It establishes a behavior-shaping update mechanism, not demonstrated improvement in future capacity. The README's reduced-mistake and performance claims remain reported claims, not reproduced results: RT-CLM-adaptive-memory.
The evidence is code-grounded inspection of SRC-1, the repository at commit 5cfe3c37e8e52b1d858d0f3df46e7f17c50991b9, with applicability cutoff 2026-10-12. The whole-system boundary includes the library, benchmark driver, shipped prompts, provider client and notebook consumers. No target execution or provider call was performed, and full result payloads were not assessed as run evidence. External provider implementations and the linked paper are outside the frozen evidence. The boundary and Source register distinguish implementation, doctrine/design and reported operation; neither implementation nor agreement among analysts establishes observed success.
Python owns scheduling, template substitution, extraction and state carry-forward. Generator and curator are successive invocations of the same configured model backend, not independent delegated workers. The remote model supplies answer and candidate reference content; the driver owns the next question, disk persistence and restart. The wrapper issues no training update on these routes, but does not pin or verify remote weight identity: RT-CMP-language-model.
The representative workflow is cumulative benchmark solving with the shipped curator prompt and optional local execution. Cumulative is the default approach, but the CLI alone does not load that prompt: without a curator path, it supplies (empty). With the prompt supplied, each round sends the question and current reference to the generator, extracts an answer, then sends the question, full generated response and previous reference to the curator. The curator does not receive the dataset target or subsequent score. The wrapper returns the answer, full response, steps and successor reference. This is a source-supported progression, not an observed delivered run: RT-RTE-cumulative-curation, workflow account.
Execution feedback and reference admission
The generator can request local Python execution. The wrapper selects the first eligible Python fence, adjusts its final line, runs a temporary script and feeds output or failure text back for model continuation. That text can later enter curation or retrieval through the full response. The three-second timeout kills the direct child; it is not process-tree containment, filesystem rollback or an inspected sandbox. Execution occurs before the continuation-depth test, so the warning at the final round is not a hard three-execution limit: RT-RTE-local-execution.
Provider-native execution is a separate branch with provider-owned execution semantics. Native mode in advanced_generate disables local execution, but native dispatch requires a truthy handle or sentinel. A local execution flag is not a universal veto on native tools or the host scoring route. Returned native text and a reused handle do not reveal which remote state later tools consume: RT-RTE-native-execution, MEM-OBJ-container-id.
Curation replaces whole reference text rather than editing validated items. Correctness assessment, source references, usage counts, preservation and refinement are prompt responsibilities. Software admits a substring after an opening cheatsheet tag; it need not find a closing tag or establish semantic validity. Missing opening tags preserve the old cumulative reference. Answer-extraction failure also does not block the curator handoff. Thus an incorrect or incomplete answer can supply a replacement that reaches the next question, although no particular such episode was observed: EPI-RTE-reference-admission, workflow annotation, EPI-CLM-verified-solutions.
The replacement supplies advisory reference context to the next generator and proposal input to the next curator. These are distinct consumer paths: experience can shape the update, and the successor can inform later solving. Neither delivery establishes reliance nor operational installation establishes epistemic acceptance. Old/new snapshots survive in steps, but the curator's full response is not separately saved. Reasons survive as live guidance only when included inside extracted reference text: RT-BAP-cheatsheet-context, RT-BAP-curator-guidance, MEM-OBJ-steps.
Alternatives, measurement and continuation
The alternatives change context construction and update order. Default generation uses an empty reference and no curator. Direct retrieval ranks prior questions by cosine similarity of precomputed question vectors and delivers corresponding full responses. Retrieval-synthesis instead curates selected pairs, prior reference and the incoming question before solving; extraction failure falls back to assembled examples, not necessarily the old reference. The hybrid combines cumulative text with retrieved examples, then curates after answering without the cumulative branch's refinement loop. Full-history mode supplies all available zipped earlier pairs without ranking or curation: RT-RTE-context-selection.
Similarity selects relevance, not correctness. Prior full responses can include code and execution feedback and remain available independently of omissions from the live reference. Top-k limits example count, not tokens; full-history assembly has no implemented trimming. Output token allowances and the curator's word guidance are not enforced input-context caps. Model synthesis could preserve, derive or conjecture propositions, but no inspected instance determines which occurred. Deterministic example assembly preserves selected strings without granting them truth: MEM-OBJ-final-output, MEM-OBJ-retrieved-selection, memory annotations, epistemic authority ledger.
After live-reference assignment and response-corpus append, the driver scores the answer against task predicates and rewrites JSONL. A bad score neither rejects the reference nor removes the response. Labels support measurement, not curator feedback. Online and notebook extraction differ; multiple-choice substring acceptance can count not red as correct for a target option red. Arithmetic scoring uses host eval, independently of generation-tool flags. Scores therefore license only their bounded predicates, not correctness of curation, explanation or execution safety: RT-RTE-answer-scoring, EPI-RTE-choice-scoring, EPI-CLM-zero-shot.
Library delivery is a returned dictionary. Driver delivery adds the target and writes accumulated checkpoints after scoring. Resume reloads the last final reference and all full responses, then skips the loaded row count. Partial argument comparisons and non-atomic rewriting support positional continuation, not exact replay or reliable crash recovery. Earlier snapshots afford external manual reconstruction, not automatic semantic rollback: RT-OBJ-output-dict, RT-RTE-checkpoint-continuation.
The exported client's separate chat route automatically retains and resupplies raw turns, with reset and replacement APIs; the benchmark message entry point bypasses it. Optional search imports external snippets or provider-generated searched text, not certified knowledge. Structured native citation metadata can be lost at text projection, while citations in text may remain. These surfaces do not belong to the selected cumulative workflow: MEM-RTE-chat-history, MEM-OBJ-messages, EPI-RTE-search-acquisition.
What this progression establishes
For sequential problem solving, the architecture supports accumulating answer-derived reference and selecting earlier experience without a correctness oracle in the update loop. Automatic trace-fed reference production with later consumption is wired; raw logging alone is not that transformation. Whether criticism of a consumed theory improved future capacity remains uninspected, as does causal attribution of any reported benefit to curation, retrieval or tools.
The theory-builder conditions remain separate. Condition 1, localized theory content, is claimed through requested reusable strategies; no particular generated theory was assessed. Condition 2, decisions depending on that content, is claimed through intended use, not established by delivery. Condition 3, content-directed criticism with blame and consequent revision, is uninspected. Condition 4, retained criticism shaping another round, is uninspected despite update iteration being wired. Whole-text replacement provides minimum replaceability, but requested finer item structure is not enforced: RT-RTE-cumulative-curation, RT-RTE-context-selection.
Reflection is separately unresolved: prior operation can update a reference and that reference has a wired later channel, but representation-mediated behavioral dependence is unobserved. Computational scheduling, proposal and admission are wired; autonomous performance of every theory-builder operation is not established. Reflective theory-building is also uninspected because criticism and revision of the applied method texts were not established. None of these conclusions implies another.
At the generator–curator–driver boundary, over later questions or resumed runs, task correctness is an independently specified improvement objective. The reference pathway supports an improvement-directed standing mechanism: successor text changes live reference organization and enters an advisory model channel. Exercised self-improvement remains unestablished because objective-bearing evidence causally shaping a particular update and later behavior causally depending on it were not demonstrated. Successful improvement requires additional outcome evidence: RT-CLM-adaptive-memory.
The assessment would change with linked traces retaining candidate references, criticisms, evidence-consuming dispositions and later consumption, plus controlled comparisons showing future-capacity change. Component attribution would require independent variation of the relevant component. A score-consuming semantic admission or rollback mechanism would change the current admission account; enforced context limits and transactional, configuration-complete restart would change budget and recovery guarantees. None is inferred from current prompt requests or checkpoint presence.
Source-read log: none; this synthesis uses only the supplied members.
Limitations
The accepted reconciliation identifies no unresolved cross-report conflict. Report verification declares no additional limits; profile verification declares the two inventory omissions below. Other rows retain consequential evidence boundaries without treating uninspected material as absent.
| Limitation | Affected SRC IDs and record citations | Inspected boundary | Conclusion prevented | Evidence that would resolve it |
|---|---|---|---|---|
| No inspected execution or controlled benefit comparison | SRC-1; RT-CLM-adaptive-memory, EPI-CLM-zero-shot | Shipped reference loop and reported performance; full results and saved notebook outputs uninspected | Observed reliance, improved future capacity, exercised or successful self-improvement, and component attribution | Linked operation/update/later-consumption traces and appropriately controlled outcome comparisons |
| Requested correctness and semantic maintenance are not established transformations | SRC-1; EPI-CLM-verified-solutions, EPI-RTE-reference-admission, RT-RTE-context-selection | Prompts, extractor and update call sites; no generated instance assessed | Universally verified retained items, checked derivation, accepted ampliative claims, actual consolidation/deduplication/synthesis | Candidate-level inputs, full curator reasoning, checking evidence and consequential acceptance records |
| Theory criticism, iteration and reflection remain unobserved | SRC-1; RT-RTE-cumulative-curation, RT-RTE-context-selection, MEM-OBJ-steps | Requested strategies and wired update/delivery; curator rationale outside reference not checkpointed | Met theory-builder conditions 1–4, criticism-improved capacity, reflective or autonomous theory-building | Localized theories, demonstrated dependence, claim-directed criticism/blame/revision and retained criticism affecting later work; method-text traces for reflective theory-building |
| Caller provenance and human content control unresolved | SRC-1; MEM-OBJ-final-cheatsheet, MEM-OBJ-messages, RT-RTE-checkpoint-continuation | Seed files, replacement/history APIs and checkpoint reload | Complete lineage or manual-admission classification | Provenance and explicit content-supply/approval records |
| Native payload and external service contracts opaque | SRC-1; MEM-OBJ-container-id, RT-RTE-native-execution, RT-CMP-language-model, EPI-RTE-search-acquisition | Local dispatch, handle reuse and returned text; external implementations unfrozen | Complete remote memory properties, isolation, immutable model identity, citation lineage and current compatibility | Frozen provider contracts, full tool/evidence envelopes and payload write/read traces |
| Profile omits local native-handle routing authority | SRC-1; MEM-OBJ-container-id, RT-RTE-native-execution, profile-verification limits | Handle enables/selects native dispatch, separately from opaque payload | Complete per-consumer authority inventory; existing routing union and wired basis remain valid | Record-grounded profile unit for local handle routing |
| Profile omits separate set_history replacement curation | SRC-1; MEM-RTE-chat-history, MEM-OBJ-messages, profile-verification limits | Client reset and replacement APIs; human versus computational caller unresolved | Complete client-history maintenance inventory; existing evolve union and wired basis remain valid | Record-grounded profile unit for replacement independent of agency uncertainty |
| Artifact provenance and evaluator coverage incomplete | SRC-1; RT-CMP-input-embeddings, EPI-OBJ-target, report verification, boundary coverage | Active scorers inspected; full vectors, Arrow payloads and auxiliary sonnet helper internals unassessed | Embedding quality, target truth, leakage, full reproducibility or a system-complete evaluator negative | Payload/provenance inspection, frozen external dataset identity and relevant auxiliary evaluator assessment |
| Delivery and controls have bounded guarantees | SRC-1; RT-RTE-checkpoint-continuation, RT-RTE-local-execution, RT-RTE-answer-scoring | Non-atomic positional restart, direct-child timeout and separate host eval | Exact replay, guaranteed crash recovery, universal tool veto or host isolation | Transactional persistence, complete restart identity checks and separately enforced/tested execution boundaries |