Dynamic Cheatsheet synthesis
Type: agentic-system-analyses/types/agentic-system-synthesis.md
Bounded synthesis
Dynamic Cheatsheet implements a sequential inference-memory loop: solve a question, propose replacement reference text, and supply retained material to later model calls. Python owns the progression; the model proposes answers, code and reference content. It is not an enclosing autonomous runtime. This account covers the shipped wrapper, benchmark driver, notebook callers, prompts, persistence, retrieval and evaluation at Git commit 5cfe3c37e8e52b1d858d0f3df46e7f17c50991b9, with an applicability cutoff of 2026-10-20. Its evidence is SRC-1 implementation and prompt doctrine/design. README performance statements are attributed reports, not inspected experiments. No target execution or historical result-payload assessment supports this account (boundary, Source register).
From a question to retained guidance
The representative workflow is a cumulative benchmark invocation with both shipped generator and cumulative curator templates explicitly supplied. That condition matters: the CLI's default curator template is literally (empty), so a bare invocation does not request the shipped correctness-review and maintenance instructions. The operator chooses the task, model and settings; the driver loads questions, initializes or restores a sheet, and calls the wrapper. The generator receives the question and sheet in a fresh user-message context. After generation, the same configured model/client receives a separate curator context containing the question, full generator output and previous sheet. These are separate roles and contexts, not independent model identities or independent ground truth (RT-RTE-benchmark-cycle, RT-RTE-cumulative-update, workflow).
The strongest supported adaptive contribution is an automatic trace-fed update pathway: solution experience feeds a curator proposal; extracted replacement text becomes retained reference material for later rounds or questions. The sheet is a process string, optionally initialized from a file and persisted in caller JSONL. Raw final solution output is retained separately for retrieval or full-history reuse. Optional extracted previous-round answers provide another, shorter continuation artifact within one cumulative invocation. These are different retained parts, not interchangeable names for a log (MEM-OBJ-cheatsheet, MEM-OBJ-final-output, MEM-OBJ-previous-answers). This establishes wired trace-fed adaptation, not improved future capacity.
The curator is asked to assess correctness, preserve useful strategies, remove redundancy and retain examples and pitfalls. Software admits its proposal by substring extraction. An opening <cheatsheet> tag suffices; a missing closing tag or an empty body need not prevent replacement. Without an opening tag, cumulative/hybrid mode keeps the previous sheet. No independently parsed correctness verdict or reference-answer comparison gates admission. Whole replacement can remove errors or useful content; older versions can survive in logs, but no automatic old-entry restoration or semantic rollback follows (EPI-RTE-curator-review, EPI-RTE-sheet-admission). Requested consolidation and deduplication therefore remain distinct from the implemented replacement operation.
What reaches the next invocation
The caller creates the cross-query horizon. The driver carries the returned sheet and accumulates final outputs; continuation reloads saved rows and the last sheet. The generator receives retained prose, examples and displayed code as interpreted reference context, not as directly executable sheet entries or enforced rules. Python enforces which text is supplied; it does not enforce the truth of that text or the model's reliance on it (RT-BAP-reference-context, RT-RTE-benchmark-cycle).
The six approaches make materially different context choices. Default supplies an empty reference; full-history mode copies all prior question/output pairs. Retrieval ranks previous question vectors by cosine similarity and supplies top-k pairs, with the most similar displayed last. Retrieval-synthesis first asks a curator to construct a custom sheet from selected pairs, the prior sheet and the next question, then generates an answer; its missing-tag fallback is selected raw pairs, not the previous synthesized sheet. Cumulative mode updates after answering and can repeat rounds. Hybrid supplies cumulative guidance plus retrieved examples, then performs one post-answer update rather than the cumulative repeated-round loop (RT-RTE-context-selection).
These fresh contexts keep curator analysis and unselected old solutions outside the next generator window. Only selected or derived text crosses back. Omitted raw pairs remain available for later Python selection; the generator has no pointer-following reload tool for old sheet entries. Pair-count limits, output-token allowances and prompt word advice do not establish a total-input or context-fit guarantee. Dense vectors are selector inputs, not correctness evidence, and their generation provenance and actual quality remain uninspected (MEM-OBJ-input-embeddings, memory read-back).
Checks, effects and delivery
The generator can request local Python execution and receive rendered results or errors in a continuation. This can supply evidence about the executed program, but not automatically about its premises, problem encoding or generalization. The subprocess uses ambient privileges; its timeout is not filesystem, network or credential isolation. Execution precedes the recursion-depth check, so the default depth/limit combination can permit four executions rather than three. Provider-native execution is a separate route, selected with a native flag and truthy session identifier. The local-execution switch guards only the local subprocess route. OpenAI/Claude request passthrough can also carry provider tools; actual acceptance and execution depend on external providers. No whole-system no-execution guarantee follows from disabling the local flag (RT-RTE-local-execution, RT-RTE-provider-execution, runtime guarantees).
Benchmark targets reach task scorers after the sheet has been returned and adopted, not the generator or curator through this call path. Ordinary false scores change counters, not memory admission or later selection. The inspected loop thus has answer evaluation but no score-driven memory veto, claim withdrawal or evidence-consuming acceptance followed by integration (EPI-ABS-score-feedback). Scoring licenses only satisfaction of its predicate: multiple-choice substring acceptance can accept a negated target-option statement, and runtime versus notebook extraction can choose different answer tags. A score does not validate the explanation or reusable strategy. GameOf24 scoring also evaluates generated text outside the generation execution flag (EPI-OBJ-score-result, RT-RTE-score-answer).
Delivery separates library return, in-memory handoff and durable checkpoint. The delivered answer remains the generator's answer, not a curator-corrected answer. The driver scores and then rewrites JSONL. Those writes are non-atomic; scoring failure or interrupted rewriting can prevent durable delivery of the newest update. Resume restores text and skips the saved row count, but does not verify all settings, dataset identity or prompt content, nor restore provider container state. Recovery is best effort, not exact replay or exactly-once delivery (workflow delivery, RT-RTE-benchmark-cycle).
Assessment at this boundary
For successive related questions, the supported mechanism is accumulated experience supplied as future reference, with model-mediated selection and replacement. For a use requiring verified reusable claims, the inspected admission route does not provide that warrant: delimiter admission, relevance ranking and post-hoc answer scoring have different targets and consequences. Actual model-output semantics remain indeterminate; the generalization request alone establishes neither faithful compression nor a newly produced ampliative claim (EPI-RTE-sheet-admission, epistemic conclusion).
For sheet entries that express formulated rules, theory-builder condition 1 is afforded by localized inspectable text and whole-unit replacement. Condition 2 is afforded by interpretation-dependent use, not observed dependence on a particular rule. Condition 3 remains operationally uninspected: no inspected criticism identifies a challenged proposition, result and blame placement leading to revision. Condition 4 has an afforded handoff mechanism, but kept criticism shaping a next round remains uninspected. These separate statuses do not establish an exercised theory builder. Criticism of a consumed theory improving future capacity is likewise unestablished. Reflection's two-way self-representation and autonomous performance of every theory-building operation are independently uninspected; automated scheduling alone proves neither (EPI-RTE-curator-review).
The sheet can enter the system's own behavior-determining guidance, so its output label does not exclude self-change. The implemented pathway supports a dispositional, improvement-directed self-change account. Exercised self-improvement remains unestablished because subsequent behavior causally depending on a particular evidence-shaped update has not been inspected; successful improvement needs separate outcome evidence. No inspected wrapper route changes model weights, but provider internals and exact mutable endpoint versions remain unresolved (epistemic conclusion, RT-CMP-language-model). Linked update/criticism/consumer traces would change the exercised-property assessment. A comparison varying memory separately from execution and other settings would be needed for memory-specific benefit attribution; an evidence-consuming admission or rollback consumer would change the warrant assessment. README gains alone establish none of those finer conclusions (EPI-CLM-performance-gains).
Limitations
| Limitation | Affected SRC IDs and record citations | Inspected boundary | Conclusion prevented | Evidence that would resolve it |
|---|---|---|---|---|
| No assessed execution or comparison payload | SRC-1; RT-CLM-adaptive-memory, EPI-CLM-performance-gains | Shipped code and prompts; historical results and saved notebook outputs available but unassessed | Observed activation, measured future-capacity improvement, gain magnitude, transfer and memory-specific causal effect | Linked run artifacts and a scoped comparison independently varying the claimed component |
| Curator semantics and criticism chain unobserved | SRC-1; EPI-RTE-curator-review, EPI-RTE-sheet-admission | Prompt construction, extraction and handoff | Successful review, faithful compression, produced novel claims, exercised theory criticism/iteration or evidence-responsive behavioral dependence | Candidate text, evidence, explicit judgments/revisions and later consumer traces |
| Initial content and vector provenance unresolved | SRC-1; MEM-OBJ-cheatsheet, MEM-OBJ-input-embeddings | Loader, exact-string alignment and selector logic; dataset/vector payloads uninspected | Initial correctness or human approval, embedding-generation lineage, retrieval quality and dataset independence | Provenance and admission records, payload inspection and alignment/quality evidence |
| Hosted payload opaque and services external | SRC-1; MEM-OBJ-container-id, RT-RTE-provider-execution, RT-CMP-language-model | Local identifiers, request assembly and returned text | Complete hosted-memory characterization, actual reuse, isolation, live compatibility or exact remote parameter state | Provider contracts, pinned versions and state/tool traces showing admission, persistence and consumers |
| Answer checks have bounded license | SRC-1; EPI-OBJ-score-result, EPI-ABS-score-feedback | Dispatched scorers, driver and evaluator cell-source consumers | Semantic correctness from every pass, explanation/strategy validity or score-grounded sheet acceptance | Validated extraction and scorer semantics; for memory warrant, an explicit evidence-consuming disposition route |
| Checkpoints do not establish exact recovery | SRC-1; RT-RTE-benchmark-cycle | Whole-file writes and partial continuation checks | Transactional durability, exact replay and restored provider state | Atomic checkpointing, complete continuation identity/state contract and recovery evidence |
| Standalone capabilities are not deployed memory routes | SRC-1; MEM-RTE-history-controls, EPI-ABS-score-feedback | History reset/set affordance and bounded shipped-loop feedback search; auxiliary checking/acquisition not fully assessed | Deployed retained-history reload or repository-wide absence of checking, acquisition or criticism | Concrete retained-artifact callers and consequential consumers for those alternative capabilities |
The report and profile verifications declare no additional limits or blockers (report verification, profile verification). No new source reads were made for this synthesis.