Case packet

Neutral case identifier: case-33b17d160ff4f4

The possible directed relationship from Artifact A to Artifact B is under review.

Artifact A

Ad hoc prompts extend the system without schema changes

Any system with an LLM agent layer has two strata: a deterministic base (files, schemas, scripts, APIs) and a prompt layer on top. The prompt layer is where new requirements get absorbed without changing the base.

The mechanism: when a requirement doesn't fit existing code or configuration, you write a natural language prompt — a markdown file, a comment, a task description — that tells the agent what to do. No code change, no schema migration, no deployment. The system's vocabulary grows at the speed of writing, not the speed of coding.

This applies everywhere agents operate. A CI pipeline gains a new check by adding a prompt to the agent's instructions, not by writing a new GitHub Action. A codebase gains a new review criterion by describing it in CLAUDE.md, not by writing a linter rule. A deployment process gains a new safety check by telling the agent "also verify X before pushing," not by adding a pre-deploy hook.

We first noticed this in the KB, where it shows up cleanly.

The KB example: collections

We needed "read multiple documents through one goal." Three formal options presented themselves:

  1. A new collection type with a schema for listing document paths
  2. A file-listing format (one path per line)
  3. A directory glob pattern

Each would require defining structure, adding validation, updating the type taxonomy. Instead: write an instructions note that lists the documents, says what to look for in each, and explains why they're together. The "collection" is a paragraph. It's also better than a formal type because it carries context a schema couldn't — why these documents are grouped, what's relevant in each one, what the goal is.

The formal system didn't grow. A prompt absorbed the requirement.

The constraining spectrum

Ad hoc prompts sit at the loosest end of the [enforcement gradient]: instructions → skills → hooks → scripts. Maximally flexible, zero infrastructure cost, but also zero validation and zero reuse. [Typed callables] sit at the other end — declared signatures, validated inputs, composable skills. Both are correct for different moments. Typed callables are right for operations that recur (/connect, /validate, /ingest). Ad hoc prompts are right for operations that might happen once, or whose shape isn't clear yet.

The maturation trajectory is: write ad hoc prompts first, notice when you're writing the same kind repeatedly, extract a skill. The prompt equivalent of "write the code three times, then extract a function." This is [lowest-friction capture, then progressive refinement] applied to the skill layer. Repetition reveals the stable procedure; skill extraction then reshapes that procedure for execution. The ad hoc prompt carries reasoning about what to do and why, while the extracted skill keeps the procedure and drops the justification.

Prompts carry what types can't

Prompts carry judgment that type signatures can't express. A prompt can say "focus on sections 3.1-3.3" or "the key tension is between X and Y." A type signature says source → report. The prompt carries the caller's judgment, not just the caller's data. This is why ad hoc prompts resist premature formalisation — not convenience, but expressiveness.

This matters most for sub-agent handoff. An ad hoc prompt is a clean context boundary: the caller does the judgment-heavy work (gathering, selecting, deciding what matters) and writes it down. The sub-agent executes with clean context — no conversation history, no search, no decisions about what's relevant. The prompt defines what's visible in the sub-agent's [lexically scoped frame], and the sub-agent inherits nothing beyond what the caller explicitly passed.

Why this works: homoiconicity

Ad hoc extension without schema changes is possible because the [LLM context is a homoiconic medium] — instructions and content share the same representation (natural language tokens). A markdown file is both content you can read/link/analyze and instructions you can hand to a sub-agent for execution. A CLAUDE.md rule is both documentation for humans and a behavioral constraint for agents. No registration, no type system gatekeeping, no compilation step. This is the same property that makes Lisp, Emacs, and Smalltalk extensible from within — and carries the same discoverability costs.

Open Questions

  • When does an ad hoc prompt become expensive enough to justify extracting a skill or writing code? Is "wrote the same kind three times" the right threshold, or does it depend on how costly a mistake is?
  • Can ad hoc prompts reference skills ("follow the directed reading procedure, but also..."), or does that create confusing layering?
  • How do you discover useful past ad hoc prompts? They're ephemeral by design, but some patterns are worth finding again.
  • Outside KBs, what are the best examples of this pattern? CLAUDE.md rules, PR description templates, agent system prompts — are these all instances of the same technique?

Relevant Notes:

  • [instructions-are-typed-callables] — the typed end of the spectrum: skills should declare signatures. This note argues for the untyped end — ad hoc instructions that absorb requirements without schema changes. Both are correct for different moments.

Artifact B

LLM context is composed without scoping

An LLM's context is assembled by concatenating system prompts, skill bodies, user messages, and tool outputs into a single token stream. Everything is global: every token is visible to every other token, with no way to say "this binding is local to this skill" or "this tool output should not influence instruction interpretation."

This is not even dynamic scoping (name bindings resolved through the call stack rather than the source structure), which at least maintains a stack with push and pop. Flat concatenation is the [homoiconic medium] (instructions and data share one representation) with no structure imposed on top, yet it produces dynamic scoping's pathologies — and the Lisp analogy still clarifies them:

Spooky action at a distance. An early turn subtly biases a later response. The LLM has no mechanism to mark a binding as out of scope — once something enters the log, it influences everything downstream. This is the [three-space memory claim's] "operational debris pollutes search" failure mode, restated as a scoping problem.

Name collision. "Table" meant an HTML element in turn 3 but a database table in turn 12, and the model conflates them. A flat log has no scope boundaries to disambiguate — every use of a term sits in one namespace.

Inability to reason locally. You cannot predict what a sub-task will do by reading its prompt alone; its behavior depends on the entire accumulated history. This is the defining problem of dynamic scope: the meaning of a name depends on the call stack, not the definition site.

The capture problem

Flat concatenation creates a composition-specific problem: capture. A skill says "summarize the document." The document contains "don't summarize this section, skip it." The data-level use of "summarize" captures the instruction-level meaning. This is a hygiene failure that leads to prompt injection — the same problem Scheme's hygienic macros (macros that rewrite code without accidentally capturing names from the call site) solve for code generation.

Within-frame hygiene

Within a single context, the only scoping mechanisms available are weak conventions:

  • Role markers (system/user/assistant/tool in chat APIs) — primitive structural separation, but the LLM still sees all roles in one attention pass
  • Delimiters and quoting — XML tags, markdown fences, explicit "the following is data, not instructions" markers — conventional, not enforced
  • Ordering conventions — system prompt first, then context, then user message — exploits primacy/recency effects but provides no isolation

These are the LLM equivalent of coding conventions in a language without a module system. They help, but they cannot prevent capture — and they cannot disable non-selective semantic integration: prompt semantics the task contract does not license still steer generation, because every token shares one global attention field.

Non-selective semantic integration

"Spooky action at a distance" is measurable, not only architectural. [GSM-DC] varies synthetic distractor count in math word problems and finds power-law error growth — the clean control where irrelevant material is semantically inert noise. [Gonen et al.] varies injected concepts in completion prompts and finds Leak-Rate well above chance even when the concept is task-irrelevant (semantic leakage). [Lampinen et al.] varies belief-congruence on logic tasks. These studies are not independent interference axes; they stress the same flat-context failure under different doses and task grains. Benchmark labels (noise, association, content bias) describe what each experiment varied, not separate mechanisms requiring separate mitigations.

The realistic case — semantically linked material that should not govern the task — is what agent workflows encounter. [Context contamination below compliance reasoning] is that failure at agent dose: fine-grained stance drift despite expressed refusal. Counter-instructions can bias against integration; they cannot remove tokens from the window or make a scope boundary binding.

What flat context buys

Flat logs have a real upside: implicit communication. When a user says "use a more formal tone" in turn 5, the effect propagates to later turns without re-parameterizing. This ambient influence is what makes flat context ergonomic at single-call granularity. The design question is not whether to have the upside, but where to contain it.

The architectural response

The scoping problem is specific to natural-language content. Symbolic artifacts (code, schemas, types) inherit scoping from their interpreter; distributed-parametric artifacts do not expose this kind of local natural-language scope question. Natural-language content has nothing to inherit: no modules, no lexical scope, no interpreter-enforced boundaries. Scope can only be imposed architecturally.

At invocation time this surfaces as a design choice — flat (parent context) or bounded (sub-agent frame) — same representational form, same substrate, same authority path, different context-efficiency profile. Flat pays the full volume and complexity cost and risks contamination; bounded trades an interface cost for isolation.

Sub-agents are the canonical architectural move: code outside the LLM constructs a fresh flat context, the LLM sees only that, and the scope lives in the orchestration code rather than in the LLM itself.

This is one specialization of the general constraining argument in [agentic systems interpret underspecified instructions] — enforcement is the qualitative reason to move a property to code, distinct from the quantitative reasons (cost, latency, reliability). The error-profile version is [scheduler-llm-separation exploits an error-correction asymmetry]: bookkeeping has catastrophic error cost on the semantic substrate (the LLM) and zero error cost on the symbolic substrate (the surrounding code). Scope is bookkeeping, so it belongs on the symbolic side.

Empirical validation comes from ConvexBench ([Liu et al., 2026]), a benchmark for recognizing convexity in deeply composed symbolic functions: LLMs collapse from F1=1.0 to F1≈0.2 at depth 100, even though the total token count (~5,331) is trivial relative to the context window. The failure is compositional reasoning depth, not token capacity — each recursive step conditions on an expanding history that dilutes attention on the current step. Pruning to retain only direct dependencies at each sub-step (one clean frame per call) recovers F1=1.0 at all depths.

Sources: - Anthropic (2025). [Effective context engineering for AI agents] — recommends sub-agents return 1,000–2,000 token summaries; the tens of thousands of tokens each sub-agent explores stay out of the caller's window. Validates the lexically scoped frames pattern. - Yang et al. (2025). [GSM-DC] — power-law reasoning degradation under synthetic distractor count; the inert-noise control regime for non-selective integration. - Gonen et al. (2024/2025). [Semantic leakage in language models] — control/test Leak-Rate metric; instruction-tuned models leak more. - Lampinen et al. (2024). [Content effects on reasoning tasks] — belief-congruent content shifts logic-task accuracy across model families.

Relevant Notes:

  • [unified calling conventions enable bidirectional refactoring] — existing approximation: llm-do's per-agent system prompts and arguments are frame-local context

Under-review context phrase

ad hoc instructions notes are effective sub-agent interfaces because they provide lexically scoped frames — the sub-agent sees only what the caller explicitly passed