Compound Engineering plugin

Type: kb/types/note.md · Tags: self-improving-systems, tool-loop

Evidence basis: first-hand reading of the EveryInc/compound-engineering-plugin checkout at commit 63b6b260 (2026-08-02), covering the README and concepts, plugin manifests, core skills and agents, converter code, scripts, tests, changelog, and the repository's own plans and solution documents. I did not operate the plugin in a live host.

Compound Engineering is a portable prompt-and-agent suite for software work. Its advertised loop is brainstorm -> plan -> work -> review -> compound: code and project knowledge are retained so later work can start with more context (README). That is a real compounding mechanism, but its object matters. Ordinarily the system changes a user's product and retains knowledge about that product. It does not thereby improve the installed plugin's own operative harness.

At the boundary assessed here, Compound Engineering is a human-inclusive, artifact-mediated software factory with several genuine proposal-selection loops and a narrow reflective surface over project operating instructions. It is not, by default, a self-improving harness. The source repository separately shows maintainers evolving and dogfooding the plugin; that historical development must not be attributed to an installed copy's runtime behavior.

Boundary, horizon, and self-representations

The assessed system is one installed Compound Engineering plugin, its host coding agent and permissions, the target repository, and the user, over multiple invocations and git commits. The default deployment assumed here keeps the active plugin corpus outside the target repository and outside any ce-optimize mutable scope; the publisher's source repository and release process are outside the runtime boundary. This makes the classification auditable rather than letting ordinary upstream maintenance masquerade as runtime self-improvement.

A documented local-development topology is an important edge case: the host can load the plugin from a checkout that is also the target repository (README). There the skills are simultaneously target artifacts and active self-representations, so work or optimization can make harness revisions addressable and later invocations can consume them without a publisher release. That expands the reflective surface; it still does not supply a standing failure-to-edit-to-verification loop or demonstrate sustained self-improvement by itself.

Four kinds of readable artifact sit inside the boundary:

  • The installed SKILL.md and agent files describe and causally organize the plugin itself. The manifest exposes the skills, while converter code loads Markdown bodies and installs them as host-native skills or agent instructions (Codex manifest, Claude parser, Codex converter). These are literal behavior-shaping self-representations, but the normal project workflow does not revise or reinstall them.
  • A target repository's AGENTS.md or CLAUDE.md can represent how this installed system should operate in that repository. Compound Engineering both consumes these instructions and, in bounded cases, writes them: ce-compound may add knowledge-store discoverability, and its headless mode can make that edit without asking (ce-compound). This is the plugin's clearest reflective pathway, because the artifact represents an aspect of the assessed system's operation and causally governs how later agents retrieve.
  • STRATEGY.md and most CONCEPTS.md entries instead represent product objectives and domain vocabulary. They influence later behavior but are ordinarily task or environment knowledge, not self-representation. A CONCEPTS.md entry crosses the line only when it describes the assessed system's own behavior-determining organization, such as its workflow or status semantics.
  • Product code, plans, tests, and most docs/solutions/ entries steer later work but usually represent the product or a past problem, not the plugin's organization. They are cumulative operative context without automatically being self-representations. A solution about the team's agent workflow can cross that line; one about an N+1 query does not.

The distinction prevents an expansive boundary from turning every maintained software project into evidence that this particular harness improves itself.

The five-question test

  1. What boundary and what self-representation? Within the declared installed-system boundary, the editable reflective surface is mainly target-repository operating instructions, plus the minority of vocabulary or solution artifacts that describe the assessed system's own workflow. The plugin's own skills and agents are behavior-shaping self-representations too, but they are frozen outside the ordinary installed loop. Product strategy, domain vocabulary, product code, and most retained solutions are the object or environment of work, not the harness.

  2. What independently specifiable objective does evidence bear on? Objectives are often explicit before the change: product strategy and requirements, test expectations, review criteria, an optimization spec's metric and degenerate gates, or the user's acceptance of a baseline. ce-optimize requires a saved spec, mutable/immutable scopes, a measured baseline, and explicit user approval before experiments begin (ce-optimize). The weaker case is ce-compound: “a solved, verified problem” supplies evidence of a fix, but the quality and scope of the generalized lesson are not independently measured.

  3. Does evidence update operative artifacts directly or through candidate selection? Both. ce-compound writes or updates one solution directly after an overlap decision. ce-compound-refresh classifies existing artifacts as Keep, Update, Consolidate, Replace, Delete, or stale and then mutates the chosen document set (ce-compound-refresh). Ideation generates many proposals before adversarial filtering; code review merges and filters reviewer findings; optimization generates hypotheses, runs isolated experiments, and retains only measured improvements. The system should therefore not be described as one uniform “learning loop.”

  4. What produces candidates, and where can one be rejected? LLM subagents, the orchestrator, and the user produce ideas, plans, findings, fixes, lessons, and experiment hypotheses. Rejection occurs at user scope and baseline gates, deduplication and confidence thresholds, independent per-finding validators, tests and lint, degenerate metric gates, and best-versus-baseline comparisons. Code review is unusually explicit: malformed and low-confidence findings are filtered; high-severity and judgment-dependent survivors receive independent validation; mechanically checkable lower-severity findings may be verified directly; and an applied fix is reverted if targeted verification fails. A lower-severity tail may also be dropped when the validation budget is exhausted (ce-code-review). By contrast, a newly written solution has only parser-safety validation as a mandatory executable check; specialized semantic review is optional and happens after the write.

  5. What carries acceptance into later runs, and how reliably is it consumed? Git-tracked code, tests, plans, docs/solutions/, CONCEPTS.md, and instruction files carry accepted changes. The dedicated learnings researcher grounds keywords in CONCEPTS.md, filters docs/solutions/ lexically and by frontmatter, then returns only relevant findings; planning, ideation, and code review dispatch it explicitly (ce-learnings-researcher). This is stronger than hoping a model remembers a folder. It is still prompt-mediated retrieval, not an enforced index: lightweight compounding skips overlap search, refresh is operator-invoked, and host compliance is not tested at runtime.

The result is a qualified classification: the installed system supports reflective improvement of a target project's behavior-shaping self-representations, but its dominant loop compounds project output and project knowledge. It does not establish ongoing improvement of the plugin's own harness.

Search, evaluation, retention, and freezes

Function Placement
Search / candidate production Present in ideation, multi-persona review, refresh classification, and especially metric-driven optimization. Ordinary solution capture has narrow, single-candidate search: research supports one assembled document, while alternative generation and comparative outcome selection are omitted.
Evaluation Present but oracle-bounded. Tests, lint, code inspection, fresh validators, hard metrics, and configured LLM judges can reject code or experiments. Mandatory semantic evaluation of a captured lesson's truth, scope, and future usefulness is omitted; freshness review compares it with later code only when invoked.
Operative retention Present. Product changes and knowledge artifacts persist in repositories, and later skills deliberately retrieve them. Optimization's experiment log is durable across local resumes but lives in gitignored .context/ scratch space, so it is not shared retention unless exported.
Retirement and rollback Present. Refresh can consolidate, replace, stale-mark, or delete knowledge, treating git history as the archive. Code-review fixes are visible diffs and isolated commits when the pre-review tree is clean; failed verification reverts the fix.
Plugin prompt corpus and edit surface Frozen relative to the installed project loop. Skill prompts, schemas, orchestration rules, host model, permissions, and plugin installation are supplied by publisher and platform outside that loop's effective update space.
Runtime harness-update function Omitted. The target workflow contains no standing function that mines its own failures, edits and verifies the installed skills, and publishes or installs a revised plugin.
Later-episode outcome feedback Mostly omitted. Product pulse can report user outcomes and later skills can reuse retained knowledge, but no mechanism attributes later cost, reliability, or autonomy gains to a prior learning and feeds that result back into selection.

Some freezes are protective. An optimization candidate may alter only declared mutable paths; its measurement machinery and immutable scope stay outside the experiment, preventing the candidate from editing its own exam. User approval before baseline execution and before new dependencies similarly protects authority and cost. Product strategy, ambiguous retirement in interactive flows, and outward publication remain human-controlled under the current oracles. Headless solution capture instead delegates or omits semantic acceptance; it does not earn stronger warrant for that migration. The reviewed source does not establish a merely affordable, budget-only freeze. The plugin's own prompt corpus is simply frozen by the default deployment topology; the repository does not argue that this is the ideal permanent boundary.

Addressability, authority, provenance, and verification

Readable artifacts give Compound Engineering strong addressability. Plans have stable requirement and unit identifiers; solutions have paths, categories, tags, and typed frontmatter; vocabulary has named entries; refresh can update, merge, replace, or remove one artifact without retraining a model. Legibility is likewise high: an operator can inspect the operative instruction or retained rule directly.

Reliability is uneven. ce-compound performs duplicate search in Full mode, chooses update rather than duplicate on high overlap, and adds an instruction-file pointer so fresh agents know the store exists. The refresh workflow checks code drift, cross-document conflicts, inbound citations, and the continuing existence of the problem domain before deletion. These are substantive retrieval and retirement controls. The executable validator, however, checks only frontmatter delimiter and YAML quoting hazards; its own docstring disclaims required-field, enum, and semantic validation, and tests confirm that narrow behavior (validator, validator tests). Prompt claims about classification, synthesis, and refresh therefore remain host-model behavior, not properties enforced by the TypeScript package.

Write authority follows host permissions and mode. Interactive flows reserve consequential ambiguity for the user; headless flows move more decisions into the agent, including direct solution maintenance and, for ce-compound, a small instruction-file edit. Code-review default mode may apply clear reversible fixes, verify them, and commit only when starting clean, but never pushes. Optimization isolates candidates in worktrees, commits the winner before integration, remeasures combined runners-up, and reverts harmful combinations. Git supplies rollback once an artifact is committed; a freshly written solution can remain an uncommitted workspace edit until a later shipping step.

Provenance is thinner than rollback. Solution frontmatter records date, module, problem type, component, and search tags, and the skill asks that session-history or auto-memory material be labeled. It does not require source snapshots, evidence citations, author identity, semantic review status, or a retained rejection record for the prose. Later consumption consequently delivers the selected lesson more reliably than the evidential history that warranted it.

Verification-bounded migration is visible throughout the suite. Computation handles repository research, diagnosis, candidate generation, deduplication, mechanical validation, test execution, and metric ranking. Humans retain subject selection, strategy and scope decisions, baseline and dependency approval, ambiguous Replace/Delete judgments in interactive flows, and outward publication. Headless capture can delegate semantic content decisions to the model and leave an uncommitted artifact without independent semantic acceptance; that is additional bare autonomy, not a verification-warranted migration. Fresh validators and executable tests move code-finding acceptance farther into computation because they check evidence external to the review prose. An LLM judge can similarly evaluate an experiment in a fresh context, but its rubric and sample remain human-specified and may share model blind spots.

What the repository's evolution does and does not show

The publisher's repository is a credible dogfood case. It contains its own docs/solutions/, instructs contributors to search them, and stores skill-design lessons alongside executable regression tests (repository instructions, solution example, regression test). The changelog records continuing revisions such as parser-safety validation, headless compounding, strategy and product-pulse skills, and the CONCEPTS.md substrate (changelog). This demonstrates that humans and agents can evolve the prompt corpus through ordinary repository development.

It does not demonstrate sustained compounding. A fix, its lesson, and its regression test often land in the same episode; same-episode gates establish acceptance, not compounding. The source provides no comparison showing that a retained harness revision helped produce a later improvement, directly or through reinvested savings. Nor does an installed plugin collect runtime failures, revise its own installed skills, and release a successor. The honest claim is therefore “designed for retained benefits to feed into later improvements, with examples of later consumption,” not “demonstrated recursively self-improving system.”

The bitter lesson does not select against Markdown

The plugin's natural-language skills, plans, and solutions are representational forms. Their readability does not itself violate the bitter lesson. The relevant question is how they are produced. ce-optimize performs genuine computational search over hypotheses and retains measured winners, and ideation searches over many textual candidates before critique. Most of the broader workflow, however, relies on a large hand-authored prompt architecture and human-selected gates. Text can be the output of search and learning; here, only some text and code artifacts are produced that way. The unresolved scaling question is whether increasingly capable search can revise the orchestration and evaluators themselves without collapsing the external verification that makes adoption warranted.

Relevant Notes: