The readable-artifact loop is the tractable unit for continual learning

Type: kb/types/note.md · Tags: learning-theory, deploy-time-learning

Treat continual learning as representational-form coevolution sets the frame: distributed-parametric, natural-language, and symbolic representational forms (ways retained behavior is encoded and consumed) all adapt durably; they are coupled; how their improvement loops relate is the open question. This note takes the practical next step. A full three-way joint optimizer is not the near-term plan. The tractable first slice is the readable pair - natural-language and symbolic together - which shares enough structure to form a single coupled optimization target. Mainstream ML is scaling the distributed-parametric loop; automating the readable-artifact loop is a parallel, under-explored avenue that stands on its own structural merits, regardless of how fast distributed-parametric updates mature.

Why the readable pair is a natural unit

The readable representational forms have distinctive properties that make them a coupled target:

  • Shared runtime medium. Natural-language and symbolic artifacts both live in (or through) the agent's context window at inference time — prompts are context; tool schemas and artifact contents load as context. Every optimization of one trades off against the other because they compete for the same finite budget. Codification makes this concrete: moving behaviour from natural-language into a tool or deterministic function changes its context cost and its verifiability at once — a single edit at the codification boundary that touches both classes.
  • Current tempo. Hand edits to readable artifacts already cycle much faster than large-model retraining. That doesn't promise automated loops will be fast — those procedures are what we're after — but it shows the upper bound isn't slow, giving the direction traction now while opaque-tempo research matures.
  • What they encode is genuinely explicit. Symbolic artifacts encode formal guarantees (type checks, test assertions, schema validation) — codification is permanent advantage in the arithmetic regime, not scaffolding waiting to be subsumed. Natural-language artifacts encode deployment-specific knowledge — organizational policy, domain rubrics, workflow conventions — that changes on its own schedule. Whatever the opaque loop eventually absorbs, these artifacts remain legible, reviewable, and versionable in ways weights are not.
  • Coupling to the opaque layer is already live. The value of improved weights depends on the artifacts they compose with; better weights with stale tools, outdated prompts, or missing evals don't produce a better system. The reverse also holds: what artifacts need to encode depends on what weights already handle well. A readable-artifact loop therefore operates inside an already-coupled system — it doesn't have to wait for three-way joint optimization to produce value.

Strictly speaking, the factoring that matters is not representational form — and not update cost either, because update cost is contingent. Today the readable artifacts update fast and most distributed-parametric mechanisms update slowly, but LoRA adapters, routing networks, memory layers, and embedding updates can move parametric mechanisms into the fast loop, and training advances will move more. What a cheap update step does not automatically acquire is a small validation radius: the cost that matters is not making the change but establishing that nothing else broke. A readable artifact has explicit dependencies — the artifacts that cite it, the tests that check it, the calls that load it — so a change can be reviewed and regression-tested within a bounded causal neighborhood. A parametric update's default radius is the whole behavior surface: the gradient step may be cheap while proving it did not damage unrelated behavior stays expensive, and behavioral probing is the only general way to find out. So the universal form of the argument is to build the tight loop around whatever combines deployment-pace updates with a bounded validation radius, with looser asynchronous coupling to whatever lacks either. A parametric mechanism that acquires both — edits that are scoped, attributable, and regression-checked — joins the fast loop on this argument's own terms, the same condition an authoritative record sets for parametric substitution. That is also the claim's falsifier: parametric methods matching readable-artifact systems on adaptation latency, retained performance, collateral regressions, rollback, and human review cost over a long task stream would collapse the factoring.

Why natural-language + symbolic, not the other pairings

Three two-class pairings are conceivable; only one has the structural couplings to support a near-term joint loop:

  • Natural-language + symbolic is the natural unit. Both are readable, both can be mutated by LLMs, both can be evaluated by execution or judgment, and both compete for the same context budget — so joint optimization has a clean target (trade-offs along the codification boundary) and a clean scarce resource to optimize against. Programming methodologies (agile, TDD, refactoring) already treat symbolic artifacts as objects of iterative improvement; their tooling (version control, diffs, tests, CI, review) transfers directly to natural-language, as KB systems demonstrate. Meta-Harness searches harness code, prompts, retrieval, memory, and context assembly as one artifact surface; Symbolic Learning supplies an earlier whole-harness optimizer with cross-node credit assignment; and Memento-Skills mutates persistent natural-language-plus-code skill folders from deployment feedback. The main missing piece for KBs is cross-session accumulation with automated selection under softer oracles.
  • Distributed-parametric + natural-language now has a narrow automated boundary case: Memento-Skills trains a parametric router to select mutable readable skills while keeping the underlying LLM fixed. That is a real coupling between forms, but not yet joint optimization under one online selection rule; RLHF and provider prompt/model bundles remain one-way or hand-tuned precursors.
  • Distributed-parametric + symbolic now has a provisional instantiation in Co-Harness: harness repair can change tools and middleware before the improved trajectories fine-tune model weights, and the updated model enters the next harness round. Because its harness also contains natural-language prompts and skills, it is an all-three-form case rather than a clean two-form experiment, and its evaluation does not isolate the recurrent coupling from extra training and search.

Each two-class unification would do for its scope what end-to-end gradient descent did for features plus classifier: extend the general method and eat the hand-crafted alternative. The natural-language + symbolic pairing has the structural couplings that give it the most traction today.

From engineering to loops

Hand-crafted work on readable artifacts — including Commonplace — occupies a specific role in this picture. It is simultaneously engineering (the current category) and loop discovery (discovering what automated evolution of these artifacts would require). The bitter-lesson logic is visible only in hindsight: SIFT and HOG were not wasted; they showed what representation learning needed to capture. Methodology work on durable artifacts serves the same role. It articulates the operators (codify, relax, constrain, adapt), the signals deployment produces, and the selection rules for when a revision sticks.

Hand-crafted evolution is also better than no evolution. Until automated loops for readable artifacts arrive, hand curation keeps deployment insights from evaporating. Without it, each session starts fresh and knowledge fails to accumulate across runs. The currently available alternative to Commonplace-style curation is not automation; it is forgetting.

The goal is to graduate from engineering to loops — first for the readable pair, where structural couplings give the most traction, and then across the wider three-form picture as fast distributed-parametric mechanisms mature.

Relevant Notes: