Learning theory

Type: index · Status: current

How systems learn, verify, and improve. These notes define learning mechanisms, verification gradients, and memory architecture that KB design draws on but that aren't KB-specific — they apply to any system that adapts through durable substrates, including but not limited to inspectable artifacts.

The collection is organized around deploy-time learning as the unifying framework. Accumulation — adding knowledge to the store — is the most basic learning operation, with reach as its key property: facts sit at the low end, theories at the high end. Two orthogonal mechanisms (constraining and distillation) transform accumulated knowledge. A third operation (discovery) produces the high-reach theories that are accumulation's most valuable items.

Foundations

  • agentic-systems-interpret-underspecified-instructions — two distinct properties (semantic underspecification and execution indeterminism); the spec-to-program projection model, semantic boundaries, and the constrain/relax cycle
  • learning-is-not-only-about-generality — accumulation is the most basic learning operation, with reach as its key property (facts at the low end, theories at the high end); capacity decomposes into generality vs a reliability/speed/cost compound; Simon's definition grounds the decomposition
  • continuous-learning-requires-durability-not-weight-updates — the live disagreement is whether durable non-weight adaptation counts as learning at all; this note makes the affirmative case and turns artifact-side adaptation from metaphor into learning proper
  • llm-learning-phases-fall-between-human-learning-modes — LLM phases (pre-training, in-context, deploy-time) occupy intermediate positions on the evolution-to-reaction spectrum rather than mapping 1:1 to human learning modes; warns against literal human-LLM learning analogies
  • in-context-learning-presupposes-context-engineering — in-context learning depends on deploy-time learning to select and organize the right knowledge; Amodei's "no continual learning needed" claim relocates the learning to the system layer rather than eliminating it

Deploy-time Learning

The organizing framework: deployed systems adapt through symbolic artifacts — durable, inspectable, and verifiable — filling the gap between training and in-context learning.

Constraining

Constraining the interpretation space — from partial narrowing (conventions) to full commitment (deterministic code). The primary mechanism for hardening deployed systems.

  • constraining — definition and spectrum: storing an output, writing a convention, adding structured sections, extracting deterministic code; codification is the far end where the medium itself changes from natural language to executable code
  • storing-llm-outputs-is-constraining — the simplest instance: keeping a specific LLM output resolves underspecification to one interpretation; develops the generator/verifier pattern and verbatim risk
  • constraining-during-deployment-is-continuous-learning — AI labs' continuous learning is achievable through constraining with versioned artifacts, which beats weight updates on inspectability and rollback
  • spec-mining-as-codification — codification's operational mechanism: observe behavior, extract deterministic rules, grow the calculator surface monotonically
  • operational-signals-that-a-component-is-a-relaxing-candidate — five testable signals (paraphrase brittleness, isolation-vs-integration gap, process constraints, unspecifiable failures, distribution sensitivity) for detecting when to reverse codification
  • error-messages-that-teach-are-a-constraining-technique — the dual-function property: effective enforcement artifacts simultaneously constrain and inform, because in agent systems the error channel is an instruction channel
  • enforcement-without-structured-recovery-is-incomplete — the enforcement gradient covers detection and blocking but not recovery; maps ABC's corrective → fallback → escalation onto each enforcement layer, with oracle strength determining viable recovery strategies

Distillation

Targeted extraction from a larger body of reasoning into a focused artifact shaped by use case, context budget, or agent. Orthogonal to constraining — you can distil without constraining (extract a skill, still underspecified) or constrain without distilling (store an output, no extraction from reasoning).

  • distillation — definition: the rhetorical mode shifts to match the target (argumentative → procedural, exploratory → assertive); the dominant mechanism in knowledge work because it creates new artifacts from existing reasoning

Information & Bounded Observers

Discovery

A third operation, distinct from both constraining and distillation: positing a new general concept and simultaneously recognizing existing particulars as instances of it. Discovery produces theories — the highest-reach items accumulation can store.

Synthesis

Oracle & Verification

Moved to LLM interpretation errors — oracle theory, error correction, reliability dimensions, and the augmentation/automation boundary now live in the dedicated error-theory area. Key notes:

Memory & Architecture

Applications

Reference material

  • Context Engineering for AI Agents in OSS — empirical study of AGENTS.md/CLAUDE.md evolution in 466 OSS projects; commit-level analysis shows constraining maturation trajectory confirming continuous learning through versioned artifacts
  • On the "Induction Bias" in Sequence Models — 190k-run empirical study showing transformers need orders-of-magnitude more data than RNNs for state tracking; architectural induction bias determines data efficiency and weight sharing, grounding the computational bounds dimension of learning capacity
  • llm-interpretation-errors — oracle theory, error correction, and reliability dimensions migrated here; the error-theory area applies verification concepts specifically to LLM interpretation failures
  • tags — applies learning theory to KB architecture and evaluation; methodology-enforcement-is-constraining bridges both areas
  • document-system — the type ladder (text→note→structured-claim) instantiates the constraining gradient for documents

Other tagged notes