In-context learning presupposes context engineering
Type: note · Status: seedling · Tags: learning-theory
Amodei argues that continual learning may be unnecessary because pre-training + RL + in-context learning suffice. The claim treats in-context learning as a given capability — million-token windows can hold enough deployment-specific information within a session, so persistent cross-session adaptation adds little.
But in-context learning doesn't happen by itself. It only works when the right knowledge is already in the window. Something has to decide what "right" means, find the relevant knowledge, organize it, and load it — all before the model sees a single token. That something is context engineering: routing, loading, scoping, and maintenance.
Context engineering is not static infrastructure. It improves over deployment time. Teams learn which knowledge to route into context, how to structure it for the model, when to prune accumulated debris, and how to scope what each agent sees. This improvement produces exactly the artifacts the deploy-time learning framework describes: versioned prompts, routing rules, retrieval strategies, schemas, evals. These artifacts are durable, inspectable, diffable, and testable — everything that in-context learning alone is not.
Amodei's move is to eliminate weight updates during deployment. But in doing so he didn't eliminate learning — he relocated it. The learning moved from the model's weights to the system layer that feeds the model's context. That system layer is continuously learning in Herbert Simon's sense: it undergoes permanent changes that improve its capacity for adaptation.
The three timescales remain necessary. Pre-training builds the general capability. In-context learning applies it within a session. Deploy-time learning builds the machinery that makes in-context learning effective — without it, the model has capability but no way to aim it at the right knowledge.
Relevant Notes:
- context engineering — foundation: the machinery (routing, loading, scoping, maintenance) that in-context learning presupposes
- deploy-time learning: the missing middle — foundation: the framework that context engineering improvement belongs to; the artifacts it produces are the substrate of deploy-time learning
- constraining during deployment is continuous learning — extends: the system-layer adaptation that feeds context engineering meets Simon's definition of learning
- LLM learning phases fall between human learning modes — extends: the three timescales remain necessary even if weight updates are eliminated; this note adds a dependency arrow between in-context and deploy-time
- Dario Amodei ingest — responds to: Amodei's claim that continual learning may be unnecessary