The boundary of automation is the boundary of verification

Type: note · Status: seedling · Tags: llm-interpretation-errors

Tasks become automatable when verification is cheap and resist automation when verification is expensive — regardless of raw model capability. This is not an observation about current limitations. It's a structural claim: generation without verification produces output, not automation. Where automation stalls, the bottleneck is typically oracle construction, not generation.

Three sources arrive at this claim through different reasoning, from different domains, using different vocabulary. They are not fully independent — the oracle-theory notes already cite Tam et al. and Rabanser et al. — but the reasoning paths are distinct enough that the convergence is informative.

The evidence

Oracle theory (internal). The oracle-strength spectrum proposes a gradient from hard oracles (exact, cheap, deterministic) to no oracle (vibes). The augmentation-automation boundary identifies the mechanism: crossing from augmentation to automation requires per-instance discrimination (knowing this output is wrong), not aggregate accuracy. Rabanser et al. find that calibration improves across model generations but discrimination trends are mixed — improving on some benchmarks, worsening on others — suggesting self-assessment is not reliably scaling, which favors external oracles. The MAKER system demonstrates the endpoint: zero errors over a million steps, achieved entirely through external hard oracles, with no reliance on model self-knowledge.

Labor economics (Tam et al.). When code is free, research is all that matters argues that AI commoditizes engineering (which has tests, specs, benchmarks — hard oracles in our vocabulary, though Tam doesn't use that term) while research taste resists automation because problem selection has no ground truth. Tam argues market pricing reflects this — quant firms paying $600k for "research taste" — though this could also reflect tournament dynamics or talent scarcity rather than oracle strength per se. Karpathy's autoresearch automates hyperparameter sweeps (verifiable) but not problem selection (unverifiable) — the boundary runs through a single tool.

Capability-timeline predictions (Amodei). Amodei's interview shows a confidence gradient: strong optimism on coding and math (where progress is measurable against tests and benchmarks) but acknowledged uncertainty on novel writing and scientific discovery (where quality is harder to verify). Amodei doesn't use oracle vocabulary — this is our interpretive frame — but the pattern is consistent: his confidence correlates with verification availability, not raw capability claims.

Why convergence matters

Any single source is explainable without the framework. Amodei's confidence split could be mere selection bias (he has benchmarks for coding, not for novels). Tam's labor-economics argument could be an investor thesis dressed up as analysis. The oracle-strength spectrum could be an internally consistent theory that happens not to be true.

But three sources — theory, market economics, and supply-side capability predictions — arriving at the same structural claim through different reasoning is harder to explain away than any single source. The convergence makes this a candidate for a general principle rather than a domain-specific observation, though the shared citations between the sources temper the evidential weight.

The practical implication

If this holds, the leverage point for expanding automation is not better models but better oracles. The engineering priority becomes: invest in verification infrastructure before capability. Spec mining manufactures oracles. Error correction amplifies weak ones. The path to automating any task starts with the question: can we build a verifier?

This applies to KB curation directly. Automating KB learning stalls on judgment-heavy mutations (synthesis, connection quality, what to skip) — exactly the operations where oracle construction is hardest. The bottleneck is not that agents can't generate candidate mutations; it's that no one can cheaply verify whether a proposed mutation improves the KB.

Caveats

  • The claim is about structure, not permanence. Oracle construction difficulty is not fixed. Domains that are no-oracle today may become hard-oracle tomorrow through better tooling, better metrics, or domain decomposition. The claim predicts where automation stalls, not that it stalls forever.
  • Convergence is not proof. Three sources agreeing could reflect a shared assumption rather than an independent discovery. All three operate within a broadly rationalist, verification-oriented worldview — a critic from a different tradition (e.g., one that values tacit knowledge or embodied practice) might see the convergence as circular.
  • Error-cost tolerance is a separate variable. Some tasks get automated despite poor verification because errors are cheap — machine translation for low-stakes content, draft generation for human review. The framework focuses on verification cost but doesn't account for domains where tolerance for unverified output is high enough that oracle construction becomes unnecessary.
  • Oracle gaming is unaddressed. The framework treats oracle availability as uniformly positive, but cheap oracles can produce pathological automation — recommendation algorithms optimizing engagement metrics, teaching to the test, RL reward hacking. In these cases, the oracle exists and is cheap, yet automation against it is actively harmful. Oracle quality matters, not just oracle availability.
  • The framework may not cover all cases. Some tasks resist automation for reasons other than verification difficulty — regulatory constraints, trust requirements, liability concerns. The title uses "the" boundary as a claim title, but the argument defends verification as the primary structural boundary, not the only one.

Relevant Notes: