h t t p s : / / a r x i v . o r g / a b s / 2 6 1 0 . 1 1 6 5 5

Ingest: Harness Evolution Hits a Ceiling: When Weight Training Should Begin

Type: types/ingest-report.md

Classification

This is an empirical scientific paper. It reports self-evolving harness runs, LoRA and full fine-tuning on the resulting trajectories, component ablations, a placebo control and a second-benchmark replication, plus one small selection-optimism bound. Author: Tian, Hu (independent researchers, corresponding), Wang (UC Berkeley), Lu (Purdue) and Wu (Stanford). The arXiv v1 preprint (8 October 2026) is under review, and the code has not yet been released.

Summary

The paper asks when to improve a long-horizon agent by evolving its harness around a frozen model and when to train the weights. An automated checker labels each failed trajectory by the first signal that fires. The classes are F1 (unbacked claims), F2 (harness friction), F3 (loops or exhausted step budgets) and F4 (a delivered plan that is poor). The composite score factors exactly as delivery rate times quality among delivered plans. On DeepPlanning (48 development and 72 held-out tasks; four-rollout means; differences below 0.035 treated as noise), an AHE-style propose-and-judge loop lifts Qwen3.5-4B from 0.16 to 0.30 on held-out tasks and 9B from 0.32 to 0.44. The gain comes mostly from fewer loops and higher delivery, and F4 is never reduced. Ablations credit capability-granting components (a state ledger, a lessons file, invented tools) rather than enforcement components. The authors attribute the F4 ceiling to the loop's evidence package: it never showed the proposer the scorer's per-check failure composition, and the keys that define which metrics are observed sat outside the surface the loop could edit. LoRA adapters trained on evolved-harness trajectories add about +0.13 held-out under the seed harness. They stack with the harness on 4B and substitute for it on 9B, where they cut F4 from about a quarter of trajectories to about one in twenty. A 9B placebo adapter trained on answer-shuffled outputs falls below the base model. On WebArena-Lite the harness gain (+0.088 on 117 test tasks) came mainly from showing two pages of history instead of one, and adapters trained on those trajectories added nothing. The authors conclude with a "diagnose-then-intervene" rule. Failure composition picks the lever. After a harness gain, the edit surface predicts whether the gain trains in: gains that changed what the model writes did, and gains that changed what the model sees stayed in the harness. That surface rule rests on one DeepPlanning-versus-WebArena-Lite contrast, which also changes benchmark, scorer and model setup.

Contribution assessment (our opinion)

The problem is real and timely. Harness search and weight training are now both routine levers, and concurrent work alternates them by schedule rather than by diagnosis. The most solid contribution is methodological: an explicit noise floor, a fresh anchor after every accepted change, held-out reporting, a placebo control and a crossed harness-by-weights evaluation. Together these let the paper say which gains a trained model keeps without the runtime, which is more than most harness-evolution papers measure. The lever-selection rule is a useful conjecture with modest support. Nine evolution lines separate by seed-time delivery and quality. But the three no-gain lines are also the weakest models, so "plan quality below about 0.18" may simply mean "model too weak". The weights branch has one positive case (Qwen 9B) and one open case (Gemma 4 12B). The evidence-package account of the F4 ceiling is plausible, but it is post hoc: the one remedy run, on a different model, was within noise. The sees-versus-writes transfer rule is the most novel idea and the least tested, with one line on one benchmark. Originality relative to Co-Harness, WHALE and Harness-Zero is qualified by the paper's own comparisons. Its distinctive additions are diagnosis-driven switching, the placebo and the crossed design. I cannot judge priority beyond those comparisons. Likely significance: a careful, small-scale empirical template and a set of testable conjectures, rather than an established decision rule.

Quotes

No source quotes have been retained yet.

Connections Found

The source is a bounded empirical case in the KB's cross-form learning cluster. Its role is twofold: evidence about diagnosis and allocation of gains between the harness and the weights, and a counterpoint to the headline of the lead article Can a theory builder running on fixed-weight LLMs learn?. The counterpoint is in the paper's title and decision rule, which send content failures to weight training. The paper's own §6.2 withdraws the general claim: the ceiling is "a property of this loop's evidence package, not of harness evolution in general". The Learning Claims section sets the paper's conclusions against our bets item by item.

It is most directly evidence for Diagnostic richness constrains outer-loop learning quality. The scorer did detect F4, so the selection oracle was not the bottleneck. The per-check failure composition never reached the proposer, and the accepted edits targeted only what the evidence package surfaced. That would add a case with a strong oracle and poor diagnostics beside the note's existing contrasts. It stays a bounded case because the single attempt to expose the composition did not produce a resolvable gain.

The same evidence-package account is the paper's own version of the first conjecture in Commonplace builds a theory builder and tests whether it learns: a loop that generates variants and selects them by score, with no formulated reason for a failure, stalls where a stated diagnosis would have directed the next attempt. The Gemma 4 12B line, where the aggregated failure composition was added to the package and the gain stayed within noise, is the first measurement in this setup of what a richer signal buys. It is a null result with narrow scope (one line, one model, counts rather than a diagnosis), and it belongs beside the conjecture as such.

It also bears on Treat continual learning as representational-form coevolution. The note leaves credit assignment across forms open. The paper's crossed seed/evolved × base/adapter design separates the two forms' contributions more cleanly than Co-Harness does. The 9B line isolates the cleanest instance: its closing harness is the seed harness plus one lessons file, and an adapter trained on that harness's trajectories matched it on held-out tasks. That is natural-language instruction content moving into distributed-parametric form by training. The same results are evidence for The deployed system, not the model alone, is the unit of learning: the WebArena-Lite gain lived in what the model sees, and a model-only loop did not capture it. They also support the side remark in Scaling absorbs scaffolding at fixed task difficulty, not at the deployment frontier that functions can migrate into weights. On 9B the migration happened by training at fixed task difficulty, not by scaling. The context-supply gain did not migrate.

The evidence-package property is also an instance of the information-omission case in Learning inside a fixed decomposition inherits its mistakes. The metric-defining keys sat outside the loop's editable directories, so the loop could not change what it observed about F4. The paper limits this to its own loop, not harness evolution generally, and that limit should be kept.

Among sources, it compares with the Co-Harness ingest on how the update form is assigned. Co-Harness routes failures per trajectory through a critic; this paper routes on aggregate first-signal composition and adds a placebo. It compares with the WHALE ingest on which lever pays where. WHALE alternates on a schedule; this paper predicts the lever from seed measurements and orders the harness first. It compares with Harness updating is not harness benefit on crossed pairing as an attribution design. The paper's loop is modelled on Agentic Harness Engineering, and its ceiling analysis is a criticism of an AHE-style evidence package led by intervention metrics.

Learning Claims (our opinion)

Mechanism on its own terms. Two adaptation loops run in sequence. First, a proposer model (Kimi K3) reads an evidence package from the current harness's runs: score, healthy rate, F1–F4 counts, process metrics and worst cases. It proposes two harness edits per round. Each edit carries a manifest and its own rollback condition. A judge applies fixed house thresholds plus the proposer's rollback condition. The better accepted edit is kept, and a fresh anchor is measured. The editable surface is prompt files, policy parameters, middleware and tools. Second, the best evolved-harness trajectories (score-filtered, STaR-style) become supervised training data for LoRA or full fine-tuning. The paper's learning claim is about allocation. Harness evolution instils process behaviour, that behaviour and plan content can be trained into weights, and the edit surface predicts which harness gains survive removal of the harness.

Theory-builder conditions, for the harness loop. - Localized content: strong as form, weak as theory. The harness consists of stated prompts, a lessons file, tools and policy, and the edits come with manifests. But what the units say is operational ("plan submission format fix", "coordinate lookup fallback tool"), not a claim about why the agent fails; the candidate log (Table 3 and Table 4, 36 candidates) contains no stated theory of the task. - Consumption: strong. The agent acts through the harness, and the ablations show particular units (lessons, ledger, tools) change scores. - Criticism: fails as the definition reads it. The proposer sees metrics and worst cases, not a formulated criticism of what the current harness says. Acceptance is outcome-threshold selection against the anchor on four rollouts. Each candidate's rollback condition is a stated test of a consequence, which is the one element of condition 3 present. The loop is variant-and-select with a bare failure signal, the comparison case the first conjecture names. The authors do criticise the loop's machinery (the evidence package, inert accepts), but outside the loop, in the research process. - Iteration: strong within a run. Accepted edits and fresh anchors shape the next round.

Persistence reaches across the rounds of one run. The evolved harness then becomes a frozen artifact, reused for training data and for frontier-model checks. The loop did not revise the evidence package, the taxonomy or the judge rules. The weight-training stage is weight adaptation by gradient on filtered trajectories. It fails conditions 1 and 3 and is outside the builder. Here the model is an addressable component whose replacement or retraining the human researchers decide.

Learning, judged separately. Held-out gains for both harness lines (+0.133 for 4B, +0.119 for 9B) and for the adapters support improved capacity on held-out tasks from the same benchmark. The causal attribution is better supported for the adapters, through the placebo, than for any single harness edit. The single-edit attribution comes from ablations run at two rollouts for 4B.

Fixed decomposition. The effective update space of the harness loop excluded the evidence package's composition, the metric definitions, the proposer, the judge thresholds, the F1–F4 taxonomy and the training recipe. F4 is defined as the residual when no process signal fires. A harness that acts only on process signals therefore has no signal that distinguishes F4 from a healthy trajectory. The "harness for process, weights for content" split is partly built into that taxonomy rather than discovered. The 9B harness gain was in fact a quality gain among delivered plans, and the 4B adapter learned process behaviour, not content. A more accurate reading is that the harness loop removed the process failures it could see, and filtered-trajectory training could carry both kinds of gain. The paper's central evidence is improvement inside this decomposition. It does not compare alternative taxonomies, evidence representations or training-signal designs.

The paper's conclusions against our bets. Each item states the paper's claim, the bet it meets, and what the paper's data say about that bet. The bets are Commonplace's, held as bets; the paper tests none of them directly.

  1. Paper: content failures are what the weights are for; harness evolution hits a ceiling at them. Our bet (the lead article): a fixed-weight system can hold its learning in text that it retains and revises, including learning that improves what the model produces. Data: the paper's own §6.2 locates the ceiling in the loop's evidence package, and the 9B line contradicts the headline inside the paper: three edits to a natural-language lessons file raised quality among delivered plans from 0.309 to 0.392 on a model whose delivery was already 97%. The paper reads that rise as removed loops (39% to 17%); the lessons edits were content instructions (plan-emission format, hotel-location checks), and the paper reports no paired same-task comparison for 9B that would separate the two readings. Verdict: the headline does not bear on the bet; the data are consistent with it.

  2. Paper: the loop stops at F4 because the proposer never saw the scorer's failure composition; surfacing it (Gemma 4 12B) gave a gain within noise. Our bet (first conjecture, criticism of content): a formulated reason for a failure, supplied to the next step, yields more learning than variant-and-select on a score. Data: the paper's ceiling is the predicted failure mode of the comparison case. The Gemma remedy is the first measured test of a richer signal in this setup, and it is null. Its scope is narrow: one line, a model the paper's own rule sends to the weights branch, and the signal added was aggregated counts to a single-call proposer without tools, not a stated diagnosis. Record it as a bounded negative, not as a refutation.

  3. Paper: gains from edits to "what the model does" (tools, prompts, lessons, blockers) train into adapters; gains from "what the model sees" (two pages of history) stay in the harness. Our reading (a bet, not the paper's): the paper does not distinguish symbolic from natural-language harness content, and its own ablations show the natural-language units (lessons, prompts) carried the gain while enforcement code stayed within noise. The boundary that fits both results is between content fixed across tasks (instructions, which training can absorb) and content specific to each task (retrieved context, which weights cannot hold). This restates the paper's rule in terms of inputs rather than outputs and rests on the same single contrast.

  4. Paper (by omission): every one of 36 candidates was a local patch; none changed the method the agent works by. Our bet (unmeasured): a harness-evolution loop can discover methods, not only patches, when its evidence states needs rather than metrics, its proposer is asked to search for analogous fields, and its evaluation measures what a method improves. Commonplace's own instance is intent-plus-end-state delegation, found by hand through von Hippel's advanced-analog-field search and recorded as intent-framed delegation; its gains are observed in use, not measured. Data: the paper's loop has none of the three conditions, so its candidate log is a negative instance of the comparison case, not evidence against the bet. The one failure the paper names and never fixed, 25% of 9B plans submitted as several drafts, is an end-state failure that no instruction addressed.

What it adds to the current account. The paper's crossed design and the 9B substitution give the form-coevolution note its cleanest instance of natural-language content moving into weights by training. The edit-surface rule, restated as fixed-versus-task-specific inputs, is a testable conjecture about which gains can move between representational forms; it rests on one contrast that also changes benchmark and scorer. The Gemma null is the first bounded datum on the criticism conjecture. The source supports the diagnostic-richness account but does not establish it here, because the remedy test failed. It puts in question any reading of "harness ceiling" as a property of harnesses as such. The authors themselves locate it in this loop.

Extractable Value

  1. A natural-language-to-weights instance isolated in one ablation cell. The 9B closing harness is the seed harness plus one lessons file; an adapter trained on its trajectories matched it on held-out tasks and cut F4 from about a quarter to about one in twenty. This is the form-coevolution note's open credit-assignment question answered for one case: instruction content fixed across tasks trained in. [quick-win]
  2. Edit surface as a predictor of cross-form transfer, restated by input kind. Gains from content the model reads on every task (prompts, lessons) trained into adapters on DeepPlanning. A gain from per-task context (two pages of history) did not train in on WebArena-Lite. The paper states the rule as writes-versus-sees; the restatement is ours. It rests on one contrast that also changes benchmark and scorer, so it should be recorded as a conjecture. [experiment]
  3. A case with a strong oracle and poor diagnostics. The scorer saw F4, but the proposer's evidence package did not aggregate it, and no accepted edit touched the 25% multi-draft rate the scorer penalised. This gives the diagnostic-richness note a contrast its existing cases lack. Keep the caveat that the one remedy attempt (Gemma 4 12B) was within noise. [quick-win]
  4. The Gemma line as the first bounded test of the criticism conjecture. Adding the scorer's aggregated failure composition to the evidence package bought +0.019/+0.012, within noise, on one model the paper's rule assigns to the weights branch. It is a null with narrow scope and belongs in the research companion's evidence as such. [quick-win]
  5. A candidate log with no method-level proposal. Thirty-six candidates across two lines, all local patches (format fixes, fallback tools, checkers, truncation detection), from a proposer that saw metrics and worst cases in a single call without tools. A negative instance for the bet that harness evolution can discover methods, bounded to loops of this shape. [just-a-reference]
  6. Crossed harness × weights evaluation as an attribution design. Evaluating seed and evolved harness against base and adapter weights on held-out tasks, with a placebo adapter, separates gains the trained model keeps from gains that need the runtime. This is reusable wherever the KB judges cross-form learning claims. [quick-win]
  7. Measurement protocol for noisy agent interventions. The protocol has five parts: a calibrated per-rollout noise floor, four-rollout means, a fresh anchor after every accept, same-night comparisons, and a sub-Gaussian bound on selection optimism (E[gap] ≤ s√(2 ln K)). With single-rollout readings over 10–30 candidates, that bound exceeds the noise line. The paper also requires evidence that an edit's code path fired before judging it, because inert edits passed the judge four times. These are concrete operational warnings for any score-selected improvement loop, including unmeasured prompt changes judged from a few runs. [just-a-reference]
  8. Score identity S = r·q̄ as a lever cap. A lever that only raises the delivery rate is capped by the quality of what the model already writes. The three weak models gained delivery while the freed trajectories landed in F4. This explains why process repair can leave the score unchanged. [just-a-reference]
  9. Capability-granting versus enforcement components. Removing gates, checkpoints and hard blocks mostly fell within noise, while removing lessons, ledger and invented tools cost the most. One exception is the 4B exit gate, which held up held-out delivery. This is a context-bound data point on which harness components carry gains for small models on one planning benchmark. [just-a-reference]

Limitations (our opinion)

The central comparisons are small and context-bound. There are two Qwen sizes on one planning benchmark, one replication line on WebArena-Lite, and 48 development tasks for every decision. The weights branch has one clear positive (9B) and one null (Gemma). The lever-selection rule's three no-gain lines are also the three weakest models. Seed plan quality and general model capability are therefore confounded, and the rule may reduce to "the harness lever needs a capable enough model".

The missing baseline that matters most is rejection-sampling fine-tuning on the model's own filtered trajectories under the seed harness. The placebo shows that shuffled outputs hurt. It does not show that the evolved harness was needed to produce a useful textbook rather than any score-filtered self-training data. Without that control, "the behaviour the harness instils can be trained into the weights" is not separated from ordinary STaR-style self-improvement. The placebo also exists only for 9B.

The process/content split is partly definitional, and it hides an observed content gain. F4 is the residual left when no process signal fires, so a harness acting on process signals cannot target it by construction, and F4 counts only plans scoring below 0.5, so an improvement that leaves a plan below that line is invisible to the F4 metric. The paper's hypothesis statement treats any gain made of removed F2 and F3 trajectories as "a behaviour". Under that reading the 9B rise in quality among delivered plans, produced by content instructions in the lessons file, is credited to loop removal. The paired same-task comparison that supports this reading for 4B (+0.011/+0.022) is not reported for 9B in the main text, where delivery was already saturated and the delivered set barely changed. The paper also bundles tools, prompts, lessons and blockers as "what the model does", without separating symbolic from natural-language content, although its ablations put the gain on the natural-language units.

The edit-surface rule rests on one WebArena-Lite line whose adapters were measured on a different night, against a coarser 0.064 noise line. That contrast differs from DeepPlanning in benchmark, scorer (LLM-judged answers) and task type, not only in edit surface.

The evidence-package explanation of the F4 ceiling is a post hoc account. Its one test was within noise on a model the rule assigns to the weights branch, and what was surfaced was aggregated counts, not a diagnosis, so the test does not reach the criticism conjecture's intended contrast. There is no matched-budget test-time-search baseline, which the authors acknowledge. The plan-to-JSON converter inflates scores by about 0.025, varies with the multi-draft rate across configurations, and sits close to the 0.035 noise line. The frontier-model checks (Kimi K3, GLM 5.3) lack noise estimates and released run data. Code was not released at capture, and nothing here was reproduced.

What the benchmark cannot see bounds every negative above. A fixed task set with a parsing scorer registers delivered, parsable plans; it does not register fewer wrong-direction runs or less supervision on open tasks, which is where method-level gains such as intent-framed delegation would show. A loop evaluated this way can only find methods whose gains its scorer measures.

Add the Gemma 4 12B line to the Evidence section of Commonplace builds a theory builder and tests whether it learns as the first bounded test of the criticism-of-content conjecture: a variant-and-select harness loop that stalled where a stated diagnosis would have pointed, and a remedy that surfaced aggregated counts rather than a diagnosis and bought a gain within noise. State its scope (one line, one model, counts not criticism) beside the result, and cite this ingest. The diagnostic-richness paragraph recommended by the connect pass is the same case seen from that note's side and can follow.

Relevant Notes: