Ingest: Natural-Language Agent Harnesses

Type: kb/sources/types/ingest-report.md

Source: natural-language-agent-harnesses.md Captured: 2026-03-28 From: https://arxiv.org/html/2603.25723

Classification

Type: scientific-paper — preprint with a defined framework (NLAH + IHR), module ablation study, and evaluation on two established benchmarks (SWE-bench Verified, OSWorld). Has methodology, experimental structure, and citations.

Domains: agent-orchestration, harness-engineering, specification-level-separation, deploy-time-learning

Author: Linyue Pan, Lexiao Zou, Shuo Guo, Jingchen Ni, Hai-Tao Zheng (Tsinghua University affiliation based on Zheng's known faculty position). Academic research group — attend to the framework's rigor and benchmark methodology, but note this is a first publication of the framework without independent replication.

Summary

The paper introduces Natural-Language Agent Harnesses (NLAHs) — a framework for externalizing agent control logic (contracts, roles, stage structure, failure taxonomy, state semantics) as portable, inspectable natural-language artifacts rather than burying them in controller code. The companion Intelligent Harness Runtime (IHR) interprets these artifacts at execution time, separating a shared runtime charter from task-specific logic, with file-backed state for durability under context truncation. Evaluated on SWE-bench Verified and OSWorld, the key finding is that "more explicit structure does not automatically improve end-task performance" — modules help most when they tighten alignment between intermediate behavior and evaluator acceptance criteria, with concentrated effects on frontier cases rather than uniform improvement. The paper positions harness engineering as a first-class research object amenable to systematic study through ablation.

Connections Found

/connect found 10 genuine connections, 2 bidirectional candidates, and 1 synthesis opportunity.

Strongest structural connections:

  • Agent runtimes decompose into scheduler, context engine, and execution substrate — exemplifies: NLAH's IHR independently derived the same three-part decomposition. Runtime charter = context engine, stage structure = scheduler, file-backed state = execution substrate. Strong converging evidence for the decomposition from an independent research group.

  • Specification-level separation recovers scoping before it recovers error correction — exemplifies: NLAH occupies exactly the intermediate regime this note describes — control flow is named (stages), state protocols are externalized (file-backed state), judgment holes are marked (roles), but execution still depends on LLM compliance through the IHR. The paper provides SWE-bench/OSWorld evidence for the intermediate regime, stronger than the OpenProse case study alone.

  • Process structure and output structure are independent levers — extends: NLAH's key finding provides empirical evidence for when process structure helps — only when it constrains the dimension load-bearing for the task's acceptance criterion. This is a concrete constraint on the open question that note raises.

Complementary connections:

  • Methodology enforcement is constraining — exemplifies: NLAH's contracts (inputs, outputs, validation gates, retry rules) are methodology enforcement at the specification level. The failure taxonomy with named recovery modes maps onto the enforcement gradient between skills and hooks.

  • LLM context is a homoiconic medium — exemplifies: NLAH explicitly treats NL as carrying "editable, inspectable orchestration logic" — artifacts are both content the LLM reads and programs the LLM executes.

  • Deploy-time learning is the missing middle — exemplifies: NLAHs are durable symbolic artifacts (portable, versioned, ablatable) that improve system behavior without weight updates. The emphasis on ablation studies is verification methodology for deploy-time learning.

  • ABC: Agent Behavioral Contracts — complements from opposite direction: ABC uses a formal YAML DSL with mathematical compliance guarantees; NLAH uses natural-language contracts interpreted by the LLM. ABC pushes toward full formalization; NLAH stays in the intermediate regime. Together they bracket the contract-specification design space.

Synthesis opportunity: The connection report identified a higher-order claim that no existing note captures: structure helps when it constrains the dimension load-bearing for the task's acceptance criterion, with a benefit ordering of scoping > process alignment > error correction. This combines NLAH evidence with the process-structure, specification-separation, and methodology-enforcement notes.

Extractable Value

  1. Empirical constraint on when process structure helps (high reach). NLAH's key finding — explicit structure helps only when it tightens alignment with evaluator acceptance criteria — is a falsifiable, high-reach claim. It says why structure sometimes fails (it constrains dimensions that aren't load-bearing for the task). This directly extends process-structure-and-output-structure-are-independent-levers with benchmark evidence. [quick-win]

  2. "Concentrated effects on frontier cases" pattern. Process structure didn't uniformly improve performance — gains concentrated on frontier cases (hard instances near the solve boundary). This is a distributional claim about where structure helps, not just whether it helps. If replicated, it has design implications: invest in structure for the hard tail, not the easy middle. Moderate reach — the pattern is plausible for any capability-constrained system, but tested only on SWE-bench. [experiment]

  3. Independent derivation of the runtime three-part decomposition. NLAH's IHR arrived at scheduler + context engine + execution substrate without citing the KB's decomposition note. This is converging evidence from an independent source, which strengthens the decomposition claim. [just-a-reference]

  4. Ablation methodology for harness engineering. The paper's framing of harness components as independently removable modules that can be systematically ablated provides a concrete experimental methodology for evaluating harness design choices. Relevant to how we evaluate our own harness decisions. [experiment]

  5. Code-to-text migration patterns (RQ3). Native code harnesses shifting to "file-backed state and artifact-backed verification" under NL execution, with improved observability but relocated reliability mechanisms. Evidence for the intermediate-regime thesis: you gain inspectability but relocate (not eliminate) the reliability burden. Low reach — specific to their migration path. [just-a-reference]

  6. Synthesis target: "structure helps when it constrains the load-bearing dimension." The connection report flagged this as a higher-order claim combining NLAH evidence with three existing notes. This is the highest-value extraction: a new principle grounded in empirical evidence that unifies existing observations. [deep-dive]

Curiosity Gate

What is most surprising? That adding a verifier module and explicit search structure sometimes hurt performance by diverging from benchmark acceptance criteria. Most harness engineering literature assumes more verification = better. NLAH's finding that verification can be counterproductive when it optimizes the wrong dimension is genuinely unexpected and worth taking seriously. It suggests a failure mode where well-intentioned process structure adds overhead without constraining the dimension that matters.

What's the simpler account? The simpler explanation for "structure helps only when aligned with acceptance criteria" is just overfitting to the evaluator: any intervention that happens to match what the benchmark measures looks helpful; any that doesn't, doesn't. This is tautological under the simpler account. The paper's claim is stronger — that you can predict which dimensions matter and design structure accordingly — but the evaluation doesn't demonstrate that predictive capability, only the post-hoc observation.

Is the central claim hard to vary? Partially. The claim "structure helps when it constrains the load-bearing dimension" is hard to vary in one sense — it's hard to argue that constraining irrelevant dimensions would help. But it's easy to vary in another — it doesn't specify how to identify the load-bearing dimension in advance, which makes it more descriptive than predictive. The finding becomes harder to vary if combined with the specification-separation ordering (scoping > process alignment > error correction), because that ordering predicts which dimension becomes load-bearing first.

Limitations (our opinion)

Benchmark-specific alignment finding may be circular. The key finding — structure helps when aligned with evaluator acceptance criteria — was discovered by evaluating on benchmarks with specific acceptance criteria. It is unclear whether this transfers to real-world deployments where acceptance criteria are ambiguous, multi-dimensional, or unstated. The finding may partly describe a property of benchmark evaluation rather than a general property of structure. This limits reach: the mechanism (constrain the load-bearing dimension) is plausible in general, but the evidence is benchmark-bound.

No comparison with alternative orchestration approaches. The paper compares NLAH modules against their own ablations (module present vs. absent), but not against alternative orchestration strategies — e.g., code-based schedulers, hybrid approaches, or simpler prompt-engineering baselines. Without these comparisons, we cannot distinguish the effect of NL-based orchestration from the effect of having any explicit orchestration. The specification-level separation note's three-regime model predicts that gains should come from moving out of flat prompting, not necessarily from the NL medium — this prediction is untested.

Self-evolution module underspecified. The "self-evolution" module that mines execution traces to tighten solve loops is described at a high level but lacks enough detail to evaluate or replicate. This matters because self-improvement is the most ambitious claim (the system improves itself), and it's the least evidenced component.

Single benchmark pair. SWE-bench Verified and OSWorld are both established, but both are code-and-system-task benchmarks. Whether NLAH's findings transfer to other agent domains (research, writing, data analysis) is untested. The concentrated-frontier-effects pattern may be specific to the difficulty distribution of these benchmarks.

No independent replication. First publication of the framework by its creators. The ablation methodology is commendable, but independent teams running the same ablations on different systems would substantially strengthen the claims.

ABC comparison gap. The paper does not cite or compare with the ABC framework (Agent Behavioral Contracts), which addresses the same problem space (constraining agent behavior through contracts) from the formal-specification direction. A direct comparison would illuminate the tradeoffs between NL-interpreted and formally-specified contracts — a question the KB is already tracking.

Write a note titled "Explicit structure helps only when it constrains the dimension load-bearing for the task" connecting to process-structure-and-output-structure-are-independent-levers, specification-level-separation-recovers-scoping-before-it-recovers-error-correction, and methodology-enforcement-is-constraining. It would argue that the three notes address overlapping but unnamed aspects of when structure helps — process vs. output dimensions, scoping-before-error-correction ordering, and the enforcement maturation gradient — and that NLAH's empirical evidence unifies them: structure helps when it constrains the dimension load-bearing for the task's acceptance criterion, with the benefit ordering being scoping > process alignment > error correction. This is the synthesis opportunity flagged by /connect.