LHTB synthesis
Type: agentic-system-analyses/types/agentic-system-synthesis.md
Bounded synthesis
From task selection to terminal effects
LHTB is an evaluation workflow for model-driven terminal work, not just a task dataset or a general problem-solving model. It supplies tasks and a modified Harbor runtime that launches agents, maintains environments, grades work and retains outcomes. This account covers that shipped workflow at Git commit 7ca0cacea9bf6a0694eaad7306bed794eaaaabce, with applicability cutoff 2026-10-15. The sole registered source, SRC-1, supplies implementation, doctrine/design and attributed results. No target execution, independently inspected run trace or causal experiment was supplied. Thus the account establishes wiring and affordances, not exercised reliability or component benefit. External model services, images, installed runtimes and sandbox deployments remain outside the captured implementation evidence. See the boundary and evidence and Source register.
An operator selects task roots, agent, model, environment and budgets. The full-suite example selects Terminus-2, openai/gpt-4.1, Docker and both task generations. Harbor expands configurations into independent trials; a trial starts and health-checks its environment and delegates the task instruction and workspace access to the selected agent. These concurrent trials are not evidence of a knowledge-sharing agent team. Harbor owns scheduling, grading and finalization; the model chooses proposed task actions. RT-RTE-trial-execution.
Terminus-2 constructs a prompt from task text and terminal state. Its model returns analysis, plans, command keystrokes and completion intent; parser and tmux code execute commands and return bounded observations. Admission of parsed commands is automatic, with no per-command human veto in this inspected loop. The action boundary is the environment under its effective user, not a restricted migration or scientific-computing API. A 60-second bound on a command's wait field does not bound the lifetime of subprocesses it launches. Installed agents and dynamically imported classes are material alternate paths: the inspected Codex adapter bypasses its CLI approval and sandbox controls, so Terminus-specific checks are not universal gates. RT-RTE-terminal-actions, RT-RTE-agent-dispatch.
Keeping work available while context changes
Workspace persistence and conversation persistence are separate. In the bundled default continuation path, another run() retains the environment but creates a fresh Terminus-2 Chat. Selecting HB_CONTINUE_MODE=same_conversation instead requires an agent resume method and retains Chat and tmux. Consequently a later phase can inspect earlier files without necessarily retaining the reasoning that produced them. RT-RTE-verifier-continuation.
Within a conversation, output filtering keeps first and last portions while omitting the interior. Low estimated free-token space can trigger a three-call handoff: a model summarizes remaining history; a reduced-context call asks questions; a fuller-history call answers them. Code installs selected prose into the foreground conversation and resets its provider response chain. This is an automatic trace-fed write with a later solver consumer, not merely accumulated turns. It supplies historical findings and continuation guidance, but no semantic gate checks fidelity or retention of reasons. Reactive overflow first removes recent turn pairs. Short and ultimate fallbacks submit task/screen prompts through the remaining unwound Chat; they do not necessarily install the successful handoff's complete replacement frame. RT-RTE-context-handoff, MEM-OBJ-handoff-context, memory qualifications.
Trajectory files preserve a separate inspection surface; their references are not automatically dereferenced by the foreground solver to restore omitted history. Nor does a provider response-chain reset establish deletion of remote state. The Responses transport can rely on opaque server-side conversation state, and model aliases do not pin immutable weights. MEM-OBJ-trajectory, RT-CMP-terminus-model.
Checking the product rather than trusting completion
The model's completion signal ends an agent phase, not necessarily the trial. When continuation and a timeout are configured, Harbor can invoke task-owned verification and use numeric reward >= 1.0 to stop on full pass. Rejection can trigger more work while budget permits. Same-conversation ordinary interim grading requires confirmed completion; other stop reasons break that loop. Timed process-reward mode additionally requires compatible same-conversation support and an ephemeral-log environment, and measures active agent segments. This is a conditional continuation protocol, not unconditional work until a hard wall-clock deadline. Verification-disabled and multi-step branches also prevent treating it as the only execution path. RT-RTE-verifier-continuation, RT-RTE-trial-execution.
The task supplies the answer oracle; Harbor executes it and parses reward files. Checking, score conversion and stopping have different meanings. Fractional credit need not mean the same fraction of substantive requirements was satisfied. A passing score licenses stopping under the task's criterion, not acceptance of a general explanation. The inspected acoustics reconstruction task checks finite numerical reference outputs with tolerances; its public feedback can pass on signature equality even when numerical payloads differ. Neither public pass nor hidden agreement establishes recovery of a physical mechanism or generalization across a real tool's domain. RT-RTE-verifier-execution, EPI-RTE-public-feedback, EPI-RTE-hidden-semantics, EPI-RTE-dense-reward.
The representative workflow is LangChain version migration: inspect and revise a project, submit declared artifacts, reconstruct in a separate verifier environment, check fixtures and behavior, and repair after rejection. The solver chooses the action sequence; this is not Harbor's distinct multi-step-task branch. Grading reinstalls from submitted source and metadata and deletes stale replay outputs before recomputing them. Agent-side installation or a copied replay file alone therefore cannot satisfy that reconstruction. Sequential gates stop on first failure, so partial credit is not a diagnosis of every remaining defect. Artifact-transfer failures warn, and the launcher separately rejects missing required paths. No inspected run establishes that migration or repair succeeded. WF-RTE-migration-grading, workflow account.
Separate verification narrows the handoff to configured artifacts plus the convention artifact channel; it does not copy the solver's entire environment. Shared verification instead uploads tests into that environment and relies on process freezing and cleanup. In the bundled implementation those hiding operations are best effort, and mounted logs can remain visible. Fixed same-conversation rejection avoids diagnostic text in that message, but default restarted feedback can read diagnostics when available; cleanup can remove their source first. A separately shipped replacement module has a binary default and fails closed on freezing failure, but its existence does not establish the identity of an installed runtime. Binary prompt construction, environmental secrecy and benchmark comparability are therefore separate claims. RT-CLM-grader-isolation, RT-RTE-runtime-patching, report verification.
Delivery and later use of retained evidence
Interim full pass does not eliminate the outer final-verification path. Timeout or nonzero agent exit can still lead to grading of the current workspace. Delivery at Harbor's boundary means a returned and saved TrialResult after attempted collection and cleanup; a delivered result can record failure, and artifact collection is not a completeness certificate. Job restart reads saved configurations and completed results to select remaining work, but does not restore an interrupted model conversation. RT-OBJ-trial-result, RT-RTE-trial-execution, RT-RTE-job-recovery.
Retention has consumers beyond job recovery. Viewer/export/upload routes expose or copy trial artifacts. A separate analysis model can inspect trial and task files, produce schema-constrained explanations, save them and reuse them later. Job aggregation receives selected summaries and check explanations rather than raw evidence through file tools. Automatic schema admission checks structure, not truth; cache existence can skip reanalysis without checking whether evidence changed. These assessments are advisory in the inspected paths: they do not withdraw rewards or automatically revise the solver. The separate task-quality checker likewise supplies rubric judgments rather than a launch veto. MEM-RTE-artifact-access, MEM-RTE-trial-analysis, MEM-CMP-analysis-model, EPI-RTE-analysis-cache, EPI-RTE-job-aggregation, EPI-RTE-task-check.
What adaptation this establishes
The strongest supported learning-related contribution is two wired trace-fed derivation chains: interaction history becomes retained context for later solving, and recorded trial evidence becomes retained interpretation for later aggregation. Both qualify as trace learning in the comparison's write-route sense. Neither establishes improved capacity for future action, faithful interpretation or parameter learning. The latter improvement question has no inspected comparison, and no demonstrated route adopts post-run analysis into subsequent task-solving capacity. MEM-OBJ-handoff-context, MEM-RTE-trial-analysis, MEM-BAP-analysis-guidance, profile verification.
Theory-builder conditions remain independently unestablished in the inspected solver/context routes: (1) no particular formulated theory is evidenced merely by requesting analysis and planning; (2) delivery of guidance does not establish decisions depending on that theory's content; (3) binary rejection does not name a challenged claim and assign blame; and (4) repeated repair does not establish retention and use of a criticism's result in a next theory round. These are evidential limits, not claims that opaque models or remaining tasks cannot perform those operations. Criticism of a consumed theory improving future capacity is likewise not established. RT-RTE-terminal-actions, RT-RTE-context-handoff, RT-RTE-verifier-continuation.
Autonomy and reflection require separate conclusions. The inspected loop automates action admission and bounded continuation after launch; it does not thereby establish an autonomous theory builder. Retained state and advisory self-diagnosis do not establish the two causal directions needed for reflection at the whole workflow boundary. Runtime maintenance affords replacement of installed machinery for later imports, but no trace shows evidence determining an update, its adoption and subsequent operation causally depending on it. Thus exercised self-improvement of LHTB is unestablished; workspace repair is an external work-product change, while context replacement establishes its own narrower behavior-shaping route. None of these qualifications negates that route. EPI-RTE-trial-analysis, RT-RTE-runtime-patching, RT-BAP-runtime-module.
For the scenario of evaluating persistent terminal work against task-owned tests, the supported architectural result is a separation of solver intent, executable effects, reference checks and recorded outcomes. For scenarios requiring universal grader secrecy, faithful long-context memory, scientific mechanism recovery or improved later solving capacity, the present evidence is insufficient. Mode-identified deployment traces could establish exercised continuation and isolation; independently varied comparisons could change the component-benefit assessment. Summary-to-source checks could establish bounded fidelity. An evidence-linked maintenance/adoption/subsequent-run trace could change the self-improvement assessment. These additions must retain their task, runtime and comparison boundaries rather than upgrade the whole system from one successful witness.
Limitations
| Limitation | Affected SRC IDs and record citations | Inspected boundary | Conclusion prevented | Evidence that would resolve it |
|---|---|---|---|---|
| No independently inspected execution or causal comparison | SRC-1; RT-RTE-verifier-continuation, RT-RTE-context-handoff, EPI-CLM-benchmark-warrant | Frozen implementation and attributed results; no target execution | Actual repair, improved future capacity, reliable activation and component benefit | Mode-identified execution artifacts and comparisons independently varying the component at issue |
| Deployment, model and installed-runtime identity remain open | SRC-1; RT-CMP-terminus-model, MEM-CMP-analysis-model, RT-RTE-agent-dispatch, RT-CLM-grader-isolation, RT-RTE-runtime-patching | Bundled runtime, one installed-agent forcing case and replacement recipe; external images/services and actual installations uncaptured | Universal containment, provider deletion, immutable model identity, parity of installed and bundled behavior, complete alternate-agent memory policy | Captured deployment/runtime identities, provider contracts and inspected relevant agent/state implementations |
| Handoff fidelity and theory-directed learning are unestablished | SRC-1; MEM-OBJ-handoff-context, RT-RTE-context-handoff, RT-RTE-terminal-actions | Source-grounded context transformation and solver requests; opaque model processing | Guaranteed retention of reasons, faithful replacement, theory-builder conditions or improved capacity from criticism | Source-aligned handoff/decision traces, explicit theory and criticism content, retained next-round use and capacity evidence |
| Finite task checks do not establish all-task or scientific validity | SRC-1; EPI-RTE-public-feedback, EPI-RTE-hidden-semantics, WF-RTE-migration-grading | Acoustics evaluator and selective migration-grader inspection; other task internals and images uninspected | Physical-mechanism recovery, broad generalization, exhaustive oracle robustness or uniform criteria across tasks | Independent oracle/fixture validation and scoped audits or comparisons for the claimed domains |
| Retained analysis has no semantic acceptance or automatic solver-adoption evidence | SRC-1; MEM-RTE-trial-analysis, EPI-RTE-analysis-cache, EPI-RTE-job-aggregation | Assessment, structural admission, cache and aggregate callers | Truth/freshness certification, accepted ampliative explanation, reward withdrawal or improved subsequent solving | Evidence-consuming semantic decisions, evidence-sensitive invalidation and a traced downstream adoption consumer |
| Viewer task-definition chat is omitted from the analyst inventory | SRC-1; MEM-RTE-artifact-access, MEM-RTE-trial-analysis, epistemic inventory, report-verification limit | Verification inspected viewer dispatch and task-keyed SDK chat session reuse/reset with task context and read tools | Exhaustive inventory of model components, viewer conversational contexts and advisory consumers; not evidence of solver reuse or automatic reply adoption | Accepted records covering this additional operator-facing model/context route and its consumers |
| Sweep reward schema does not establish effective success pruning | SRC-1; MEM-RTE-sweep-selection, report-verification limit | Selector reads singular reward; ordinary producer declares plural rewards |
Successful outcome-driven pruning of current standard trial outputs | A compatible producer/consumer contract and traced selection on those outputs |
| Complete trajectory delivery and published-data lineage remain uninspected | SRC-1; MEM-OBJ-trajectory, MEM-OBJ-process-reward, MEM-RTE-artifact-access, report-verification limit | Primary timed-checkpoint copy, separate continuation traversal and episode selection; not every export/subagent format | Complete audit-history retention/delivery, export fidelity or checkpoint-to-published-CSV lineage | Full reference/payload traversal and normalization evidence, plus an inspected checkpoint-to-CSV derivation |
| Some retained objects and remote exports have no established later consumer | SRC-1; MEM-RTE-checkpoint-retention, EPI-RTE-task-check, MEM-RTE-artifact-access | Hidden checkpoint storage, optional quality-check output and viewer/export/upload boundary | Additional checkpoint reload, saved-check reliance, downstream training or cross-trial solver-learning routes | Identified consumer implementation and delivery/adoption evidence for each retained payload |
| Runtime modification is afforded, not exercised self-improvement | SRC-1; RT-RTE-runtime-patching, RT-BAP-runtime-module | Maintenance recipe and replacement code, without deployment trace | Evidence-responsive operative self-change or successful improvement of LHTB | Objective-linked update determination, adoption, live authority path and subsequent operation causally dependent on that change; separate outcome evidence for benefit |