Meta-Harness report verification

Type: agentic-system-analyses/types/agentic-system-verification.md

Verification

The current reports support publication of a bounded implementation account with the local limits below. I read every supplied member, its type, the record contract, correction answers, artifact check and previous verification. Memory supplies one corrected answer; the other answer files are empty. The artifact check reports none, leaving no structural failure to route. I independently confirmed the frozen checkout revision and clean status and inventoried its tracked tree. The source reads below establish SRC-1 implementation or doctrine/design evidence, not observed operation or causal benefit. I executed no target code, tests, examples or provider calls.

Correction and dependent findings

The previous blocker is repaired. MEM-OBJ-evolution-summary remains a valid container; MEM-OBJ-summary-results and MEM-OBJ-summary-rationale now separately declare its result/control and copied explanation parts. Independent reads of both drivers' summary constructors and resume parsers confirm automatic measurement/control-field writes, copying of model-authored hypotheses and terminal changes, and deterministic consumption of iteration counters. Both proposer skills request later interpretation of the history. These support the parts' separate lineage, admission and consumer authority, without granting software-derived warrant to hypotheses. SRC-1, reference_examples/text_classification/meta_harness.py, reference_examples/terminal_bench_2/meta_harness.py, reference_examples/text_classification/.claude/skills/meta-harness/SKILL.md, reference_examples/terminal_bench_2/.claude/skills/meta-harness-terminal-bench-2/SKILL.md.

I checked the corrected coverage, write-side, read-back and integration passages, and the current reconciliation. They consistently preserve the distinction. EPI-OBJ-val-results overlaps the numerical summary subset but explicitly excludes hypotheses and spans additional artifacts. It is not an exact replacement for either new part. The reconciliation correctly withdraws the earlier missing-part conflict without superseding the valid container. The transient RT-OBJ-pending-eval remains distinct from its later copied content.

Runtime and lens support

I traced classification proposal, import admission, benchmarking, summary writes and explicit finalization, including direct benchmark/inner-loop alternatives. Benchmark command construction and result loading confirm save/reload paths, reuse without candidate-digest invalidation, required seed groups and metric/Pareto selection. Inner-loop reads confirm online prediction-before-update, offline labelled batches, best-epoch restoration, checkpoint logging, concurrent evaluation and save-before-final-evaluation ordering. These support RT-RTE-classification-search, RT-RTE-classification-finalize, MEM-RTE-classification-memory, EPI-RTE-example-context, EPI-RTE-answer-check and EPI-RTE-frontier-selection. Evaluation omits explicit learning calls but cannot prevent candidate mutation inside prediction. SRC-1, reference_examples/text_classification/meta_harness.py, reference_examples/text_classification/benchmark.py, reference_examples/text_classification/inner_loop.py.

The few-shot, FewShotAll, NoMemory and interface reads support the live examples, generic checkpoint and partial log distinctions in MEM-OBJ-examples, MEM-OBJ-memory and MEM-OBJ-classification-log. All received labels are retained; per-input sampling is hash-seeded, not semantic matching; raw questions can replace full inputs; the character cap survives FewShotAll; NoMemory's restored state does not affect prediction. Evaluator reads support dataset-convention agreement and the AEGIS substring countercase, not target truth. SRC-1, reference_examples/text_classification/agents/fewshot_memory.py, reference_examples/text_classification/agents/fewshot_all.py, reference_examples/text_classification/agents/no_memory.py, reference_examples/text_classification/memory_system.py, reference_examples/text_classification/data/evaluators.py.

Provider key construction, response persistence and wrapper truncation support MEM-OBJ-completion-cache, MEM-RTE-completion-replay and classification observation filtering. Keys omit exact backend weights. The recorded original prompt does not certify delivery after truncation. SRC-1, reference_examples/text_classification/llm.py; see EPI-RTE-observation-filter.

I independently traced terminal import/subclass checks, optional smoke, discovered-trial scoring, frontier replacement, summary writes, baseline reuse, main-loop failure handling and optional final evaluation. The adapter's hard-subset and extra-flag paths are alternatives, not held-out tests. Smoke checks runtime compatibility without requiring positive reward; warnings do not complete denominators; the final branch discards success status before parsing an existing job directory. These support RT-RTE-terminal-search, EPI-RTE-terminal-scoring, MEM-OBJ-frontier-val and the workflow account. The workflow faithfully separates instructed child self-checks from the driver's independent veto and retained winner identity from deployment or frozen bytes. SRC-1, reference_examples/terminal_bench_2/meta_harness.py, reference_examples/terminal_bench_2/scripts/run_eval.sh, reference_examples/terminal_bench_2/.claude/skills/meta-harness-terminal-bench-2/SKILL.md.

KIRA command, image, native-call, overflow and episode-loop reads support RT-RTE-kira-execution, its memory annotation, EPI-OBJ-terminal-output and filtering findings. Whole marker-bearing lines disappear; overflow retries copy surviving messages before adding summary/fallback text; image inference receives selected multimodal inputs rather than full chat. The cache decorator marks the last three messages without excluding earlier messages. Inherited summarization and output limiting remain outside this source's inspected implementation. SRC-1, reference_examples/terminal_bench_2/agents/baseline_kira.py, reference_examples/terminal_bench_2/anthropic_caching.py, reference_examples/terminal_bench_2/prompt-templates/terminus-kira.txt.

Complete pilot outer-agent/controller reads and direct default-suite/local-verifier reads support RT-RTE-pilot-search, EPI-RTE-source-check, EPI-RTE-suite-scoring, EPI-RTE-history-selection, MEM-OBJ-evaluation-history and EPI-OBJ-evaluation-results. Lexical/AST admission, child equality rewards, projected feedback and historical-max scoring have different targets and consumers. Passing source can score zero; multiple configured attempts do not establish multi-result aggregation. Normal tool charging is narrower than compute spending and differs from verifier arithmetic. Neither mechanism authenticates history or binds final code to the best evaluated snapshot. SRC-1, experimental/harbor_meta_harness/agents/meta_harness.py, experimental/harbor_meta_harness/controller.py, experimental/harbor_meta_harness/suite.toml, experimental/harbor_meta_harness/tasks/select-harness/tests/test_outputs.py, experimental/harbor_meta_harness/tasks/budget-rollups/tests/test_outputs.py, experimental/harbor_meta_harness/tasks/dedupe-events/tests/test_outputs.py, experimental/harbor_meta_harness/tasks/reconcile-ledger/tests/test_outputs.py.

The identical reference wrappers' skill loading and CLI configuration support RT-BAP-proposer-instructions, not policy compliance or deployed isolation. Candidate import/staging supports RT-BAP-candidate-execution before frontier selection. Configurations and provider-call paths support all five runtime component records' distinctions between external parameters, configured aliases, unpinned exact weights and no implemented weight update. SRC-1, reference_examples/text_classification/claude_wrapper.py, reference_examples/text_classification/config.yaml, the terminal adapter and pilot controller cited above, and experimental/harbor_meta_harness/agents/terminal_harness.py.

Root documentation, onboarding and both skills support RT-CLM-fixed-model, RT-RTE-domain-onboarding, MEM-OBJ-reports and EPI-CLM-mechanism-tests. Requested reports and criticism remain affordances/design claims, not produced accepted explanations. The epistemic ledger's implemented checking, disposition, retention and adaptation do not upgrade operation status. Runtime separately qualifies learning, theory conditions, reflection and dispositional self-change; no successful self-improvement follows. SRC-1, README.md, ONBOARDING.md, and both proposer skills cited above.

Identity, relations and conflicts

I examined every declaration's referent, source anchor, retained load-bearing passage and evidence status; every Part of relation; annotations; supersession; and reconciliation disposition. No evidenced absence is declared. The sole supersession of EPI-OBJ-examples by MEM-OBJ-examples has identical list/serialization referents and construction evidence. Epistemic subroutes are contained within their runtime search parents; pilot history is a retained-output part of the pilot route; both new summary parts belong to the summary container. Cross-route memory and observation-filter records properly explain overlap rather than claiming a false single parent.

The reconciliation's checkpoint/example, manifest/result, cache/weight and whole-loop/subroute distinctions hold. Its KIRA fallback qualification does not contradict runtime: the original-instruction/screen description concerns fallback text, not the entire remaining context. Partial classification logs, terminal final-pass parsing and differing pilot budget owners also qualify rather than contradict the supplied accounts. No unresolved cross-report conflict remains. Coarse cross-artifact records retain heterogeneous consumers in prose/routes, subject to the local inventory limit below.

Consequential omission search and stopping

The tracked tree and producer/consumer dependencies guided three questions beyond checking listed anchors. Could alternate entry paths defeat universal isolation, holdout, budget or recovery claims? Direct classification commands, terminal adapter variants/baseline reuse and standalone controller preserve only the reports' already path-specific guarantees. Could shipped child variants or suite preparation add an overlooked knowledge-acceptance mechanism? Direct reads of collection preparation, pilot baseline, inspect/validate variant, terminal scaffold and editable seed found lexical admission metadata and model-facing policies, not an enforced explanation-acceptance route. SRC-1, experimental/harbor_meta_harness/scripts/prepare_collections.py, experimental/harbor_meta_harness/agents/baseline.py, experimental/harbor_meta_harness/agents/inspect_validate.py, experimental/harbor_meta_harness/agents/terminal_harness.py, experimental/harbor_meta_harness/tasks/select-harness/environment/harness.py. Finally, does per-answer evidence actually reach durable validation/test output? Following inner-loop output construction confirms the local discrepancy below.

Further omission expansion stopped because these material challenges have supported dispositions, with no remaining identified challenge to the central bounded conclusions. Full dataset inspection, additional test bodies and external runtime internals are not needed to retain positive wiring findings while withholding complete provenance, summary fidelity, empirical gain and system-complete negatives. Unsearched behavior is not evidenced absence. The pilot README's probes remain reported operation, not independently observed runs. SRC-1, experimental/harbor_meta_harness/README.md; see boundary.

Blockers

none

Limits

  • EPI-OBJ-predictions describes retention in prediction-result JSON, but ordinary validation/test CLI writers serialize _build_output, which contains aggregate metrics and metadata rather than per-example predictions. Prediction dictionaries exist in the returned evaluation structure; training predictions can enter MEM-OBJ-classification-log. SRC-1, reference_examples/text_classification/inner_loop.py. Withhold an inference that ordinary validation/test JSON durably retains per-answer evidence or permits replay of every comparator decision. This local storage overstatement does not change answer checking, aggregate persistence or selection, and no controlled comparison value asserts a complete prediction archive.

    json.dump(_build_output(val_result), f, indent=2) --- reference_examples/text_classification/inner_loop.py

  • MEM-OBJ-evaluation-history, EPI-OBJ-evaluation-results and EPI-OBJ-val-results retain coarse cross-artifact or mixed containers. Their prose and separate checking/consumer routes preserve supported distinctions without emitting a conflicting uniform status. Inventory comparisons at source-snapshot, admission-flag, metric-field and winner-identity level remain incomplete. Do not extract one common check, consumer, form or authority for all contents, or turn overlap into exact identity. Positive persistence, scoring and executable-authority conclusions remain supported.
  • RT-RTE-pilot-search and the epistemic coverage account do not separately inventory the shipped inspect/validate child's requested local-validation policy. SRC-1, experimental/harbor_meta_harness/agents/inspect_validate.py. This is model-facing guidance, not an enforced proof or supplied successful check. The partial inventory preserves controller/verifier conclusions, but comparisons of task-level self-check instructions across shipped pilot variants remain incomplete.

    Run a local validation command against the artifact before you stop. Do not claim completion until the required output file exists. --- experimental/harbor_meta_harness/agents/inspect_validate.py