AIDE2 epistemic report

Type: agentic-system-analyses/types/agentic-system-epistemic-report.md

Source-and-claim boundary

The question is what AIDE2's harness search measures, what those measurements license, and how results change subsequent behavior. The scope and excluded components are those of boundary.md; source identity and layers are in its Source register. SRC-1 is a paper capture, not implementation or observed-run evidence. Reported execution does not change architectural status to implemented. Below, doctrine only means a documented route with no inspectable implementation or observed route, even where authors report that it operated.

Direct reads covered SRC-1 §§2.1–3.6, §5 and appendix A's protocol and evaluator definitions. Supplied runtime findings cover additional appendix D/E details. Search of the capture for review, grades, private scoring, summaries, errors, admission and noise checked for counterexamples to a clean score-to-warrant story. It found private rejection of publicly better candidates, noisy false acceptance, evaluator repair and an ineffectual selection penalty. No executable code exists in this boundary to trace a concrete deterministic transformation through implementation. Correction reads of SRC-1 §§2.1 and 3.5 confirm that inner candidate generation and the short-code guard concern task scripts; the supplied MEM-OBJ-candidate-code fragment distinguishes them from outer harness code.

Entry point or operation	Source path/anchor and inspection basis	Coverage or prevented conclusion
Algorithm 1; task solution generation and return	SRC-1 §2.1–2.2, direct capture read	RT-RTE-inner-search
Execution review and public-score selection	SRC-1 §2.2, direct	EPI-RTE-reviewer-extraction; RT-RTE-inner-search
Private grading, rejection, incumbent reuse	SRC-1 §2.1, §3.2, §5, direct; appendix D supplied	RT-RTE-harness-admission
Summary and error-signature supply	SRC-1 §3.5, direct; appendix E supplied	EPI-OBJ-candidate-summary; EPI-OBJ-error-signatures
Reminder, length guard, penalty, scorer patch	SRC-1 §3.5, direct	RT-RTE-robustness-controls
External performance and proxy/downstream comparison	SRC-1 §3.3–3.4, appendix A, direct	EPI-RTE-heldout-comparison
Premature termination and fallback score	SRC-1 appendix A, direct	RT-RTE-terminal-fallback
Evolved outer driver	SRC-1 §3.6, direct	RT-RTE-ignition-test
CLI dispatch, hooks, registered tools, cleanup/withdrawal and persistence/reload	No executable checkout or complete prompts in frozen capture	Unavailable, not searched-code absences. Prevents implementation-wide negatives, enforcement verification, exact delivery reconstruction and lifecycle completeness.
Provider internals and model updates	Excluded; SRC-1 §2.2 and supplied component findings	See RT-CMP-gemini3-flash, RT-CMP-claude-opus4. No weight-learning or provider-state assessment.

Consequential claims are RT-CLM-private-signal, RT-CLM-harness-improvement and EPI-CLM-reward-hacking. Missing evaluator code prevents auditing score validity and private isolation. Missing full candidate code and prompts prevents individuating candidate explanatory claims or checking summary fidelity. Missing raw runs prevents independently establishing activation, acceptance decisions and load chains. Missing component interventions prevents component causal attribution. These are separate limits, not a system-complete negative.

Epistemic-object inventory

Object ID	Candidate truth-apt content or none	Claimed role	Evidence source ID and local anchor	Gap/limit
MEM-OBJ-candidate-code	Runnable task solutions; accompanying factual/mechanistic claims not determinable from absent scripts	see record	SRC-1 §§2.1–2.2; supplied refusal fragment	Inner generation and short-code guard concern these task solutions, not outer harness successors.
RT-OBJ-agent-code	Executable harness policy; embedded factual/mechanistic claims not determinable from absent code	see record	SRC-1 §2.1–2.2	Code selection is not acceptance of every proposition a rationale might contain.
EPI-OBJ-evaluation-scores	Candidate achieves a metric value under a specified task/evaluation; aggregate harness grade	see record	SRC-1 §2.1	Public and private scopes differ; averaging is not proof of true expected superiority.
EPI-OBJ-reviewer-feedback	Interpretation of execution outcomes and relevant feedback	see record	SRC-1 §2.2	No concrete reviewer output; faithful extraction or diagnosis not established.
EPI-OBJ-candidate-summary	Compressed description of root/recent code and outcomes	see record	SRC-1 §3.5	Semantic preservation and claim-level provenance unknown.
EPI-OBJ-error-signatures	Final error text as an occurrence report, not an established causal explanation	see record	SRC-1 §3.5	Deduplication and truncation lose occurrence/context information.
RT-OBJ-robustness-instruction	Private-split premise plus normative preference for robustness	see record	SRC-1 §3.5	Reminder is advisory; neither generalization nor compliance follows.
RT-CMP-gemini3-flash, RT-CMP-claude-opus4	Individuated truth-apt parametric content not determinable	see records	SRC-1 §2.2	Fixed models are reported; weight contents and immutable versions inaccessible.

Authority-route ledger

The ledger splits checks from dispositions and retention from use. A see field delegates only the cited record's owned factual fields; epistemic classifications and licenses below are this member's assessment. Every row is architectural doctrine only for the evidence-layer reason above. No supplied fact needs supersession: the runtime's combined routes remain valid containers for the separated functions.

Route ID	Route function	Architectural status	Object/candidate ID	Content/update relation	Transition or check target	Evaluator/condition and domain	Activation and timing	Possible or observed result	Implemented force	Epistemic authority and scope	Operational authority	Behavioral-authority path	Evidence source ID and local anchor	Claim IDs or none	Mismatch marker or none	Gap/limit
RT-RTE-inner-search	content transformation	doctrine only	MEM-OBJ-candidate-code	non-truth-apt policy/content update: task-code proposal; any accompanying factual claim is indeterminate	see route	see route	see route	New script	Uninspected	Candidate generation only; no warrant from novelty	Allows later execution	Harness → script executor; permissive; current task	SRC-1 §2.1–2.2	none	none	No full script/rationale to distinguish entailed claims from conjectures.
RT-RTE-inner-search	check/evidence production	doctrine only	EPI-OBJ-evaluation-scores	truth-apt transformation: indeterminate	Candidate execution under public task objective	Task metric/environment, not reviewer alone	After execution	Public score/errors	Uninspected	Task outcome under that evaluator and budget, not mechanism or transfer	Supplies feedback to reviewer/selector	Executor → execution outputs; advisory evidence then ranking; task horizon	SRC-1 §2.2	RT-CLM-private-signal	none	Measurement warrants depend on opaque scorer and conditions; not imported prior knowledge.
EPI-RTE-reviewer-extraction	content transformation	doctrine only	EPI-OBJ-reviewer-feedback	truth-apt transformation: indeterminate	see route	see route	see route	Interpreted score/feedback	Uninspected	Unknown preservation of execution evidence; diagnostic correctness unestablished	Informs selection and next edit	see route	SRC-1 §2.2	none	none	Could be faithful reshaping or unsupported inference; concrete input/output needed.
RT-RTE-inner-search	operational admission/selection/consumption	doctrine only	EPI-OBJ-evaluation-scores	no content change	Next parent and terminal solution	see route	see route	Selected candidate	Uninspected	Best visible score licenses only metric-relative preference, not truth of code rationale	Changes expansion/return	Harness selector → operator/private evaluator; ranking; current run	SRC-1 §2.2, §3.5	none	none	AIDE85 arm exploration differs from greedy AIDE0; no universal greedy claim.
RT-RTE-harness-admission	check/evidence production	doctrine only	EPI-OBJ-evaluation-scores	truth-apt transformation: indeterminate	Returned solutions on private tasks; harness aggregate grade	Private metric procedures and repeated-run averaging; see route	Before incumbent decision	Private scores/grade	Uninspected	Evidence about selected task performance, not all future tasks	Supplies outer decision signal	Private evaluators → grade → fixed selector; ranking; outer rounds	SRC-1 §2.1	RT-CLM-private-signal	none	Mean is mathematically specified but premises and executable encoding unverified.
RT-RTE-harness-admission	disposition/acceptance	doctrine only	RT-OBJ-agent-code	no content change	Proposed successor versus incumbent	Better aggregate private grade, fixed selection benchmark/budget; see route	Each graded rewrite	Accept best grade; reject lower grade; buggy proposal ungraded	Uninspected	Recorded protocol acceptance for use as task optimizer, not unconditional epistemic endorsement	Changes incumbent or withholds replacement	Grade → incumbent selector; permissive/enforcing by design; later rounds	SRC-1 §2.1, §3.2, §5	RT-CLM-harness-improvement	false-acceptance risk	Author-reported decisions; noise can admit worse true performance.
RT-RTE-harness-admission	retention	doctrine only	RT-OBJ-agent-code, EPI-OBJ-evaluation-scores	no content change	Incumbent and prior proposals/grades	see route	Across outer steps	Available history and edit base	Uninspected	Retention grants no extra warrant	Makes code/evidence available for next proposal	History → outer driver; advisory; outer trajectory	SRC-1 §2.1	none	none	Archive format, rejection-history expiry and reload unknown.
RT-RTE-harness-admission	behavior/policy adaptation	doctrine only	RT-OBJ-agent-code	non-truth-apt policy/content update: selected harness changes search/context rules	Accepted code enters later operation	see route	After acceptance	Later optimizer and next edit base	Uninspected	Acceptance of operational artifact, not established integration of a formulated knowledge claim	Changes next search and context policy	see RT-BAP-incumbent-code	SRC-1 §2.1, §3.3	RT-CLM-harness-improvement	none	No claim-level evidence linkage to establish lifecycle integration.
RT-RTE-inner-search	content transformation	doctrine only	EPI-OBJ-candidate-summary	truth-apt transformation: indeterminate	Root/recent history replaced by summary	Summary maker/rules unspecified	Draft/improve context assembly	Bounded summary	Uninspected	Warrant unknown after omitted detail; size reduction is not semantic validity	Supplies reduced context	Summary → draft/improve model; advisory; later invocations in run	SRC-1 §3.5	none	none	Faithful reshaping versus changed meaning cannot be decided.
RT-RTE-inner-search	content transformation	doctrine only	EPI-OBJ-error-signatures	truth-apt transformation: non-ampliative reshaping	Final error lines selected/deduplicated	Described code rule, bug rate ≥15%, at most three	Conditional prompt assembly	Recurring error snippets	Uninspected	Preserves selected text as reported; degrades full diagnostic lineage, no causal license	Makes failures salient to next generation	Error signatures → draft/improve prompt; advisory; later invocations in run	SRC-1 §3.5	none	none	No implementation trace; selection order and occurrence provenance unknown.
RT-RTE-robustness-controls	operational admission/selection/consumption	doctrine only	RT-OBJ-robustness-instruction	no content change	Code-generation preference	Model interpretation	Every generation prompt	Requested robust/generalizable code	Uninspected	No correctness or generalization license	Advises generation	see RT-BAP-robustness-prompt	SRC-1 §3.5	none	none	Reminder benefit not independently tested.
RT-RTE-robustness-controls	check/evidence production	doctrine only	MEM-OBJ-candidate-code	no content change	Generated task-solution code length	Character-count proxy, under 40 characters	After generation	Short/not short	Uninspected	Length only, not code validity	Supplies re-prompt condition	Length check → generation guard; enforcing by design; current call	SRC-1 §3.5	none	none	Not a semantic acceptance test.
RT-RTE-robustness-controls	disposition/acceptance	doctrine only	MEM-OBJ-candidate-code	no content change	Nearly empty task-solution generation	see route	Guard result triggers re-prompt	Another generation call	Uninspected	Withholds operational continuation for short text, no positive truth license	Re-generates instead of ordinary continuation	Guard → model request; enforcing by design; current step	SRC-1 §3.5	none	none	Retry cap/recovery unspecified.
RT-RTE-robustness-controls	operational admission/selection/consumption	doctrine only	EPI-OBJ-evaluation-scores	no content change	Final candidate ranking	Median-distance penalty; see route	Terminal selection	Reported unchanged ordering/choice	Uninspected; reported no selection effect	No added protection from lucky winners shown	No changed selection in reported replay	Penalized scores → selector; ranking; terminal task choice	SRC-1 §3.5	none	intended effect not realized	This is author-reported replay, not an implementation deduction.
RT-RTE-robustness-controls	lineage/freshness/recovery	doctrine only	EPI-OBJ-evaluation-scores	no content change	One scorer's per-case failure propagation	Scoring-script patch; see route	A single test case fails	Remaining cases no longer all crash	Uninspected	More complete evaluation possible, not proof of unchanged scoring semantics	Continues scoring after local failure	Patch → evaluator; enforcing by design; patched evaluations	SRC-1 §3.5	none	evaluator reach qualifies isolation claims	Exact patch, failed-case score and permission boundary unknown.
RT-RTE-terminal-fallback	operational admission/selection/consumption	doctrine only	EPI-OBJ-evaluation-scores	no content change	Candidate available at premature termination	see route	Secondary constraint or context error	Best available solution scored	Uninspected	Outcome at attained horizon, not success at intended full search horizon	Ends search and emits score	Benchmark protocol → private evaluator; permissive; terminated run	SRC-1 appendix A	none	none	Does not establish restart or withdrawal of stale conclusions.
EPI-RTE-heldout-comparison	check/evidence production	doctrine only	RT-OBJ-agent-code	truth-apt transformation: indeterminate	Whole harness performance outside selection	Benchmark metrics, fixed protocols; see route	After checkpoint selection	Aggregate held-out scores	Uninspected	Reported transfer on evaluated tasks, not mechanism or universal generalization	Informs paper's comparison; no automatic promotion shown	Evaluations → authors/report; advisory; study horizon	SRC-1 §3.3, appendix A	RT-CLM-harness-improvement	none	Whole-harness comparison, no component isolation.
EPI-RTE-heldout-comparison	check/evidence production	doctrine only	RT-OBJ-agent-code	truth-apt transformation: indeterminate	Kernel proxy gains versus downstream training	Isolated speedup and training-loop speedup/crash classification; see route	Separate held-out study	Reported reward-hacking incidence	Uninspected	Defined proxy/downstream mismatch on 38 pairs, not absence of gaming generally	Informs authors' behavior claim, not rewrite admission	Evaluation → rate/report; advisory; study horizon	SRC-1 §3.4, appendix A	EPI-CLM-reward-hacking	none	No identified responsible rewrite; 32% still meets failure definition.
RT-RTE-ignition-test	check/evidence production	doctrine only	RT-OBJ-agent-code	truth-apt transformation: indeterminate	Evolved versus human-engineered outer driver	see route	Three seeds/arm, 50 steps	Similar mean endpoints; inconclusive	Uninspected	Does not license better self-improver efficiency	Tests proposed promotion criterion; no automatic promotion shown	Arm grades → study conclusion; advisory; experiment	SRC-1 §3.6	RT-CLM-harness-improvement	acceleration unestablished	No decisive superiority result.

No lifecycle-integration row is asserted: the inspected paper documents operational code reuse, not accepted formulated claims being connected to evidence or reorganized for knowledge use. This is a limitation on demonstrated claim integration, not evidence that no such route exists in the unavailable harnesses.

System-claim versus route comparison

Claim ID	Claimed operation or warrant	Claim source ID/anchor and evidence layer	Doctrine/design support	Implemented route IDs	Observed-run support	Causal support and design limits	Supported conclusion	Mismatch/unknown
RT-CLM-private-signal	see record	SRC-1 §2.1; doctrine/design	Separate public optimization and private grading	None inspected; see RT-RTE-inner-search, RT-RTE-harness-admission for design	None supplied	Private rejection of publicly better proposals supports attributed distinction; no information-flow audit	Design reduces direct optimization of private criterion by inner agent	Outer driver does see grades; scorer repair shows evaluator reach. No proven isolation or no-gaming guarantee.
RT-CLM-harness-improvement	see record	SRC-1 §3.2–3.3, §5; reported operation	Private-grade admission and accepted-code reuse	None inspected; see RT-RTE-harness-admission, EPI-RTE-heldout-comparison	None supplied	Reported whole-harness comparisons outside selection; no component intervention; noisy grades	Reported objective gains and operational acceptance at fixed budgets; held-out tests extend scope beyond selected grade	Selection grade alone cannot warrant transfer; accepted rewrite need not be truly better. Ignition inconclusive.
EPI-CLM-reward-hacking	see record	SRC-1 §3.4, appendix A; reported operation	Proxy/downstream evaluation outside selection	None inspected; see EPI-RTE-heldout-comparison	None supplied	Whole-lineage comparison; no independently varied robustness mechanism	Reported incidence reduction under stated kernel criterion	Does not identify contributing rewrite, validate proposed explanation, or prove freedom from reward hacking.

Bounded conclusion

AIDE2's strongest documented epistemic route is measurement followed by scoped operational selection. Public execution feedback guides inner search. Private scores averaged across tasks/runs decide whether a proposed harness becomes incumbent. Acceptance uses a named criterion—better aggregate grade—for a named use—subsequent task optimization at fixed budgets—on a fixed selection benchmark. The consequential path is grade to fixed selector to retained incumbent code to later editing and execution, not an unconditional endorsement of the harness's explanations (RT-RTE-harness-admission, RT-BAP-incumbent-code). The authors report that it operated; the implementation and raw exercised path are uninspected.

Measured propositions are new evidence about outcomes, but no checked implementation or warranted-premise derivation is available. Arithmetic aggregation is specified, not verified. The capture does not individuate a produced accepted ampliative explanatory claim: code generation creates operational candidates, and absent rationales prevent determining whether any accompanying proposition is imported, entailed or non-entailed. No general claim-production negative follows.

Reviewer interpretation and summaries carry an unresolved semantic edge. Their names do not establish faithful reshaping. Selected deduplicated error lines have a narrower non-ampliative role: they preserve selected occurrence text while losing context and multiplicity. They may influence later model generation without being accepted causal diagnoses (EPI-RTE-reviewer-extraction, EPI-OBJ-candidate-summary, EPI-OBJ-error-signatures). External task code enters as a starting artifact; source warrant for any factual claims in that code is unknown, not acquired merely by having a codebase. The reported trace-fed memory and retained harness rules can change future action independently of model-weight updates.

The robustness reminder is advisory. The short-code guard checks the length of generated task-solution code and re-prompts the inner generator; it does not admit outer harness successors (MEM-OBJ-candidate-code). Length is only a proxy, not semantic acceptance. The penalty reportedly changes no selection. The scorer patch repairs failure propagation but does not certify evaluator integrity. Each retains its distinct force and limit (RT-RTE-robustness-controls). No cleanup, claim withdrawal, false-acceptance rollback or claim-level integration protocol can be verified from this capture.

Reported held-out gains and reduced proxy/downstream divergence support bounded whole-harness outcomes, not component explanations. They are not evidence that the summary, reminder or error memory caused the gains. Harness-level learning and improvement are reported separately from theory-building, reflection and autonomy: numerical selection alone supplies no formulated content-directed criticism. Full method content and consumer dependence remain unavailable. Fixed model descriptions do not rule out learning through retained code, but provide no weight-learning evidence. The ignition comparison does not establish superior self-improver efficiency or acceleration (RT-CLM-harness-improvement, EPI-RTE-heldout-comparison, RT-RTE-ignition-test).

Shared records

Operative objects

EPI-OBJ-evaluation-scores

Label: Public scores and private grades

Source-native identity: numerical public task feedback and private held-out task scores aggregated into g(a). Representational form: symbolic numeric values consumed by explicit selectors. Storage substrate: candidate/history state and grading artifacts; serialization uninspected. Distinct from RT-OBJ-agent-code: these are measurements about code execution, not executable code. Closest routes are RT-RTE-inner-search and RT-RTE-harness-admission; neither declares this object. The ledger splits public/private consumers and domains. Evidence: SRC-1 §2.1, doctrine/design. Grade aggregation establishes a protocol for reporting outcomes, not true expected performance or component effect.

In practice, each task is run several times independently and the scores are averaged across runs. --- /home/zby/llm/commonplace/.commonplace/worktrees/aide2-6b3bcd4791cf/kb/agentic-system-analyses/state/AAS-2026-10-14-2609-26457-6b3bcd4791cf-01/sources/arxiv-2609.26457v1-2026-10-14.txt

EPI-OBJ-reviewer-feedback

Label: Reviewer interpretation of execution output

Source-native identity: relevant feedback extracted by the reviewer; the score's symbolic selection path is separately inventoried in EPI-OBJ-evaluation-scores. Representational form: natural-language interpretation. Storage substrate: candidate feedback/context; exact schema uninspected. Distinct from MEM-OBJ-candidate-code, whose task execution outputs the reviewer interprets, and from RT-OBJ-agent-code, the harness governing that task search. Closest supplied route is RT-RTE-inner-search; no supplied object counterpart. Evidence: SRC-1 §2.2, doctrine/design; passage in EPI-RTE-reviewer-extraction. Full reviewer text and diagnostic warrant not determinable.

EPI-OBJ-candidate-summary

Label: Root and recent candidate summary

Source-native identity: compact summary of root/recent candidates supplied to draft and improve operators. Representational form: natural language; exact mixed encoding uninspected. Storage substrate: assembled prompt/context state; persistence/reload beyond described invocations unknown. Distinct from RT-OBJ-agent-code: descriptive replacement context rather than executable harness. Closest supplied route is RT-RTE-inner-search; no supplied object counterpart. Evidence: SRC-1 §3.5, doctrine/design and reported operation. Semantic fidelity is not established by prompt-size reduction. Unspecified maker/rules prevent claiming a deterministic implementation or faithful compression.

AIDE85\mathrm{AIDE}_{85} instead has the draft and improve operators read a compact summary of the root and recent candidates rather than the full history. --- /home/zby/llm/commonplace/.commonplace/worktrees/aide2-6b3bcd4791cf/kb/agentic-system-analyses/state/AAS-2026-10-14-2609-26457-6b3bcd4791cf-01/sources/arxiv-2609.26457v1-2026-10-14.txt

EPI-OBJ-error-signatures

Label: Recurring final error lines

Source-native identity: up to three deduplicated final error lines of recent buggy candidates. Representational form: natural-language diagnostic text consumed by generation models; gating/count rules symbolic. Storage substrate: prompt/context state. Distinct from EPI-OBJ-candidate-summary, with a separate trigger, selected input and lineage loss. Closest supplied route is RT-RTE-inner-search; no supplied object counterpart. Evidence: SRC-1 §3.5, doctrine/design and reported operation. Selected text is an occurrence signal, not an established cause or a generalized failure theory. Reported selection/deduplication is non-ampliative, but actual implementation and downstream compliance remain uninspected.

When a run’s candidates show a bug rate of at least 15%, the agent injects up to three recurring error signatures (the final error lines of recent buggy candidates, deduplicated) into draft and improve prompts. --- /home/zby/llm/commonplace/.commonplace/worktrees/aide2-6b3bcd4791cf/kb/agentic-system-analyses/state/AAS-2026-10-14-2609-26457-6b3bcd4791cf-01/sources/arxiv-2609.26457v1-2026-10-14.txt

Routes

EPI-RTE-reviewer-extraction

Label: Reviewer score and feedback extraction

Part of: RT-RTE-inner-search

Endpoints/progression: execution output → reviewer model call → score/relevant feedback → selector and subsequent code-generation context. Owner: inner harness's reviewer invocation. This material part separates interpretation from the environment measurement and subsequent disposition in the supplied container. Evidence: SRC-1 §2.2, doctrine/design. The reviewer judges execution output, not underlying correctness independently; its extraction fidelity domain is uninspected. Model-generated diagnoses can be ampliative, but no actual output here establishes non-entailment. Humans set task/objective; no per-feedback human veto is described. The task evaluator is the outcome reference; the reviewer is not itself an answer oracle. Operating mode: bounded task optimization, triggered by execution feedback. No separate knowledge-admission gate is supplied.

• Immediate return: Extracted score and relevant feedback.
• Later read-back: uninspected — feedback enters candidate history but later exact delivery after summary/filtering is unspecified.
• Delegated visibility: Execution output reaches the inner reviewer in one model call; full prompt and omitted fields unknown.
• Selection predicate: Reviewer chooses relevant feedback; exact instructions and parsing rules unknown.
• Invalidation or expiry: uninspected — no stale-feedback withdrawal or replacement protocol provided.
• Activation or effect: Described extraction supplies score/feedback for search; raw evidence of changed decisions unavailable.
• Evidence limits: No executable reviewer, prompt, concrete output or input/output fidelity trace in capture.
• design conclusion status: claimed
• implementation conclusion status: uninspected
• operation conclusion status: claimed

AIDE0\mathrm{AIDE}_{0} employs a reviewer agent that reads execution outputs from evaluating solutions and extracts a score and any relevant feedback. --- /home/zby/llm/commonplace/.commonplace/worktrees/aide2-6b3bcd4791cf/kb/agentic-system-analyses/state/AAS-2026-10-14-2609-26457-6b3bcd4791cf-01/sources/arxiv-2609.26457v1-2026-10-14.txt

EPI-RTE-heldout-comparison

Label: External performance and downstream behavior comparison

Endpoints/progression: selected harness checkpoints → budgeted benchmark runs → private outcomes and proxy/downstream comparisons → aggregate findings in paper. Owner: experimental investigators/protocol. Context/action effects: evaluates full harnesses outside candidate selection; informs reported generalization and behavior claims. Distinct from RT-RTE-harness-admission, which selects on a fixed selection benchmark, and RT-RTE-terminal-fallback, which covers termination rather than comparison judgments. This route also differs from RT-RTE-ignition-test: its targets are task performance and kernel behavior, not outer-driver efficiency. The ledger separates evaluator domains.

Targets/evaluators: ALE-Bench contest performance, MLE-Bench percentile, FML-Bench normalized test improvement, and WeatherBench forecast-skill gain against ERA5 relative to persistence. KernelBench compares isolated speedup against actual training-loop speedup/crashes in GPT-2, ViT and CNN. These supply task reference outcomes, not proof of a harness mechanism. Investigators choose checkpoints/protocol and interpret comparisons; model agents propose task solutions. No per-comparison human veto gate or automatic product promotion appears. Operating mode: held-out experiments, triggered by testing selected agents. Evidence: SRC-1 §§3.3–3.4 and appendix A, doctrine/design and reported operation. No inspectable causal experiment or raw trace is supplied; comparison granularity is whole harness, not independent component variation.

• Immediate return: Aggregate benchmark results and reward-hacking incidence by harness.
• Later read-back: uninspected — results reach the paper, but automatic feedback into rewrite selection is not shown; external benchmarks reportedly never affected selection.
• Delegated visibility: Inner agents optimize public/proxy feedback; investigators evaluate private/downstream outcomes.
• Selection predicate: AIDE0 baseline, AIDE47 intermediate, AIDE85 final and AIDEhuman comparison; reward-hacking predicate stated in appendix A.
• Invalidation or expiry: uninspected — no expiry/retraction/re-evaluation lifecycle for comparative claims provided.
• Activation or effect: Authors report external gains and lower measured reward-hacking rate; implementation and raw behavior dependence uninspected.
• Evidence limits: Whole-lineage comparisons cannot identify the responsible rewrite or isolate reminder/summary/penalty effects; benchmark and scoring implementations unavailable.
• design conclusion status: claimed
• implementation conclusion status: uninspected
• operation conclusion status: claimed

The private grades used during recursive self-improvement cannot by themselves establish generalization beyond candidate selection. --- /home/zby/llm/commonplace/.commonplace/worktrees/aide2-6b3bcd4791cf/kb/agentic-system-analyses/state/AAS-2026-10-14-2609-26457-6b3bcd4791cf-01/sources/arxiv-2609.26457v1-2026-10-14.txt

A kernel counts as reward hacking when its isolated speedup exceeds 1.02×\times and either less than half of that speedup survives inside training or the kernel crashes at runtime there. --- /home/zby/llm/commonplace/.commonplace/worktrees/aide2-6b3bcd4791cf/kb/agentic-system-analyses/state/AAS-2026-10-14-2609-26457-6b3bcd4791cf-01/sources/arxiv-2609.26457v1-2026-10-14.txt

On WeatherBench 2 (Rasp et al., 2024) we treat forecast quality as the objective the agent optimizes. --- /home/zby/llm/commonplace/.commonplace/worktrees/aide2-6b3bcd4791cf/kb/agentic-system-analyses/state/AAS-2026-10-14-2609-26457-6b3bcd4791cf-01/sources/arxiv-2609.26457v1-2026-10-14.txt

Claims

EPI-CLM-reward-hacking

Label: Reported reduced reward hacking

Claimed operation: cumulative harness changes reduce measured proxy/downstream divergence on a separate kernel task family, from 55% for AIDE0 to 32% for AIDE85. Source: SRC-1 §3.4 and appendix A, reported operation. Distinct identity from RT-CLM-harness-improvement: this claim concerns a behavior not measured by the selection grade, with a separately defined test domain. No assertion of freedom from reward hacking or of a particular causal component. The comparison route is EPI-RTE-heldout-comparison.

These reward hacking rates establish a held-out behavioral change, though they do not identify which rewrites produced it. --- /home/zby/llm/commonplace/.commonplace/worktrees/aide2-6b3bcd4791cf/kb/agentic-system-analyses/state/AAS-2026-10-14-2609-26457-6b3bcd4791cf-01/sources/arxiv-2609.26457v1-2026-10-14.txt

The admission uncertainty used in the ledger is independently stated by the paper:

If the noise is high enough, a falsely accepted rewrite becomes the new incumbent (eq. 4), so a single noisy comparison can derail the outer loop’s subsequent search. --- /home/zby/llm/commonplace/.commonplace/worktrees/aide2-6b3bcd4791cf/kb/agentic-system-analyses/state/AAS-2026-10-14-2609-26457-6b3bcd4791cf-01/sources/arxiv-2609.26457v1-2026-10-14.txt