AIDE2 synthesis
Type: agentic-system-analyses/types/agentic-system-synthesis.md
Bounded synthesis
Rewrite the research process, not the model
AIDE2's strongest supported contribution is reported improvement of an entire research-agent harness through evaluation-guided code rewriting. The authors report seven accepted rewrites in an eight-day, 100-node trajectory, raising the incumbent private grade from 0.703 to 0.778. Separate held-out comparisons extend the performance assessment beyond the selection benchmark. These are attributed whole-harness results, not independently inspected execution or component-level causal effects (RT-CLM-harness-improvement, EPI-RTE-heldout-comparison). Whether this process also produces a more efficient self-improver remains unresolved.
The evidence is SRC-1, the frozen text capture of Recursive self-improvement of AI research agents, arXiv:2609.26457v1, captured as arxiv-2609.26457v1-2026-10-14, with applicability cutoff 2026-10-14. The boundary classifies the target as a builder or improvement plane: the paper is complete, but the executable loop is only partially exposed. Its Source register supplies doctrine/design and reported operation, not implementation, raw observed runs or independently inspectable causal experiments. Model providers, benchmark implementations and the wider production product are outside this account. Report and profile verification found no remaining blockers or additional verification limits; that acceptance does not upgrade the source's evidence strength (report verification, profile verification).
The editable object is the harness surrounding a model: search policy, context assembly, prompts and verification machinery. The principal experiment begins with AIDE0 as inner agent and AIDEhuman as outer driver. Gemini 3 Flash supplies inner model calls and Claude Opus 4.7 supplies outer calls; the paper reports holding each loop's model fixed, without immutable provider revisions. This is not a model-weight training account (RT-OBJ-agent-code, RT-CMP-gemini3-flash, RT-CMP-claude-opus4).
Produce solutions before deciding which harness to keep
The experiment supplies an initial harness, an outer agent, selection tasks, dollar budgets and a trajectory length. It grades the initial harness, then requests incumbent rewrites. The outer driver receives prior proposed agents and grades; its reviewer explores evaluation artifacts over several steps. Each proposal is an agent codebase, not a task solution or an admission decision. The workflow account establishes this representative documented progression, but not a trace of any individual proposal.
To grade a harness, the infrastructure runs it on task codebases. Draft explores ideas, debug repairs errors and improve refines promising candidates. Execution produces public task feedback. A one-call inner reviewer extracts a score and relevant feedback; the task evaluator, not the reviewer model alone, supplies the outcome reference. The editable harness chooses the next candidate and the solution returned at the task budget. AIDE0 selects parents greedily. AIDE85 instead selects among five strategy arms using UCB1, with 30% softmax sampling over arm best scores; it expands the best node under the chosen arm and forks the global best under another arm every five steps (RT-RTE-inner-search, EPI-RTE-reviewer-extraction, MEM-OBJ-strategy-state).
Private held-out scoring consumes each returned task solution. Averaging across tasks and independent repeated runs yields the harness grade. A fixed outer selector keeps the best graded harness; the proposer cannot substitute its own approval. About a quarter of graded rejected rewrites reportedly improved the public signal but failed the private comparison. Rejection preserves the incumbent as the next edit base. Buggy proposals receive no grade. At the trajectory limit, delivery means returning the highest-grade harness, not installing a production service (RT-RTE-harness-admission, workflow annotations).
This is acceptance for a specific operational use—task optimization at the selection budgets—against a specific criterion—better aggregate private grade. It does not endorse every explanation associated with the code. Noise can admit a genuinely worse successor and redirect later search. Separating public optimization feedback from private scoring reduces direct inner optimization of the selection criterion by design; the outer driver still sees private grades. No information-flow audit verifies hidden-data isolation (RT-CLM-private-signal, EPI-OBJ-evaluation-scores, RT-RTE-harness-admission).
Reuse bounded experience and executable organization
Within task search, AIDE85 replaces the full prior code/output history in draft and improve prompts with compact root/recent summaries. A separate failure route activates when the candidate bug rate reaches at least 15%: it supplies up to three recurring, deduplicated final error lines from recent buggy candidates. Summaries and signatures are advisory inputs to later model calls, not enforced repairs or accepted causal diagnoses. Recent selection removes detail from delivered context without establishing deletion from storage (MEM-OBJ-candidate-summary, MEM-RTE-summary-delivery, MEM-OBJ-error-signatures, MEM-RTE-failure-injection).
The authors report smaller, roughly constant prompts through character-based reconstruction and no context-window terminations for AIDE85, AIDE47 or AIDEhuman in the evaluated runs. That supports a bounded context outcome, not universal window safety. Summary creation control and exact contents are unspecified; automatic delivery alone cannot establish automatic summary production or its trace-learning classification. By contrast, automatic signature extraction, reviewer-derived selection feedback and evaluation-guided harness rewriting support claimed trace-fed adaptation routes. None independently establishes improved future capacity from a particular memory component (MEM-CLM-prompt-compression, MEM-RTE-summary-delivery, MEM-RTE-failure-injection, EPI-RTE-reviewer-extraction, profile verification). Physical storage, complete role-specific context policies and reload remain unresolved.
Across outer rounds, accepted code carries greater force than advisory history: it becomes the next edit base and determines later search/context choices when executed. Embedded prompt instructions retain model-interpreted force. The capture does not expose whether the code preserves its historical reasons or whether later consumers read those reasons (RT-BAP-incumbent-code, RT-BAP-robustness-prompt).
AIDE85 also adds a robustness reminder, re-prompts when generated task code is shorter than 40 characters, and patches one held-out scorer so a single failing case does not crash all cases. These have distinct consequences: advisory guidance, a length-based retry and evaluator failure containment. The length guard does not admit outer harness successors or certify correctness. A proposed ranking penalty reportedly never changes the selected candidate. The scorer patch demonstrates reach into evaluation machinery, not general permission limits or private-data access (RT-RTE-robustness-controls). In held-out protocols, secondary constraints or context errors can terminate search and trigger scoring of the best available candidate; this is fallback delivery, not resumption (RT-RTE-terminal-fallback).
What improved, and what remains unestablished
For budgeted task research, the reported whole-harness gains and bounded-prompt outcomes support evaluation-guided changes to future task organization. A separate kernel study reports reward-hacking incidence falling from 55% to 32% across 38 kernel/training-loop pairs. Its criterion requires isolated speedup above 1.02× plus either loss of more than half that speedup in training or a runtime crash there. This is a scoped behavioral comparison, not freedom from gaming, and it identifies no responsible rewrite (EPI-CLM-reward-hacking, EPI-RTE-heldout-comparison).
Learning as improved capacity for future action is author-reported at the full-harness level, independently of the narrower trace-fed-write classification. Self-improvement is likewise supported as a claimed design and reported pathway within the improvement-plane boundary: evaluation evidence shapes revisions; accepted revisions change its inner-agent organization; retained code enters later control; subsequent evaluations reportedly exercise changed agents. The absent inspectable evidence-to-update-to-consumer-to-run chain prevents independently established exercised attribution. Successful improvement has separate author-reported outcome support; neither component causality nor improvement of the unchanged principal outer driver follows (RT-CLM-harness-improvement, RT-BAP-incumbent-code).
Theory-building is a separate unresolved assessment. Condition 1, a localized formulated theory, cannot be established from absent full code and rationale. Condition 2, decisions depending on what that theory says, is not established by code reuse alone. Condition 3, criticism identifying a challenged claim, result and placement of blame, is not established by numerical selection. Condition 4, that criticism's result shaping another round, is not established merely by retained proposals and grades. These are evidential limits, not findings that theories or criticism are absent. Whether criticism of a consumed theory improved future capacity therefore remains unestablished. Reflection's self-representation and two causal directions, and autonomous theory-building's complete computational role coverage, are also not established by the supplied runtime account. Automatic proposal/admission is supported locally, not a complete autonomy verdict (RT-CLM-harness-improvement, RT-RTE-harness-admission, epistemic bounded conclusion).
For the stronger scenario of improving the improvement process itself, the ignition test supplies a partial result: evolved AIDE47 reportedly produces accepted changes as outer driver. But three 50-step runs per arm yield endpoint means of 0.780 versus 0.782 for AIDEhuman, with substantial variance. The comparison does not establish superior self-improver efficiency or acceleration (RT-RTE-ignition-test). Larger repeated comparisons could change that assessment. Component interventions could identify which context or robustness changes matter. Complete harnesses, prompts, evaluator code and raw lineage-linked runs could turn the documented progression into an independently inspectable execution account; they would not by themselves establish generalization beyond the tested domains.
Limitations
| Limitation | Affected SRC IDs and record citations | Inspected boundary | Conclusion prevented | Evidence that would resolve it |
|---|---|---|---|---|
| Documented rather than inspectable execution | SRC-1; RT-RTE-inner-search, RT-RTE-harness-admission, RT-CLM-harness-improvement | Frozen paper §§2–3 and reported protocols; no code checkout or raw runs | Verified runtime enforcement, complete handoffs, independently established exercised self-improvement, deployment guarantees | Full harness/evaluator implementation and raw runs linking evidence, revisions, loading and later behavioral dependence |
| Private isolation, evaluator semantics and noisy admission | SRC-1; RT-CLM-private-signal, RT-RTE-robustness-controls, RT-RTE-harness-admission | Described public/private split, scorer repair and noise warning | Guaranteed hidden-data isolation, scorer integrity, true superiority of every accepted successor or automatic recovery from false acceptance | Access contracts, scorer/patch code, audit traces, repeated regrading and documented correction policy |
| Incomplete context-production and lifecycle account | SRC-1; MEM-RTE-summary-delivery, MEM-RTE-failure-injection, MEM-OBJ-review-feedback, MEM-OBJ-strategy-state, MEM-CLM-prompt-compression | AIDE85 §3.5 and appendix E; other agent/role policies only partially exposed | Exact summary agency or fidelity, automatic summary trace learning, full selection budgets, signature expiry, context isolation, storage/reload or complete alternative-route coverage | Production rules, complete prompts and payloads, rate/window definitions, state schemas and persistence/withdrawal traces for all included alternatives |
| Outer history and rationale retention unknown | SRC-1; RT-OBJ-agent-code, RT-RTE-harness-admission, RT-BAP-incumbent-code | Proposal/grade reuse and incumbent selection in §2.1 and appendix D | Complete archive coverage, rejected/no-grade visibility, restart/rollback semantics or later consumption of design reasons | Proposal archive, rationale payloads, read-back records and checkpoint/reload contracts |
| Outcome contrasts do not isolate components | SRC-1; RT-CLM-harness-improvement, MEM-CLM-prompt-compression, EPI-RTE-heldout-comparison, EPI-CLM-reward-hacking | Reported full-harness comparisons and separate kernel criterion | Attributing gains to summaries, signatures, reminders or particular rewrites; universal robustness or transfer | Independently varied component comparisons with raw outcomes; further task/domain evaluations for broader claims |
| Theory, reflection and autonomous theory-building evidence incomplete | SRC-1; RT-CLM-harness-improvement, RT-RTE-harness-admission | Harness selection and reuse, without full method content or criticism traces | Establishing conditions 1–4, criticism-led learning, reflective causal organization or complete autonomous theory-building | Formulated method content, consumer-dependence evidence, claim/result/blame criticism and next-round retention traces; role-by-role and self-representation evidence |
| Self-improver efficiency comparison inconclusive | SRC-1; RT-RTE-ignition-test | Three seeds per driver, 50 steps per run | Superior sample efficiency or accelerating recursive improvement | Larger repeated driver comparisons measuring improvement per evaluation budget and retained changes across generations |
| Early termination and window safety are protocol-bounded | SRC-1; RT-RTE-terminal-fallback, MEM-CLM-prompt-compression | Held-out constraints and reconstructed prompts in appendices A/E | Guaranteed primary-budget exhaustion, arbitrary context-window safety or resumable workers | Termination/recovery implementation, raw requests and broader stress runs under specified limits |
No unresolved cross-report conflict or additional report/profile verification limit remains. The rows preserve the accepted records' conclusion limits rather than infer absent behavior. No frozen-source read was needed for this synthesis.