Ingest: How Do Agents Fail on AutoResearch?

Type: kb/sources/types/ingest-report.md

Classification

An empirical arXiv preprint with a benchmark construction, 800 agent rollouts, an inductively developed 45-pattern failure taxonomy, human annotation, an artifact-aware agent-judge pipeline, calibration statistics, case studies, and released evaluation artifacts.

Author: the AutoResearchEval and ARFT research team, reporting first-party analysis of its own benchmark and judging system. Direct access to complete trajectories and the disclosed human-calibration protocol are useful credibility signals, but the retained snapshot omits the byline and affiliations, and the study is not an independent evaluation of the authors' framework.

Summary

The paper constructs 100 research tasks across seven scientific domains, including 70 open-ended tasks without an execution-feedback signal and 30 target-anchored tasks, then runs each task once under eight harness-model combinations for 800 full trajectories. Its AutoResearch Failure Taxonomy (ARFT) assigns 45 patterns across six lifecycle stages plus cross-stage dynamics and four proposed root-cause pillars. A human-calibrated agent judge inspects code, data, logs, and reports rather than only the final output; against human labels on 50 trajectories, this pipeline reaches 80.7% pattern recall and $\kappa=0.75$, versus 63.5% and $\kappa=0.53$ for a single transcript-only call. Across 12,712 attributed hits, the paper emphasizes unsupported claims, invalid proxy optimization, and especially failures that agents themselves noticed but did not repair: uncorrected self-awareness appears in 660 of 800 analyses. This is useful evidence for artifact-aware diagnosis and for making review findings govern delivery, but it does not establish that the shared failures are irreducibly model-level, because task structure, prompts, budgets, evidence partitions, judging machinery, and most orchestration choices remain fixed rather than experimentally varied.

Quotes

No source quotes have been retained yet.

Connections Found

This paper is an empirical anchor for Diagnostic richness constrains outer-loop learning quality and Reasoning production is not reasoning evaluation: a judge with access to the producing artifacts diagnosed more human-labelled failures than a transcript-only call, and polished endpoints often concealed routes that did not support them. The measured advantage belongs to the full artifact-aware judging bundle, not to artifact access alone.

Its 660 uncorrected-self-awareness cases are direct evidence for Knowledge storage does not imply contextual activation: the relevant flaw was already present in the live review context but did not change the report or next action. Frequent metric misalignment, circular validation, and grader-fitting likewise instantiate A proximate target is checked for achievement, not for warrant, because agents improved a measurable score without establishing the scientific relation the score was meant to certify.

Methodologically, the source is a technical basis and limitation for Trajectory-aware evaluation of transforming agent workflows. It shows that full artifacts can expose failures unavailable at the endpoint, while that proposal's blinded pairing, controlled trace mutations, repeat runs, and fixed adoption criteria address causal and calibration questions AutoResearchEval does not isolate.

Extractable Value

1. Artifact inspection changes the diagnostic evidence surface. The artifact-aware pipeline improved pattern recall by 17.2 percentage points over the single-call transcript baseline and exposed report-code, method-conclusion, and right-for-the-wrong-reason failures. This supplies measured support for diagnostic richness, provided the KB attributes the result to the combined evidence, navigation, rubric, execution, and quality-control treatment. [quick-win]

2. A review is effective only when its findings can block or revise delivery. In 82.5% of analyses, the agent identified a severe flaw but shipped without repairing it. Commonplace review designs should therefore distinguish producing a correct critique from closing the critique-to-action transition and should require either remediation, claim withdrawal, or an explicit unresolved-failure disposition. [quick-win]

3. ARFT is a useful audit vocabulary, not yet a calibrated prevalence prior. Its stage-by-root-cause matrix separates where a failure appears from why it occurs and gives operators concrete search terms for trajectory audits. Individual frequencies remain weaker evidence because agreement is reported only in aggregate, cognitively judged patterns are acknowledged as lower-confidence, and the judging rubric forces stage coverage and issue-count floors. [just-a-reference]

4. Proxy success must be audited against the method it stands for. The paper's cases of answer-key transcription, a precomputed answer behind a never-opened execution gate, circular validation, and metric substitution show why a passing endpoint needs a route-level warrant check. This extends the proximate-target connection from theory to research-agent artifacts. [experiment]

5. Shared failure across systems does not identify the failing layer. The eight tested pairs vary models mostly within one harness, use only one rollout per pair-task, and leave the six-stage task representation and evaluation pipeline fixed. A follow-up must vary review gates, revision authority, evidence representation, or lifecycle decomposition before deciding whether the limitation belongs to the model, harness, protocol, or their interaction. [deep-dive]

Limitations (our opinion)

The strongest results are configuration-bounded. Each agent could condition on the supplied premise and tension, its interaction history, tool outputs, task data, and its own code, data, decision file, report, and review. Its available operations included retrieval where enabled, shell and code execution, file creation and revision, analysis, and self-review. The effective hypothesis class was the frozen backbone's conditional policy over text and tool calls, enlarged by whatever programs it could author and execute during the episode; model weights were not updated. Fixed outside that update space were the mined task representation, the six-stage lifecycle, domain and task selection, open-ended versus target-anchored partition, harness interfaces, web policy, budgets, one-rollout protocol, and acceptance format. Differences and improvements inside this space do not validate those fixed choices, as Learning inside a fixed decomposition inherits its mistakes explains.

The model-level attribution is therefore stronger than the experiment supports. Six backbones run under Claude Code, while GPT-5-mini appears only with Codex and Gemini 3.5 Flash only with Gemini CLI, so model and harness are not fully crossed. Recurrence across all pairs shows a shared phenomenon in the tested compound configurations; it does not distinguish a backbone limitation from a shared prompt, task decomposition, budget, tool policy, or absent external acceptance gate. The authors explicitly leave orchestration interventions untested.

The artifact-aware comparison is also a bundled treatment. Relative to the transcript-only call, the agent judge receives full generated artifacts, can navigate and selectively execute code, follows a detailed stage rubric and nine iron rules, passes an automated coverage and anchor checker, and may be regenerated after gate failures. Its higher recall supports the whole pipeline. It does not isolate artifact access, agentic navigation, the rubric, execution, or regeneration as the cause. Likewise, the 70 open-ended and 30 target-anchored task sets differ in domain and novelty-move composition, so their results do not isolate the effect of external feedback.

The prevalence estimates depend on a judge whose validation is aggregate over 50 trajectories. No per-pattern or per-pillar agreement is reported, the same set is described as the calibration sample and the human-labelled validation set, and Cognitive Depth and Adaptability labels are acknowledged as harder to adjudicate. The production checker additionally requires broad stage coverage, minimum issue counts, and thicker rewrites after failed gates. Those constraints improve audit completeness but may also shape which failures are found and how often. The 12,712 hits and individual percentages should be treated as judge-mediated estimates, not direct observations with uniform uncertainty.

Finally, the task suite favors computationally runnable work with public data, excludes purely wet-lab experiments, and runs each pair-task once under fixed time and token budgets. It therefore samples an important but selective part of scientific research and cannot separate stable failure propensity from run stochasticity or resource pressure. No implementation repository, released trajectory corpus, or judge code was inspected for this ingest, so the report does not independently reproduce the benchmark, taxonomy counts, or calibration results.

Recommended Next Action

Update Trajectory-aware evaluation of transforming agent workflows with one AutoResearchEval evidence-and-boundary subsection that records the artifact-aware pipeline's recall gain, adds review-to-revision closure as a measured outcome, and treats artifact access as a bundled treatment until controlled evidence-surface ablations isolate it.