h t t p s : / / p u b . s a k a n a . a i / m a s s /

Ingest: Multi-Agent Self-Supervision (MASS)

Type: types/ingest-report.md

Classification

This is the authors' web write-up of a research paper (arXiv 2610.12176, "Recursive Self-Improvement through Multi-Agent Self-Supervision"). It states a method, reports experiments and tables, and defers several results to figures and to the paper. It is classified as a scientific paper because its claims rest on reported experiments. The capture is the full web page, but the figures were not captured as data. Results shown only in Figures 3-8 are known here only from the prose that describes them. Author: Hyunin Lee (UC Berkeley, work done during a Sakana AI internship), with co-authors including Matei Zaharia and Yujin Tang according to the BibTeX. The work comes from a credible industry and academic lab. The results are author-run and have not been reproduced independently.

Summary

MASS studies homogeneous recursive self-improvement on non-verifiable research tasks: one model serves as the task-solving team, the workflow optimizer, and a pairwise evaluator. In an inner loop with frozen weights, the optimizer revises a written workflow (roles, instructions, output contracts, and call order or "hops"). A candidate replaces the champion workflow only when the model's own pairwise judgment prefers its workspace. In an outer loop, the champion runs 18 times per task, the model ranks the results with a Bradley-Terry model, the workflow text is removed from the orchestrator prompt, and the model is LoRA-tuned on up to 15 top trajectories per task. Two rounds on Qwen3.6-27B over 12 synthetic finance, robotics and pharmacy tasks raise external-judge win rates against the base model to 69.9% on three held-out tasks. The external judges are GPT-5.5 and Claude Opus 4.8. Score per output token reaches 1.2-1.6x the base model on four research benchmarks but falls to 0.94-0.96x on SWE-bench Verified and Terminal-Bench 2.0. The write-up adds four further claims, all inside this one decomposition. The untrained optimizer and evaluator also improve, with evaluator agreement with the external judges rising from 73% to 93%. Evolved workflows concentrate task information in contracts and hops. Multi-agent trajectories give more win rate per supervised token than single-agent ones. Upgrading the optimizer matters more than upgrading only the evaluator. The authors close by naming shared blind spots in a homogeneous loop as a safety risk.

Contribution assessment (our opinion)

The problem matters: as tasks move past what cheap verifiers or human experts can check, self-improvement needs supervision that does not come from a stronger outside judge. MASS's contribution is a concrete, complete loop. It runs workflow search and self-judged distillation in one model, and it measures progress against withheld external judges. Several of its diagnostics are more useful than the headline gain. These are the workflow-transfer control (Table 2), which separates a better workflow from a better executor; the evaluator-versus-optimizer swap; and the matched-token comparison of multi-agent and single-agent trajectories. Each of these tests attacks a rival explanation of why the loop improves. The support is still narrow. There is one base model, 12 synthetic tasks, three held-out tasks, two rounds, and LLM judges as the outside measure. The public benchmark gains are reported per output token, which mixes quality with brevity. The transfer pattern, gains on research-like benchmarks and slight losses on coding and terminal benchmarks, is consistent with the loop learning a task family, not general capability. Originality is hard to judge from this capture. Workflow search (ADAS, AFlow) and trajectory self-distillation both have precursors, and the combination with a fully homogeneous judge seems less explored, but this ingest has not surveyed the literature. As a tentative judgment, the demonstrated value is a useful, partly controlled case study, and the safety framing about correlated self-judgment is its most durable idea. The claim that the loop is "sustainable" is not yet supported beyond two rounds.

Quotes

No source quotes have been retained yet.

Connections Found

MASS's main role in this KB is a correlated-oracle case of a full self-improvement loop. It bears on the error-correction cluster. Error correction works with above-chance oracles and decorrelated checks predicts the risk MASS's own safety section names: the same model generates and judges, so a shared blind spot is not filtered. Rising agreement with external judges (73% to 93%) is a measured soft-oracle trajectory, not a decorrelation result. False-positive generation is filtered; false-positive acceptance becomes operative gets a weight-form instance: a flawed workspace accepted by the self-evaluator becomes training data for the next model, which then also judges. This is the note's "same oracle that made the original mistake" case, carried across rounds.

The source is also a qualifying counter-case to the boundary of automation is the boundary of verification. In MASS's role-swap ablation, a stronger external evaluator alone helped a little, while a stronger optimizer as well helped a lot. That result holds only in this decomposition: one base executor, 12 research tasks, and a pairwise judge over whole workspaces. It is evidence for open-ended improvement must allocate search before decisive evaluation is available, since candidate generation limited improvement when evaluation was held fixed and the search was noisy. It compares with a proposal-selection improvement loop requires search, evaluation, and operative retention on function allocation: the optimizer searches, the champion-replacing judge evaluates, and LoRA retains. One model fills all three roles, so retention changes the machinery that searches and evaluates next round. MASS's "recursive self-improvement" should be read through the KB's frame-relative self-improving system definition: here the objective is self-judged, and the outside measure is a pair of withheld LLM judges.

For orchestration, MASS is a reported instance of the distributed-parametric decomposition-policy row in agent orchestration occupies a multi-dimensional design space. Workflows found by search are trained into the orchestrator with the plan stripped, and the gains are measured without a supplied workflow. This is the acquisition path that the Mismanaged Geniuses Hypothesis proposes. Its finding that evolved workflows concentrate task information in contracts and hops is weak evidence for agent orchestration needs coordination guarantees, not just coordination channels. The evidence is weak because the optimizer's system prompt already directs search toward contracts and hops. For representational form, MASS is evidence for treat continual learning as representational-form coevolution. It moves learned organization from a readable artifact into weights and then deletes the readable form, the reverse of that note's bet.

Among sources, MASS compares most closely with Co-Harness on harness-then-weights alternation. Co-Harness admits verified trajectories on checkable math and keeps the harness; MASS admits self-ranked trajectories on non-verifiable tasks and strips the workflow. It compares with Terminal-RSI on the trajectory-admission oracle: both start from a 27B Qwen base, but Terminal-RSI admits protected-verifier passes where MASS admits self-judged winners. MASS's 0.96x on Terminal-Bench marks where the two diverge. It compares with Automated Design of Agentic Systems on outer-loop machinery: ADAS keeps a fixed meta-agent, while MASS retrains the model that serves as its meta-agent. It compares with Autoreason on incumbent retention under pairwise judgment. MASS uses the same model as judge, which is exactly the judge-hygiene variable Autoreason's fresh blind judges control.

Learning Claims (our opinion)

Mechanism on its own terms. MASS has two nested learning processes. The inner loop is text-space search: the optimizer reads the feedback history and rewrites a natural-language workflow, the team executes it, and the evaluator compares the result with the champion's workspace and explains its preference. The outer loop is weight learning: the top self-ranked champion trajectories become LoRA training data with the workflow text removed. The model is meant to learn how to organize the work from the task alone. The source calls this "self-supervision": the training signal comes only from the model's own pairwise judgments, whose correctness is unknown.

Mapping onto Commonplace concepts. The inner loop is a proposal-selection loop over a stated artifact. The outer loop is the KB's "criticism applied through weights" boundary case: the workflow is a localized theory during search, but the retained result is in weights where no unit says anything. MASS therefore runs the KB's preferred form (readable, addressable workflow text) as a temporary scaffold and treats weights as the persistent form. This puts the KB's readable-artifact bet in question only weakly. Table 2 shows that round-1 workflows still help the base model (64% win rate when the same base model executes both), so the readable artifact kept transferable value that MASS chose to discard. The source does not test keeping the workflow alongside the trained model.

Theory-builder conditions, judged separately. - Localized content: met for the inner loop. Workflows are written plans with separately named roles, instructions, contracts and hops. Not met for the outer loop's retained result. - Consumption: met. The workflow is added to the task prompt and directs how the team divides and checks work. The example leakage contract shows a unit whose content gates later agents. - Content-directed criticism: partially met, weak evidence. The evaluator states its reasons, but it criticizes completed workspaces, not workflow units. Whether the optimizer's revisions aim at what particular contracts or hops say, or behave closer to score-guided variation, is not shown in this capture. The mutual-information analysis describes where task information ends up, not why a unit was changed. - Iteration: met within a run. The champion and the feedback history persist across iterations and shape the next proposal. - Persistence: stated results persist only within a run, per task. Across rounds, results persist only in weights, with the stated workflow removed. By the definition, the theory builder stops at each freeze, and the cross-round improvement is weight adaptation.

Learning judgment. The reported improvement is observed, not established as general. Win rates rise against the base under external LLM judges, and score per token rises on similar benchmarks. Causal attribution is partial. Table 2 separates workflow quality from executor quality for round 1. The matched-token comparison gives some support to multi-agent trajectories as better training data. But those trajectories were also preferred by external judges in 85.1% of comparisons and were 2.4x longer, so "multi-agent structure" and "higher-quality data" are not separated. Role generalization (the evaluator and optimizer improve without being trained in those roles) is an observed correlation across rounds. The capture does not show a control that tells it apart from general capability gains on this task family.

Juxtaposition with Commonplace. Both attack improvement on tasks without a cheap verifier, and make opposite bets on each axis this KB treats as decisive. Retained form: MASS uses the readable workflow as a scaffold and keeps weights; Commonplace keeps the readable artifact and holds weights fixed, because a localized unit is what later criticism can find and fix. Table 2 shows the discarded workflow still carried value. Judge: MASS is homogeneous, one model generates, judges and is retrained on its own verdicts; Commonplace requires decorrelated checks, so MASS is the configuration the error-correction note predicts fails silently. Acceptance: MASS accepts on pairwise preference; the KB's epistemic stance accepts a change when it survives a test it could have failed, and MASS has no refutation step. Unit of criticism: MASS criticizes whole workspaces; a theory builder needs content-directed criticism of named units, which MASS meets within a run and loses at each freeze. Human role: both fix the outer frame by hand, but Commonplace keeps humans in the acceptance path for judgment-heavy mutations, where MASS moves the evaluator inside the model. MASS is ahead on two counts. It is a complete running loop with gains against a withheld judge, where automating KB learning remains open, and its executor-fixed transfer control is a design this KB should borrow. Its ablation also challenges the KB's verifier-first priority within its scope. The open question cuts the other way too: Commonplace has no comparable measurement that its readable loop improves outcomes.

Fixed decomposition. The effective update space is bounded by choices fixed outside the loop: the three-role split, the four-component workflow schema, an optimizer prompt that emphasizes contracts and hops, whole-workspace pairwise comparison, Bradley-Terry ranking, and a 12-task synthetic suite. Under learning inside a fixed decomposition inherits its mistakes, the contracts-and-hops finding is improvement inside an instructed schema, not evidence that contracts and hops are the right decomposition of workflows. The shared-blind-spot risk is the same limit at the evaluator. If the shared model cannot distinguish a flawed workspace from a good one, no number of rounds can repair that distinction from inside the loop. The rise in external-judge agreement cannot rule this out, because the external judges are themselves LLMs and may share some blind spots with the trained model.

Extractable Value

  1. Optimizer-bottleneck ablation as a counter-case to verification-first framing -- the role swap held the executor fixed and varied which roles got a stronger model. A stronger evaluator alone helped a little; adding a stronger optimizer helped a lot. Among ingested sources, only Harness-updating is not harness-benefit runs a comparable swap, and there the proposer's identity barely mattered; MASS is the case in which the proposer's strength dominated. It qualifies the automation-boundary note's claim that the bottleneck is "typically oracle construction", but only within this decomposition: one executor, 12 research tasks, and a whole-workspace pairwise judge. [quick-win]
  2. Correlated self-judgment compounds across rounds, not just within a vote -- MASS's safety section describes a flawed result accepted by the self-evaluator, trained into the next model, and then judged by that model. This is a worked example for the decorrelated-checks and false-positive-acceptance notes: correlation in a training loop can reinforce an error round after round instead of only capping a vote's accuracy. The mechanism is argued, not demonstrated. [quick-win]
  3. Workflow-transfer control as a reusable experimental design -- Table 2 has the base model execute round-1 workflows (64% win over base-search workflows). This separates "the harness got better" from "the model got better" in a harness-then-weights loop. Co-Harness lacks this control, and the representational-form coevolution note says Co-Harness "does not isolate coevolution". The control applies to any alternating harness/weights claim. [experiment]
  4. Admission-oracle contrast across MASS, Co-Harness and Terminal-RSI -- all three alternate trajectory generation and weight training on similar bases, but each admits trajectories differently: self-judged, harness-validated, or protected verifier. MASS's gains on research-like benchmarks and slight losses on SWE-bench and Terminal-Bench are one data point for a conjecture that the admission oracle bounds how far such loops transfer. This is a candidate synthesis note, not an established claim. [deep-dive]
  5. Distilling the plan away as a form choice -- MASS removes the workflow from training prompts so the organization moves into weights. It is a concrete distributed-parametric acquisition path for the orchestration design-space row that currently rests on a proposal. It is also a direct contrast with the KB's preference for keeping readable artifacts. [quick-win]
  6. Contracts as where task-specific coordination lives -- the leakage validation_report example and the information analysis suggest that output contracts, not role wording, carry task-specific knowledge in evolved workflows. This is useful vocabulary for the coordination-guarantees note. It is weak evidence, because the optimizer's prompt steered search toward contracts and hops. [just-a-reference]
  7. Text-space search needs a stabilizer -- the authors report noisy workflow search and separate reaching a good workflow from improving on it stably, suggesting an analogue of a learning-rate schedule. This is an operational warning for any LLM-driven artifact search loop. It is context-bound and rests on a figure this capture does not include. [just-a-reference]

Limitations (our opinion)

  • The outside measure is also LLM judgment. Every quality claim rests on GPT-5.5 and Claude Opus 4.8 as judges, and so does the "evaluator converges" claim. This decorrelates from the trained model only partly. Shared biases across frontier LLMs (length, presentation, report polish) could produce the same rising win rates and agreement without a matching gain in research correctness. No human or task-grounded check is reported in this capture. This is the gap that error correction works with above-chance oracles and decorrelated checks predicts matters most.
  • Small and synthetic. There is one base model, 12 synthetic tasks (three held out), and two rounds. "Sustainable" RSI is asserted from two rounds of evidence.
  • The benchmark metric mixes quality with brevity. Public benchmark gains are reported as score per output token. A model that writes less at the same score improves on this metric. Raw scores are not given in the capture. The 0.94-0.96x results on SWE-bench Verified and Terminal-Bench suggest task-family specialization rather than general improvement.
  • Confounded mechanism claims. The contracts-and-hops finding follows an optimizer prompt that emphasizes contracts and hops. The multi-agent-versus-single-agent comparison also varies trajectory quality (85.1% external preference) and length (2.4x). Role generalization lacks a control that separates it from general task-family gains. Each result is attributable only to the compound configuration that produced it.
  • No baseline against other self-improvement loops. The capture compares MASS rounds with the base model and with single-agent distillation. It does not compare against verifier-filtered distillation, other workflow-search methods, or keeping the workflow in the prompt after training.
  • Capture boundary. Figures 3-8 and the paper's detailed results are not in the snapshot. Claims resting on them, including the optimizer-bottleneck ablation, the workflow-information analysis, and noisy search, are known only through the authors' prose summary.

Update the boundary of automation is the boundary of verification with a scoped qualification. Add MASS's optimizer-versus-evaluator ablation as a counter-case where candidate generation, not oracle quality, was the binding constraint. State its limits beside it: one executor, 12 research tasks, a pairwise LLM judge, and results reported only in a figure.

Relevant Notes: