h t t p s : / / a r x i v . o r g / a b s / 2 6 0 8 . 1 6 6 4 7

Ingest: The Dual Nature of Generalization in On-Policy Distillation

Type: types/ingest-report.md

Classification

Scientific paper: Zhaoyi Li, Deyang Kong, Yuan Wei and colleagues report controlled post-training experiments, benchmark comparisons and distributional diagnostics. The retained observation is arXiv v2, dated 23 August 2026, by researchers affiliated with USTC, Peking University, IQuest Research, MBZUAI and Zhejiang University. The authors supply primary experimental evidence for their configurations; independent reproduction is not established here.

Summary

Every Coin Has Two Sides examines on-policy distillation (OPD), in which a teacher supplies token-level feedback on student-generated responses. With the policy-gradient distillation procedure held fixed, changing training-problem difficulty has little effect on final math accuracy in the reported comparisons, including when teachers solve none of four sampled responses. Transfer across language, reasoning horizon and task domain is generally stronger for teacher–student pairs sharing a concrete initialization lineage. In the tested shared student, math-only supervision can improve code or science performance, but can also reduce science accuracy toward a weaker teacher's level. Two-teacher experiments with fixed domain routing show that changing teacher and prompt proportions changes performance outside each teacher's assigned domain; routing supervision therefore does not isolate its consequences. These results support broader regression evaluation of weight updates. They do not establish the authors' stronger explanation that alignment of the whole policy causes the transfer, nor compare shared-weight distillation with alternative ways of separating expert capabilities.

Quotes

No source quotes have been retained yet.

Connections Found

The paper supplies a concrete parametric example for governing behaviour-changing writes: in the tested shared-weight OPD setup, validating only the routed prompt domain would miss consequential regressions elsewhere. This supports that note's regression-validation requirement, while post-training experiments do not test its deployment-time governance loop. The reversal of teacher assignments is useful because a teacher still transfers capabilities outside its nominal teaching role.

The evidence also illustrates the boundary of an experimental contrast. Mixture experiments challenge strict confinement of teacher influence, whereas comparisons among existing model lineages do not isolate origin from capacity, initialization and post-training history. In terms of representational form, the retained changes are distributed-parametric. A domain label chooses the supervision source; it does not create a separately editable record of that domain's knowledge.

Learning Claims (our opinion)

The mechanism is teacher-guided policy optimization. The student generates responses, and the teacher evaluates next-token probabilities on the student's visited prefixes. A sampled-token reverse-KL estimate reinforces a token when the teacher gives it higher probability than the student and suppresses it otherwise. The learned state is the student's weights; the teacher remains the reference distribution. Correct complete teacher answers are not required by this update rule. Similar gains from teacher-easy and teacher-hard question subsets show that useful supervision can remain when end-to-end success is absent from a small teacher sample. They do not identify which reasoning operations were transferred.

The benchmark gains are evidence of improved task capacity, with losses on other tasks making the judgment objective-dependent. They are learning without a theory builder. The retained change is the student's weights, where no unit says anything, so condition 1 fails. Token-level reverse-KL updates are gradient descent, not criticism of what a stated theory says, so condition 3 fails. Student reasoning traces could state, criticize, and revise conjectures within a response, which would be a builder at the lowest persistence grade, one reasoning episode; the paper does not examine the traces, so that stays unestablished, and it does not change the verdict on the weight update. Weight changes alone also do not establish separately revisable parts of an addressable theory.

The effective update space permits changing a shared token policy conditioned on prompts and generated history. The architecture, teacher distributions, training objective and available domains remain supplied by the experimenter; in the multi-teacher runs, domain routing remains fixed while teacher assignments or mixture proportions are varied. Consequently, the results show improvement and interference within this arrangement, not evidence that its decomposition is preferable to alternatives. As learning inside a fixed decomposition requires, those untested choices remain distinct from the successful weight updates. The paper adds a useful distinction to that analysis: a fixed input partition can route feedback without partitioning the effects of learning. It does not show that every alternative decomposition would suffer the same interference.

Extractable Value

  • [quick-win] Bound regression checks by affected behaviour. The shared-weight OPD experiments provide a concrete example for the governing-writes note: a math teacher can lower science accuracy despite receiving only math prompts, and swapping teacher assignments does not eliminate off-domain transfer. The transferable warning is that the nominal update domain does not establish the boundary of its effects. The evidence concerns this distillation procedure, not every update method or Commonplace's readable artifacts.
  • [experiment] Evaluate the full capability profile of a candidate teacher. In the tested fixed-routing mixtures, increasing a teacher's share can harm another domain; adding more math prompts from a cross-origin math teacher can even lower math scores when it displaces effective supervision from a same-origin teacher assigned science/IF. A future distillation evaluation should compare teacher–student compatibility and off-domain outcomes, rather than rank teachers only by standalone target-domain accuracy. Shared lineage is a useful observed grouping, not a causally isolated selection rule.
  • [just-a-reference] Separate useful local feedback from complete solution success. Three 25K math subsets selected by four teacher rollouts—always solved, never solved in that sample, and random—reach similar final accuracy with OPD otherwise held fixed. Matched approximately 8K pools of grade-school and very difficult questions also produce substantial gains, though generally below the diverse math mixture. This supports the availability of useful token supervision on sampled failures; it neither proves the teacher cannot solve those questions nor proves that difficulty never matters.

Limitations (our opinion)

Compatibility is not an identified cause. Same-origin means sharing a concrete initialization checkpoint through post-training, not merely belonging to one model family. The origin comparisons use existing models with different sizes, recipes and capability profiles. Increased top-16 next-token overlap is consistent with greater compatibility, but is a local distributional diagnostic, not a measurement of agreement over every possible context. Generic shared representations and interference remain sufficient candidate accounts of the observed transfer without establishing the authors' whole-policy explanation. Appendix C also reports substantial language and horizon transfer for the cross-origin Qwen3 pair, and a same-origin student exceeding its teacher on composed math: neither a universal same-origin requirement nor a strict teacher-performance ceiling follows.

The strongest routing conclusion is narrower than a clean causal decomposition. Within each multi-teacher setting, teacher share and prompt-domain share move together. The main role-swapping comparison additionally changes the student's initial state: Setting 1 uses Dev-1.5B after ten science/IF OPD steps, whereas Setting 2 starts from DS-distill-1.5B. Within-setting contrasts and supplementary base-student results support off-domain influence more precisely than attributing differences between those main settings solely to swapped roles. Benchmark-level mixture responses are not strictly monotonic. No comparison tests isolated parameter modules, adaptive routing or larger expert pools.

Precision and outcome scope are limited. Runs use at most 200 training steps, with 128 prompts and four student rollouts per prompt. Evaluation reports stochastic Avg@K, averaging independently verified responses rather than selecting the best response. The paper does not provide repeated training-seed uncertainty establishing that the student-filtering gains of 0.6 and 0.4 percentage points are reliable. Its takeaway incorrectly labels already-solved problems as pass-rate zero; the table and procedure instead discard pass-rate-one problems. Centered smoothing uses future checkpoints, so plotted curves cannot establish precise online change times. Tables are unsmoothed.

Transfer to KB operations remains an inference. The experiments cover text reasoning in math, code, science and instruction following. They do not test deployed continual adaptation, interactive agents, KB artifact revision, or transfer of harmful behaviour. The last is a risk the authors discuss, not an observed safety result. No implementation was inspected and no training or evaluation code was executed for this ingest.

Update Continual learning requires governing behaviour-changing writes with this bounded post-training example showing why regression validation must extend beyond an update's nominal domain.