Ingest: Intern-S2-Mobius: Foundation Model with Decoupled Knowledge and Reasoning

Type: kb/sources/types/ingest-report.md

Classification

An arXiv v1 preprint with a proposed foundation-model architecture, training-from-scratch and continual-pretraining comparisons, throughput and output-length measurements, qualitative cases, and an official implementation release. Author: the Intern-S2-Mobius Team at Shanghai AI Laboratory; the paper and official release provide primary design evidence, but the four-day-old v1 preprint has no reported venue or independent replication.

Summary

Mobius-v0 separates layer-local token mixing from a pool of FFN experts shared across layers. The paper calls the shared experts Memory and the attention or linear-attention stages Reasoners: hidden states pass through successive reasoners, select sparse experts, and return the expert output to the residual stream. A 7B-A1B MoE trained from scratch reportedly reaches a Transformer's 1T-token MMLU score after 0.626T tokens. At 35B, the authors convert Qwen3.5-35B-A3B, continue pretraining for 1T tokens, then apply SFT and RL; the resulting model reports a 67.88 versus 65.05 general-benchmark average, much larger gains on three scientific tasks, 1.5 times shorter output on average across five reasoning benchmarks, and average request-throughput gains rising from 2.9 times at batch 16 to 4.6 times at batch 256. Those 35B comparisons do not isolate architecture from the additional training and post-training, and the paper explicitly leaves its self-evolution, world-model, scientific-discovery, and hardware claims for future validation.

Code Grounding

The association is direct: its README names Intern-S2-Mobius, describes the same architecture and results, links the model release, and includes the technical report.

Implemented. The configuration exposes a block count, hybrid linear/full-attention layer pattern, routed-expert count, and experts selected per token (configuration). The released defaults specify four shared blocks, 256 routed experts, and top-eight selection, though the repository does not include the deployed 35B checkpoint's config.json. The model constructs the configured number of shared routed MoE blocks, uses hidden-state-conditioned top-k routing, assigns each decoder layer to layer_idx % num_blocks, and combines that output with a per-layer shared expert (router and decoder layer, model assembly and traversal). This confirms block-wise shared sparse FFN access across layers. It is more precise than the paper's “global vector database” metaphor: the operative state is distributed expert weights selected by a learned router, and the default implementation partitions it into four shared blocks while retaining layer-local experts.

Artifact-supported but not executed. The README provides Transformers and serving commands, recommends MTP speculative decoding, names OpenCompass, states maximum evaluation lengths, and republishes benchmark, throughput, and output-length figures (evaluation and deployment documentation). These artifacts show how the release is intended to be served, but the checkout has no dependency manifest, tests, training scripts, evaluation scripts, raw scores, benchmark configuration, hardware record, or released result logs. Its bundled technical-report PDF also differs byte-for-byte from the version-pinned arXiv v1 PDF, so the arXiv snapshot remains the paper authority.

Paper-only. The checkout does not demonstrate that shared experts contain semantically identifiable knowledge, that selected experts causally mediate correct answers, or that parameter sharing implements a backward residual connection in the stronger semantic sense. The released forward method traverses the configured decoder layers once; no explicit adaptive latent-iteration loop or dynamic per-token iteration budget appears in this checkout. The claimed data efficiency, shorter reasoning, throughput, benchmark quality, catastrophic-forgetting advantage, and future self-evolution properties remain paper claims. No model code was executed: the repository supplies no cheap test environment, and running inference would require downloading a 35B checkpoint and dependencies, which this ingest deliberately did not do.

Quotes

• Source extract (verbatim): We introduce Mobius-v0, an architecture that comprises a globally shared Memory (FFN) that stores knowledge vectors and multiple Reasoners (Self-Attn) that iteratively achieve compositional reasoning.
• Source location: Abstract.
• Source extract (verbatim): Using hidden states as cache and carrier, reasoners repeatedly query memory for required knowledge-vectors, while the knowledge is transmitted back to reasoning operators.
• Source location: Abstract.
• Source extract (verbatim): Therefore, at larger parameter scales, we employ a block-wise partitioning technique analogous to MoE [63] to partition the FFN, with sparse activation during forward propagation.
• Source location: Section 2.3, “Disentangling Knowledge Vectors and Reasoning Operators.”

Connections Found

The source is a useful same-form role-separation case for Axes of artifact analysis and representational form: Memory and Reasoners have distinct architectural roles but remain distributed-parametric. Its implemented top-k router provides a concrete model-internal selection path bearing on Knowledge storage does not imply contextual activation, without showing semantic faithfulness. The experiment interpretation rests on Learning inside a fixed decomposition inherits its mistakes, because compound gains do not independently validate the fixed memory partition, routing, hybrid attention, training recipe, or serving path.

The paper's self-evolution forecast is a counterpoint to The deployed system, not the model alone, is the unit of learning and Continual learning requires governing behaviour-changing writes, not just storing content: expandable parametric storage does not govern which behavior-changing write to accept, validate, coordinate, or revert across a deployed system. KBLaM supplies the closest model-integrated memory comparison, while Externalization in LLM Agents and Reason Wide, Not Deep place knowledge and displaced reasoning work in localized artifacts instead of shared weights.

Extractable Value

1. Architectural role separation is orthogonal to representational form. Mobius separates routed expert weights from reasoner layers, but both are distributed-parametric and reviewed through behavior and probes. The design does not thereby gain claim-level provenance, selective invalidation, or readable governance. [quick-win]
2. “Global Memory” resolves in code to partitioned shared expert pools. The default configuration constructs four MoE meta-blocks; layers address them cyclically, route each hidden state to eight of 256 experts, and then add a layer-local expert. This implementational detail prevents the database metaphor from being misread as one addressable store or universal all-layer access to every expert on every step. [quick-win]
3. Routing is a selection mechanism, not evidence of semantic activation. Hidden states determine top-k expert selection, so the release implements a storage-to-computation path. Neither code inspection nor router frequency shows that an expert carries the required knowledge, that the answer used it faithfully, or that the shared pool caused the result. This sharpens the activation note's model-internal boundary. [deep-dive]
4. The reported throughput is a system outcome with unresolved attribution. The paper itself says individual passes pay higher memory-access cost; aggregate speed comes largely from shorter outputs, while the release recommends MTP serving. Architecture, 1T-token continual pretraining, SFT/RL, output style, speculative decoding, serving framework, batching, and hardware therefore need separate controls before “latent reasoning causes 4x speedup” is warranted. [experiment]
5. Short visible reasoning does not establish internalized deliberation. Two step-aligned examples and one layerwise prediction example show concise, target-aligned behavior, but do not distinguish latent computation from length-shaping post-training, benchmark-specific answer style, or suppressed verbalization. The source is a parametric counterpart to passive skill distillation, not yet evidence that either route preserved the same reasoning process. [experiment]
6. Model architecture does not close the self-evolution loop. Shared experts may change interference and expansion economics, but the paper does not implement deployment feedback, proposal selection, acceptance criteria, regression tests, cross-form coordination, or selective rollback. Its self-evolution section identifies a possible substrate property, not an operative continual-learning system. [deep-dive]
7. The experiments improve within a tightly fixed update space. Behavior can condition on token history, hidden-state and cache/recurrent state, and learned router scores; it can compose hybrid token mixing, residual updates, top-k expert calls, and subsequent decoding; its router and expert weights can express mappings inside that architecture. The four-block partition, modulo layer assignment, top-k basis, hybrid layer pattern, checkpoint conversion, training/post-training recipe, benchmark set, maximum lengths, and serving protocol remain fixed. The training-from-scratch comparison varies the compound architecture, while the 35B comparison also varies training history; neither isolates the adjacent choices. [deep-dive]

Limitations (our opinion)

Static source inspection confirms only that a mechanism is present. The GitHub release lacks the deployed checkpoint configuration, weights, tests, manifests, training code, evaluation code, raw predictions, throughput harness, and hardware/software record. It therefore cannot reproduce—or independently verify—the training, benchmark, latency, throughput, or quality results. The default values in the configuration class should not be assumed to describe the released 35B checkpoint without its actual config.json.

The 7B result is a single MMLU learning curve with no reported seeds, uncertainty, or compute-normalized comparison; reaching the Transformer's 1T-token score at 0.626T tokens supports local token efficiency, not general knowledge compression. The 35B comparison is less isolating: Mobius receives architecture conversion, 1T tokens of continual pretraining, SFT, and RL, while the displayed baseline is Qwen3.5-35B. Gains, regressions, reasoning length, and scientific-task scores can therefore reflect additional data or post-training rather than architecture. The paper's toy compositional-generalization result uses an unreleased future Mobius version and cannot validate the released model.

Throughput is reported by batch-size curves without enough reproduction detail. Average gains range from 2.9 to 4.6 times across the displayed batch sizes, but several individual tasks show mixed scaling; average output length is only 1.5 times shorter, with much larger reductions on MMLU-Pro and GPQA. Without equal training, equal decoding and MTP settings, hardware, framework revisions, prompt/stop conditions, and raw length distributions, the causal mechanism and operating range remain unknown.

Finally, the central architectural terms outrun the direct evidence. Shared expert weights are not a database of individually governed facts. Parameter sharing across layers does not by itself prove backward information flow or compositional generalization. A single forward traversal through layers is not an observed adaptive latent-reasoning loop. The self-evolution, world-model, scientific-discovery, and SSD-backed knowledge-expansion sections are explicitly prospective and should not be promoted as results.

Recommended Next Action

Update Axes of artifact analysis with Mobius as a bounded worked case showing that knowledge/reasoning role separation inside model weights does not change representational form or supply localized-artifact governance.