Ingest: Scaling Monosemanticity: Extracting Interpretable Features from Claude 3 Sonnet

Type: types/ingest-report.md · Tags: observability, artifact-analysis

Classification

This is an empirical research paper with a method, scaling analysis, quantitative comparisons and many intervention case studies. It was published on 2024-05-21 in Anthropic's Transformer Circuits thread, not through peer review. Author: Anthropic's interpretability team (Templeton, Conerly, Olah, Henighan and 22 others). They have direct access to the production model and lead this research line. The model is proprietary, its size is withheld, and outsiders cannot reproduce the results on it.

Summary

The paper trains sparse autoencoders (SAEs) on one site in Claude 3 Sonnet: the residual stream at the middle layer. An SAE is a learned dictionary that rewrites each activation vector as a sparse sum of "feature" directions. The authors train ReLU SAEs with an L1 sparsity penalty at 1M, 4M and 34M features. Fewer than 300 features are active per token, and the reconstruction explains at least 65% of activation variance. The SAE training loss follows power laws in compute, and the authors use those laws to set training budgets. The features read as abstract concepts that are often multilingual and multimodal. For four hand-picked features, model-rated specificity is high at strong activations. Randomly sampled features are more interpretable than randomly sampled neurons. Coverage is incomplete and depends on concept frequency: a concept is more than 50% likely to get its own feature at a training-data frequency slightly below 1/(alive features). The 34M SAE has features for only about 60% of London boroughs. The central causal test is clamping. The harness replaces the SAE reconstruction with one in which a single feature is held at a chosen value, keeps the reconstruction error unchanged, and applies this at every input and token position. Clamping the Golden Gate Bridge feature at 10x its maximum makes the model self-identify as the bridge. Other clamps hallucinate code errors, elicit sycophantic praise, or reveal a "forgotten" word. On seven chosen cases, few-shot difference-of-means steering vectors matched feature steering in two and failed in five. Every one of these results is a finding about one decomposition: a single-layer, residual-stream, L1 SAE. Alternative decompositions were not compared.

Contribution assessment (our opinion)

Before this paper, SAE results came from small or one-layer models. The paper tests and survives the conjecture that sparse dictionary learning would fail on a production model. That makes it a significant feasibility result for mechanistic interpretability. Its stronger contributions go beyond the feature gallery. Each intervention test could have failed: an interpretation drawn from activation contexts predicts behavior in contexts where the feature is inactive. The frequency-threshold result is a quantitative, checkable regularity about what a fixed-size dictionary resolves. The scaling laws turn SAE training into ordinary engineering, though only against a proxy loss. The authors concede that this loss has no ground-truth link to interpretability ("Inability to Evaluate"). That concession limits the scaling laws and the dictionary-size comparison, not the demonstration that interventions work. The causal evidence consists of selected successful cases, without success rates or measured side effects. So the paper shows that features can steer the model. It does not show how often a found feature does what its label predicts. My background knowledge suggests roughly concurrent SAE scaling work at other labs, so priority is shared. That knowledge is not a literature check.

Quotes

In this work, we focused on applying SAEs to residual stream activations halfway through the model (i.e. at the “middle layer”). --- kb/sources/.snapshots/scaling-monosemanticity-claude-3-sonnet.md @ sha256:ab50b7c80ddec75af5651ea71a9c3ab41bd743ae9899e2b87699e98f0205dbb6 — Our SAE experiments

For all three SAEs, the average number of features active (i.e. with nonzero activations) on a given token was fewer than 300, and the SAE reconstruction explained at least 65% of the variance of the model activations. At the end of training, we defined “dead” features as those which were not active over a sample of $10^{7}$ tokens. The proportion of dead features was roughly 2% for the 1M SAE, 35% for the 4M SAE, and 65% for the 34M SAE. --- kb/sources/.snapshots/scaling-monosemanticity-claude-3-sonnet.md @ sha256:ab50b7c80ddec75af5651ea71a9c3ab41bd743ae9899e2b87699e98f0205dbb6 — Our SAE experiments

Our goal is just to demonstrate that interpretable features exist, leaving strong claims (such as most features being interpretable) to a later section. --- kb/sources/.snapshots/scaling-monosemanticity-claude-3-sonnet.md @ sha256:ab50b7c80ddec75af5651ea71a9c3ab41bd743ae9899e2b87699e98f0205dbb6 — Four Examples of Interpretable Features

We find that feature steering is remarkably effective at modifying model outputs in specific, interpretable ways. It can be used to modify the model’s demeanor, preferences, stated goals, and biases; to induce it to make specific errors; and to circumvent model safeguards (see also Safety-Relevant Features). We find this compelling evidence that our interpretations of features line up with how they are used by the model.

--- kb/sources/.snapshots/scaling-monosemanticity-claude-3-sonnet.md @ sha256:ab50b7c80ddec75af5651ea71a9c3ab41bd743ae9899e2b87699e98f0205dbb6 — Influence on Behavior

For instance, we see that clamping the Golden Gate Bridge feature 34M/31164353 to 10× its maximum activation value induces thematically-related model behavior. In this example, the model starts to self-identify as the Golden Gate Bridge! Similarly, clamping the Transit infrastructure feature 1M/3 to 5× its maximum activation value causes the model to mention a bridge when it otherwise would not. In each case, the downstream influence of the feature appears consistent with our interpretation of the feature, even though these interpretations were made based only on the contexts in which the feature activates and we are intervening in contexts in which the feature is inactive.

--- kb/sources/.snapshots/scaling-monosemanticity-claude-3-sonnet.md @ sha256:ab50b7c80ddec75af5651ea71a9c3ab41bd743ae9899e2b87699e98f0205dbb6 — Influence on Behavior

We find that activations of a random selection of SAE features are significantly more interpretable on average than a random selection of MLP neurons. --- kb/sources/.snapshots/scaling-monosemanticity-claude-3-sonnet.md @ sha256:ab50b7c80ddec75af5651ea71a9c3ab41bd743ae9899e2b87699e98f0205dbb6 — Features vs. Neurons

We find increasing coverage of concepts as we increase the number of features, though even in the 34M SAE we see evidence that the set of features we uncovered is an incomplete description of the model’s internal representations. For instance, we confirmed that Claude 3 Sonnet can list all of the London boroughs when asked, and in fact can name tens of individual streets in many of the areas. However, we could only find features corresponding to about 60% of the boroughs in the 34M SAE. This suggests that the model contains many more features than we have found, which may be able to be extracted with even larger SAEs. --- kb/sources/.snapshots/scaling-monosemanticity-claude-3-sonnet.md @ sha256:ab50b7c80ddec75af5651ea71a9c3ab41bd743ae9899e2b87699e98f0205dbb6 — Feature Completeness

We don't think the existence of these features should be particularly surprising, and we caution against inferring too much from them. It's well known that models can exhibit these behaviors without adequate safety training or if jailbroken. The interesting thing is not that these features exist, but that they can be discovered at scale and intervened on. In particular, we don't think the mere existence of these features should update our views on how dangerous models are – as we'll discuss later, that question is quite nuanced – but at a minimum it compels study of when these features activate. --- kb/sources/.snapshots/scaling-monosemanticity-claude-3-sonnet.md @ sha256:ab50b7c80ddec75af5651ea71a9c3ab41bd743ae9899e2b87699e98f0205dbb6 — Safety-Relevant Features

it's important to note that the present work does not show that any features are actually useful for safety. Instead, we merely show that there are many which seem plausibly useful for safety. --- kb/sources/.snapshots/scaling-monosemanticity-claude-3-sonnet.md @ sha256:ab50b7c80ddec75af5651ea71a9c3ab41bd743ae9899e2b87699e98f0205dbb6 — Safety-Relevant Features

Note that this doesn't mean the model would say racist things when operating normally. --- kb/sources/.snapshots/scaling-monosemanticity-claude-3-sonnet.md @ sha256:ab50b7c80ddec75af5651ea71a9c3ab41bd743ae9899e2b87699e98f0205dbb6 — Bias Features

We urge caution in interpreting these results. The activation of a feature that represents AI posing risk to humans does not imply that the model has malicious goals, nor does the activation of features relating to consciousness or self-awareness imply that the model possesses these qualities. How these features are used by the model remains unclear. --- kb/sources/.snapshots/scaling-monosemanticity-claude-3-sonnet.md @ sha256:ab50b7c80ddec75af5651ea71a9c3ab41bd743ae9899e2b87699e98f0205dbb6 — Features Relating to the Model’s Representation of Self

Superficial Limitations. In our work, we perform dictionary learning over activations sampled from a text-only dataset similar to parts of our pretraining distribution. It did not include any “Human:” / “Assistant:” formatted data that we finetune Claude to operate on, and did not include any images. --- kb/sources/.snapshots/scaling-monosemanticity-claude-3-sonnet.md @ sha256:ab50b7c80ddec75af5651ea71a9c3ab41bd743ae9899e2b87699e98f0205dbb6 — Limitations, Challenges, and Open Problems

The objective we optimize – a combination of reconstruction accuracy and sparsity – is only a proxy for what we really are interested in, interpretability. --- kb/sources/.snapshots/scaling-monosemanticity-claude-3-sonnet.md @ sha256:ab50b7c80ddec75af5651ea71a9c3ab41bd743ae9899e2b87699e98f0205dbb6 — Limitations: Inability to Evaluate

Unfortunately, we don't think this fully avoids the problem: features which are partly represented by later layers will still be impossible to properly interpret. --- kb/sources/.snapshots/scaling-monosemanticity-claude-3-sonnet.md @ sha256:ab50b7c80ddec75af5651ea71a9c3ab41bd743ae9899e2b87699e98f0205dbb6 — Limitations: Cross-Layer Superposition

Getting All the Features and Compute. We do not believe we have found anywhere near “all the features” that exist in Sonnet, even if we restrict ourselves to the middle layer we focused on. We don't have an estimate of how many features there are or how we'd know we got all of them (if that's even the right frame!). We think it's quite likely that we're orders of magnitude short, and that if we wanted to get all the features – in all layers! – we would need to use much more compute than the total compute needed to train the underlying models. This won't be tenable: as a field, we must find significantly more efficient algorithms. --- kb/sources/.snapshots/scaling-monosemanticity-claude-3-sonnet.md @ sha256:ab50b7c80ddec75af5651ea71a9c3ab41bd743ae9899e2b87699e98f0205dbb6 — Limitations: Getting All the Features and Compute

Interestingly, we find that obtaining interesting results typically requires clamping feature activations to values outside their observed range over the SAE training dataset. We suspect that this is because we perturb only one feature at a time, which typically might be co-active with several correlated features with related meanings. At the same time, clamping feature activations to too extreme a value (say, ±100× their observed maximum) typically causes the model to devolve into nonsensical behavior, e.g., repeating the same token indefinitely. --- kb/sources/.snapshots/scaling-monosemanticity-claude-3-sonnet.md @ sha256:ab50b7c80ddec75af5651ea71a9c3ab41bd743ae9899e2b87699e98f0205dbb6 — Methodological Details: Feature Steering

Connections Found

The KB has no prior treatment of sparse autoencoders or feature steering. For the occasion, this source is the activation-side companion and counterpoint to ROME on the localization question in representational form. That definition calls content localized under a consumption path when the consumer on that path takes up separately identifiable units to inspect or revise. The SAE-instrumented forward pass is such a path, and it reads feature directions (top activations plus model-written labels) and revises them (clamp values). Three qualifications bound the result. First, the units are directions in one layer's activation space. The weights are not decomposed or edited: the clamp lives in the harness and is applied at inference. Weights stay distributed-parametric under the path on which the system runs the model as its interpreter. What becomes localized is the middle-layer activation, under a path that includes the SAE. Second, the units are relative to the instrument. A 1M-feature "San Francisco" feature splits into eleven features at 34M. Up to 35% of activation variance stays in an error term, and concepts rarer than about 1/(alive features) get no dedicated unit. Third, nothing fixes a clamp's consequences: the value is found by trial, typically between -10 and 10, and extreme values produce nonsense.

The content-versus-scope result from the ROME ingest holds here more starkly. Clamping is content-addressable for covered concepts. A prompt or a pair of contrasting prompts finds the feature, and the clamp changes that concept's contribution. But the intervention is applied "for every model input, and at every token position", so it carries no applicability boundary. The Golden Gate model talks about the bridge everywhere, which shows that missing boundary. The read side gives partial scope information: activation examples show where a feature fires, with specificity measured and sensitivity left unmeasured. Reading where a unit fires is still not rescoping what it does. Unlike ROME, nothing is written to the model, so the change is not durable in weights.

Secondary roles: the paper is evidence for opacity depends on the inspection question and budget. Coverage grows with dictionary size and compute. The authors also estimate that finding "all the features" would cost more compute than training the model. The paper bears on inspectable artifact, not supervision, defeats the blackbox problem by giving partially addressable units inside a model, but those units lack the stable identity, completeness and stated rules of a readable script. Its safety discussion separates having a deception or bioweapons feature from when that feature activates. Knowledge storage does not imply contextual activation draws the same distinction, at a different level. Among sources, this paper compares with Verbalizable Representations Form a Global Workspace as another Anthropic instrument for reading Claude-internal concepts, and with Reading Between the Dots on internal readouts as a monitoring surface.

Extractable Value

1. A decomposition localizes activations under an instrumented path, not weights under the interpreter path -- This answers the occasion's first question. SAE features are units that a harness can read and revise. So the localization question gets a "partly yes" under a path that includes the SAE, for the concepts the dictionary resolves at one layer. Under the interpreter path, the weights remain distributed-parametric. The localization is instrument-relative (feature splitting), partial (at least 65% variance explained, frequency threshold) and inference-time only. Representational form's default "probe distributed-parametric artifacts behaviorally" therefore needs a qualified exception, not a reversal. The claim rests on one SAE design at one layer. [deep-dive]

2. Content-addressability without scope-addressability, a second instance after ROME -- Clamping reaches a concept but has no applicability condition. By construction it applies at every position and input. Activation statistics describe where a unit fires, but they give no handle for narrowing where an intervention acts. ROME and this paper differ in mechanism: a weight write versus an activation clamp. Both show the same gap, which makes content- and scope-addressability look like separate properties across mechanisms rather than a quirk of one editor. Scope could be added only from outside, by a harness that clamps conditionally. The paper does not test that. [deep-dive]

3. The intervention test is real but selective -- Interpretations come from activation contexts and are tested where the feature is inactive, so the test could fail. The reported cases are chosen successes. They come with no success rate, no side-effect measurement, and prompt-sensitive results such as the code-rewriting case. The probe comparison uses seven cases chosen because feature steering worked. Any note citing "behavior changed as predicted" should say that the prediction was thematic and the sample selected. [quick-win]

4. Dictionary resolution follows concept frequency -- A concept gets a dedicated unit at a frequency of roughly 1/(alive features). Rarer concepts can still be represented by combining features. This gives a measurable answer to "how many units does a decomposition need to cover a concept?" and bounds any claim of addressability by coverage. Its use for opacity depends on the inspection question and budget is direct: a measured, budget-dependent inspection case for distributed-parametric content. [quick-win]

5. Unsupervised decomposition amortizes direction-finding -- One dictionary-training run gives millions of candidate units, and finding a relevant one then takes a few prompts. This is the paper's case against bespoke probes. In the few-shot regime, its evidence is the two-of-seven comparison. [just-a-reference]

6. The source does not contain the public Golden Gate Claude demo -- The paper reports the 10x clamp and describes the result in a sentence. The steered transcripts appear only as omitted images in this capture. The public demo model was a separate release and is outside this source. [just-a-reference]

Limitations (our opinion)

The intervention evidence is qualitative case selection. No experiment reports how often a feature's label predicts the clamped behavior, what else the clamp changes, or how the result depends on the clamp value. The probe-steering baseline has seven cases. Those cases were chosen because feature steering succeeded, and the scaling factors were tuned by hand. The authors acknowledge both points. The paper tests one decomposition: a ReLU+L1 SAE on the middle-layer residual stream, trained on text-only pretraining-like data. Its comparisons vary dictionary size and contrast features with neurons and with few-shot probes. They do not compare other layers, other SAE objectives, transcoders or cross-layer methods. Claims about "the model's features" are therefore claims about this dictionary's units. Feature splitting shows those units change with dictionary size, which makes the claim easy to vary. A simpler account fits several results: clamping a large out-of-range vector along any semantically coherent direction will push outputs toward that theme. That account weakens "the model uses this feature" more than it weakens "this direction steers". The steering harness keeps the SAE error term and perturbs only the reconstruction. Results therefore depend on SAE fidelity, which is at least 65% variance explained, not complete. The SAE loss is a proxy without a ground-truth objective, so the scaling laws optimize something only loosely tied to interpretability. Specificity scores come from Claude 3 Opus, with manual spot checks. Sensitivity was not measured at scale. The model is proprietary and unreproducible, and the authors have an institutional interest in the safety framing, though they hedge it heavily. The capture is marked full-source, but images are placeholders. Plots, steered outputs and the deception case study's transcripts are absent, so the ingest relies on the authors' prose descriptions of them.

Recommended Next Action

Add an entry to kb/log.md that records ROME and this paper together as two mechanisms, a weight write and an activation clamp. Both reach content without scope. The entry should also record that SAE decomposition localizes activations only under an instrumented path, not weights under the interpreter path. This should happen before any amendment to the "Addressability and the localized forms" section of representational form. The entry should name the untested experiment that would test the scope claim: a harness that clamps a feature only under a stated condition, measured for collateral effects outside it.