Near-decomposability may make local inspection affordable

Type: types/note.md · Tags: learning-theory, artifact-analysis

An artifact may be affordable to inspect when the question can be answered by reading one part and summaries of its interactions with the rest. A division whose parts interact weakly enough for this is a near-decomposition. This note conjectures that knowing such a division can make local inspection affordable. It leaves open whether other inspection methods can be affordable without one.

The proposal applies to three sources of opacity, where answering the question exceeds the inspection budget: scale, the reference's form, and deliberate obfuscation. It transfers Herbert Simon's account of dynamic systems to dependencies within artifacts; that transfer is this note's conjecture, not Simon's claim. Description length is explored as a proxy for the amount to explain. Finding and checking the explanation also cost work.

The condition

Simon's Architecture of Complexity introduces the condition as a second approximation after perfectly separable systems: a theory of nearly decomposable systems, in which the interactions, verbatim, "among the subsystems are weak, but not negligible." Two propositions follow, quoted verbatim with the capture's typography: "( a ) in a nearly decomposable system, the shortrun behavior of each of the component subsystems is approximately independent of the short-run behavior of the other components; ( b ) in the long run, the behavior of any one of the components depends in only an aggregate way on the behavior of the other components."

The proposed transfer reads interaction as dependency between the units of an artifact: a call, a citation, a shared variable, a weight that two computations use. If the question's relevant dependencies fit inside one block and the remaining effects can be captured by interface summaries, local inspection need not read the whole artifact. The block and those summaries determine how much material must be examined. Whether that material is affordable to understand and check is a further question.

Dense coupling, or a hub every block depends on, can defeat local reading while each unit stays readable.

In this transfer, the horizon and tolerance determine which interactions may be ignored or summarized. A decomposition useful for one question need not support another. The analogy does not establish that every artifact has a short horizon on which questions localize, or that every long-horizon question reaches every block.

A proposed description-length proxy

The proposed proxy for inspection cost follows Grünwald's crude two-part form of the minimum description length principle, verbatim: "The best point hypothesis H ∈ H(1) ∪ H(2) ∪ . . . to explain the data D is the one which minimizes the sum L(H) + L(D|H), where • L(H) is the length, in bits, of the description of the hypothesis; and • L(D|H) is the length, in bits, of the description of the data when encoded with the help of the hypothesis."

Mapped to inspection, the hypothesis is the block and its rules, and the residual is what the aggregates and the exceptions add. In the prompt case, the hypothesis is the coarse English account and the residual describes observed behavior left unexplained by it. The proposed empirical test probes that residual; a description-length score additionally requires a specified encoding.

The measure is relative to a description method. Grünwald requires one that is, verbatim, "restrictive enough" that the shortest description is computable, chosen by prior knowledge of the domain. For inspection, the description method is the vocabulary of units the consumption path exposes. The same artifact therefore has different costs under different paths: opacity is path-relative.

A found explanation and its residual supply an upper bound on the shortest description available in the chosen description method. The restriction quoted above makes the minimum computable in practical MDL; a search that merely retains the shortest explanation found so far need not have reached it. Neither quantity bounds the effort of finding or checking that explanation without a further cost model. Description length is a proposed proxy for the amount to explain, not a measure of that work. No calibrated relation between these costs is established here.

The hypothesis side can be gamed. Grünwald notes, verbatim, that the length of a hypothesis "can be very large under one code, but quite short under another", so a crude code for hypotheses is "in danger of becoming arbitrary". A comparison of inspection costs is honest only when the vocabulary of units is fixed before the comparison. The KB computes neither description length nor the work of finding or checking an explanation. The proposed proxy fixes what a cost comparison has to hold constant.

One failure, three sources

The conjecture reads the three sources of opacity as different obstacles to local decomposition:

  • Scale. A decomposition is known, and the block containing the question is too large to read, or a hub couples it to everything. Each unit is readable; the question is not affordable.
  • The reference's form. The units exposed by the reference may not align with the question. Kernels and tensor operands expose runtime operations without supplying a readable instruction-level rule. A coarse semantic description of the prompt may still answer the question, as the English approximation account explains. Opacity remains where that description is insufficient and the relevant detail cannot be established affordably; the absence of a separate parameter block alone does not establish it.
  • Deliberate obfuscation. A transformation may hide or disrupt a decomposition while preserving the function. Whether this explains the inspection obstacle must be established for the transformation and question. The cryptographic results summarized in the opacity note do not by themselves establish this structural account.

Known but expensive, unknown, and disrupted would then be three states of one obstacle. Opacity's definition does not depend on this unification succeeding.

Finding an intervention point does not establish safe repair

Mechanistic interpretability is the search for a decomposition where none is exposed. Templeton and co-authors train sparse autoencoders on a production model's activations and test the found units by intervention. Clamping the Golden Gate Bridge feature to ten times its maximum activation changes behavior as the feature's interpretation predicts; verbatim, "In this example, the model starts to self-identify as the Golden Gate Bridge!"

Under the path that addresses features, the activations are localized for the behaviors the found features cover, and the authors are explicit that the coverage is partial. Verbatim, they "could only find features corresponding to about 60% of the boroughs" of London in their largest dictionary, and call the found set "an incomplete description of the model's internal representations". So localization under an interpretability path is graded by coverage, and the form under the path on which a system runs the model is unchanged.

ROME is the same move on weights: a factual association is located, edited, and the edit's effect checked.

Finding an interpretable feature provides a place to intervene. It does not establish that the intervention preserves properties the system relies on. Even weak interactions can break a required invariant: an effect small enough to ignore when predicting typical behavior may be decisive when checking correctness.

The Golden Gate Bridge intervention demonstrates behavioral influence. It does not establish coupling strength or, without a stated requirement, an invariant violation. To assess whether a decomposition supports a particular repair, we must check whether its interaction summaries capture the effects relevant to the invariants that repair must preserve.

Consequence for addressability

The proposed transfer to repair concerns how much must be checked after an edit. If a decomposition captures the dependencies relevant to the test, weak interactions across its blocks may let the inspector assess the edit within one block and its interfaces. Establishing that the decomposition captures those dependencies is part of the work.

That checking cost is separate from locating a repair candidate in an addressable theory, whose parts can be inspected and revised individually. Locating a candidate and establishing that its repair preserves required invariants are separate achievements.

Consider a shared conversion routine that adds one where its stated rule requires adding two. A trace through that operation can identify a candidate fault even when many callers depend on the routine. Repairing it still requires checking the affected callers. Many callers establish a broad dependency footprint, not by themselves strong interaction between blocks.

For a direct coupled case, consider the explicitly stated equations x = y + 1 and y = 2x: each equation is separately editable although changing either can change both variables. Separate editability does not require independent consequences.

Scope

  • Simon's result is a theorem about linear dynamic systems described by a nearly decomposable interaction matrix. Reading interaction as dependency between an artifact's units is this note's transfer. It needs a worked case on a real codebase or KB dependency graph before it can enter a definition as more than a mechanism.
  • Decomposability is a structure. Opacity is a cost judgment on top of it, relative to the question, to the decompositions known, and to the budget. Composing a model from understood numerical operations does not by itself supply an affordable instruction-level description. A coarse semantic decomposition may supply one without locating separate parameter blocks.
  • Simon flags a selection effect: a world may look hierarchic because its non-hierarchic parts elude understanding. The mechanism explains the affordable cases without showing that they predominate.

Open Questions

  • Compression. Simon ties near-decomposability to the redundancy that makes a compact description of a system possible, and the epiplexity ingest's time-bounded description length is the nearest formal neighbor. Whether inspection cost and compressibility are one quantity or two is a separate task.
  • Whether obfuscation is exactly the destruction of near-decomposability, or only one route to it.
  • Whether affordable abstraction or certificate checking can answer a question without the near-decomposition proposed here. Such a case would limit the mechanism to local reading rather than opacity in general.
  • The worked case named under Scope.

Relevant Notes: