Ingest: Studying Software Architecture Through Design Spaces and Rules

Type: types/ingest-report.md

Classification

This is an SEI technical report (CMU/SEI-90-TR-18, November 1990) that summarizes a PhD thesis. It states a method, works it through one domain, and reports one small validation experiment, so it is a scientific paper with a modest empirical component. Most of the detail is deferred to the thesis and to a companion report, neither of which is captured. The figures, including the full list of sample dimensions, are raster images missing from the text capture. Author: Thomas G. Lane, then a CMU School of Computer Science doctoral researcher on the Software Architecture Design Principles Project, working in the Mary Shaw lineage of early software-architecture research.

Summary

Lane proposes studying software architecture through a design space: a set of dimensions, each with a small discrete set of alternatives, where a concrete system is a point. He splits the dimensions into a functional side (requirements and evaluation criteria that hold just before gross structural design) and a structural side (the design choices). He then adds design rules, which he says are rarely hard constraints. A rule is a positive or negative weight on a combination of alternatives from two or more dimensions. Rules come in two classes: functional-to-structural rules that let requirements drive structure, and structural-to-structural rules that keep a design internally consistent. A design is scored by summing applicable weights, and a combinatorial search ranks the structural alternatives for a given set of functional values. The worked domain is user-interface software, with 25 functional and 19 structural dimensions over a three-part component model (application-specific, shared UI, device-dependent). The mechanical rule set has 622 primitive rules, which reduce to about 170 at a reasonable abstraction level. The validation used six systems not studied while building the space. Their designers classified them on every dimension, the program recommended structures from the functional values, and the recommendations were compared with the actual designs. Agreement was "moderate to substantial" on the kappa scale, and rule errors found during the test were deliberately left uncorrected. The one dimension group where agreement failed was representational choices. Lane conjectures that these depend on design-method considerations the run-time-structure space does not represent. The space was built by surveying existing systems, on an explicit biological-taxonomy analogy. Lane names its limits: little insight into never-seen structures (though it can expose untried combinations of known alternatives), the risk of overlooked dimensions, and the equal difficulty of removing extraneous ones. Only about 10–12 of the 25 functional dimensions turned out to matter, and many structural choices were dropped because they correlated with retained ones. These results all hold within one fixed space and component model. The experiment tests the rules against designers inside that decomposition and does not compare it with any alternative way of cutting the space.

Contribution assessment (our opinion)

The problem is important. Architecture knowledge was then, and largely still is, held as craft experience with no form that supports reuse or testing. Lane's contribution is to make that knowledge a criticisable artifact. A dimensional classification plus explicit weighted rules yields recommendations, and those recommendations can be checked against held-out designers. The held-out comparison, with the rules frozen once the experiment began, is the strongest part of the work. It is a real test that the rules could have failed, and their failure on representational dimensions localizes a gap in the space rather than just lowering a score. Design-space taxonomies existed before (Bell and Newell are cited), and from background knowledge the weighted-rule-plus-validation combination looks like the more distinctive step. Priority cannot be settled from this capture. The evidence is thin: six systems, designer self-classification without inter-rater checks, an aggregate kappa band with no per-dimension figures in this report, and rules tuned to the author's own judgments. These limits narrow the claim from "the approach is valid and powerful" to "a simple rule set over a well-chosen space recovered a meaningful share of expert agreement in one domain." They do not undermine the method contribution. Its likely significance is as an early, testable formulation of design-space reasoning, not as a validated design tool.

Quotes

No source quotes have been retained yet.

Connections Found

For the design-space study, Lane is the closest tested precedent in the KB for a design space that carries rules from requirements to structure and is checked against outside designs. The central comparison is with First-principles analysis maps a design space before selecting within it (compares-with, on how a space is constructed). Lane's space is explicitly catalogued from a survey, and he reports the costs the note predicts for catalogues: weak insight into new structures, a missing dimension surfaced only when an experiment failed, and extraneous dimensions. He also claims the space exposes untried combinations, a property the note gives to generated maps. That makes Lane a hybrid case for the note's question about classifications that only catalogue practice. Build Systems à la Carte is the generated counterpart (compares-with): its grid is derived from the abstractions involved, while Lane's comes from a survey with empirically fitted rules.

Lane's representational-dimension failure is evidence for Learning inside a fixed decomposition inherits its mistakes (is-evidence-for), as an information-omission case. Adjusting rules inside fixed dimensions cannot recover a distinction the space does not represent. The support is qualified: the rules were hand-tuned rather than learned, the sample is six systems, and Lane frames the attribution as a hedged conjecture.

On the shape of the space itself, Lane compares with Agent orchestration occupies a multi-dimensional design space. Both replace a single ladder with separable dimensions, but only Lane adds a requirement side and recommendation rules. He also states a pruning criterion the note lacks: drop a dimension that correlates with a retained one. He compares with the agent-memory review framework on classifying systems along controlled values. His single-category-per-dimension scheme is strained by his own cT example ("mostly implicit", with limited internal procedural use), which is the mixed-value failure the framework addresses with one-hot columns. Against Alexander's patterns, Lane offers a different unit of retained design knowledge. Alexander's is a context/problem/forces/solution document. Lane's is a dimension plus weighted pairwise rules, which can be scored mechanically and tested. The agentic-system analysis design brief aims to chart the design space of agentic systems, and Lane is an adjacent prior method for that goal (see-also).

Lane's held-out comparison is evidence for derivation and inheritance give starting warrant; discriminating evidence or proof earns scope (is-evidence-for). It is a real test the rules could have failed, but with no rival space and no base-rate baseline it earns scope only inside the one space it tested.

Two sources ingested for the same design-space study sit on either side of Lane. Ritchey's General Morphological Analysis (compares-with) prunes a cross-product of values with pairwise exclusions typed as logical, empirical or normative. It supplies the typing that Lane's few hard rules lack, while Lane shows that compatibility can be weighted rather than only pruned. ATAM (compares-with) comes from the same SEI lineage. It records tradeoff points with written reasoning and the assumptions under which each judgment holds, the record form Lane's weighted rules lack. In turn, Lane offers the outcome test that ATAM's method report does not.

One vocabulary caution applies throughout. Lane's "codifying design practice" is the ordinary sense of the word. In the KB's sense of codification (defined-in), only his mechanically evaluated rule set and ranking program count. His prose guidelines for human designers are constraining, not codification.

Extractable Value

1. Give the design-space study a requirements side, separate from its decision axes -- Lane's functional/structural split is what turns a classification into a design aid. Requirement axes hold the situation, structural axes hold the choices, and rules map the first to the second. A Commonplace study that sits between a problem statement and an ADR can record the problem's position on requirement axes and read candidate regions off the rules. Without that side, a map can describe options but cannot recommend among them. [quick-win]
2. Distinguish graded favour/disfavour rules from hard compatibility exclusions -- Lane reports that hard rules are rare at the architecture level, and most relations are weighted preferences resolved by the preponderance of evidence. A study's compatibility rules therefore need at least two kinds: exclusions that remove a region, and weighted preferences that rank surviving regions. Collapsing them would either overstate weak evidence as impossibility or lose it entirely. [quick-win]
3. Two rule classes with different jobs -- Requirement-to-structure rules drive selection, and structure-to-structure rules enforce internal consistency (and make search combinatorial). Lane also names, and leaves untested, a third class: rules among requirement axes, for estimating the cost of a requirement set. A study schema that types rules by which sides they connect makes clear which rules an ADR author consults for which question. [quick-win]
4. A held-out validation protocol for a space and its rules -- Freeze the rules, take cases not used to build them, have their owners classify them, compare the predicted with the actual choices, and report per-dimension agreement. Lane's protocol is a ready template for testing whether a Commonplace design-space study is reliable rather than decorative. In the KB setting, past ADRs not used to draft the study are the natural held-out cases. The result tests the rules within the chosen space, not the choice of space. [experiment]
5. Localized rule failure is a signal of a missing dimension -- Lane's rules failed only on representational choices, and he read that as the space lacking design-method axes, not as rules needing more tuning. A study should record per-dimension disagreement with held-out cases, and treat a dimension where no rule fits as a candidate for revising the axes. This connects revisability to the fixed-decomposition note: some errors can only be fixed by changing the map. [experiment]
6. Axis pruning criteria: low impact and correlation with retained axes -- Less than half the functional dimensions mattered, and many structural choices were predictable from one key dimension (application-interface abstraction level). A study can test candidate axes the same way and fold a correlated axis into the one that predicts it. Lane says pruning is as hard as discovery and that, for now, only refinement through use removes such errors. [quick-win]
7. Granularity, discretization and labelling choices are part of the space -- Lane flags choosing the useful granularity of alternatives as a key problem. He coarsens continuous quantities into a few levels when early design needs no more precision. He also treats the component labelling as a deliberate analysis choice that changes with the level of the system being studied. A study should record these choices explicitly, because they determine what the rules can see. [just-a-reference]
8. A design space is useful before it has rules -- Lane argues the space alone helps a designer avoid overlooking an option, the requirement side reminds them of crucial considerations, and imperfect rules help by quickly rejecting inferior choices. This supports shipping a study incrementally: axes first, then rules as evidence accumulates. [just-a-reference]

Limitations (our opinion)

• Small, self-reported validation. Six systems, each classified by one of its own designers, with agreement summarized only as a "moderate to substantial" kappa band. This report gives no per-dimension figures, confidence intervals or inter-rater check of the classifications, and the experiment details are deferred to the uncaptured thesis. Designers describing their own systems in the author's vocabulary may also align their answers with that vocabulary, which would inflate agreement. The result shows the rules are not arbitrary. It does not establish how reliable they are.
• No rival space was tested. The comparison is between rule predictions and actual designs within one fixed set of dimensions and one three-part component model. Nothing compares this decomposition with an alternative, so the "considerable leverage" Lane credits to the space is inferred, not measured. Without a rival space or even a naive baseline, such as always predicting the most common alternative per dimension, the reported agreement cannot be attributed to the space rather than to base rates. As in Learning inside a fixed decomposition inherits its mistakes, the method can refine rules only inside the space it starts with.
• Catalogued spaces inherit their sample. The space and rules were fitted to surveyed systems and the author's judgment. Lane concedes the method gives little insight into new structures. Its rules encode past consensus, which can be wrong in ways that agreement with other experts cannot detect. The "significant body of agreement among expert designers" also shows shared practice, not that the practice is good.
• The weighted-sum scoring is unjustified. Rule weights are hand-set and additive, and the report gives no account of how weights were chosen or how sensitive rankings are to them. A study that copies the scoring would inherit an aggregation model that has not been checked.
• Single domain and era. User-interface software around 1990 was chosen because the field was unsettled. Lane's speculation that flexibility requirements dominate structure is explicitly offered as possibly an artifact of the method or the domain. Transfer to KB or agent-system architecture is plausible but untested.
• Capture gaps. Figures 1-1 to 2-2 (the illustrative space, the component model and the sample dimension table) are not in the text capture, so the full list of dimensions and the actual rule notation cannot be checked here.

Recommended Next Action

When drafting the design-space study's type spec, add a requirement-axes section separate from the decision axes, and type each compatibility rule both by the sides it connects (requirement-to-decision or decision-to-decision) and by whether it is a hard exclusion or a weighted preference. Cite this ingest as the precedent for that split and for a held-out validation check against past ADRs.