Ingest: Concept Bottleneck Models

Type: kb/sources/types/ingest-report.md

Source: concept-bottleneck-models.md Captured: 2026-07-26 From: https://arxiv.org/abs/2007.04612

Classification

Genre: scientific-paper -- an ICML 2020 architecture paper with a stated model family, three training variants, two benchmark datasets, accuracy comparisons against end-to-end baselines, an intervention protocol, and reported failure modes. The genre recorded on the snapshot is correct. The snapshot was first captured from the arXiv abstract page and then extended in place with a body extraction covering the training variants, datasets, results, intervention procedure, and the authors' own caveats, so the specifics cited below are checked against the paper rather than inferred from the abstract. Domains: interpretability, addressability, human-oversight Author: Pang Wei Koh, Thao Nguyen, Yew Siang Tang, Stephen Mussmann, Emma Pierson, Been Kim, Percy Liang (Stanford, Google Brain). High authority signal; Kim and Liang are established in interpretability and robust machine learning respectively, and the paper is widely cited as the modern restatement of the concept-bottleneck idea.

Summary

The paper builds a predictor as f(g(x)), where g maps the raw input to a vector of human-supplied concepts and f maps only those concepts to the label — so every path from pixels to prediction is forced through a named, human-readable layer. Three fits are compared: independent (f trained on true concepts), sequential (f trained on g's predictions), and joint (weighted sum of concept and task loss, hyperparameter λ), positioned as a spectrum with the standard end-to-end model at λ → 0. On knee x-ray grading (OAI, 10 radiologist-assessed clinical concepts) the bottlenecks slightly beat the end-to-end baseline (RMSE ≈ 0.418 against 0.441); on bird identification (CUB, 112 attributes) they lose slightly (0.199–0.243 error against 0.175). The distinctive move is test-time intervention: replacing a predicted concept with its ground-truth value and propagating the change downstream cuts OAI RMSE from above 0.4 to roughly 0.3 after two corrections, without retraining. The paper is equally useful for what it reports going wrong — linear c → y maps respond poorly to intervention despite matched pre-intervention accuracy, and joint models with λ too small learn misaligned concepts where intervening increases error — leading the authors to conclude that task and concept accuracy alone do not predict how well intervention will work.

Connections Found

This source is the architecture-side counterweight to the KB's opacity-and-addressability cluster, and its arrival is overdue: Theory-mediated learning may improve sample efficiency under structured shifts now cites this ingest as its evidence that a legible interface can be installed by supervision before training, rather than only recovered by tooling after it. The sharper landing is Only explicit retention is currently durable, writable, and addressable at once, whose table gives every tacit form a flat "no" on addressability and whose named falsifier is interpretability-grade editing of a weight-encoded commitment. Concept bottlenecks approach that falsifier from an unanticipated direction — not tooling recovering an interface after training, but supervision installing one before it — and stop short of crossing it, because the correction is per-inference and nothing is retained. That combination is what makes the source valuable rather than merely adjacent: it is a graded middle with a legible boundary.

The same evidence tempers Inspectable artifact, not supervision, defeats the blackbox problem, whose load-bearing line is that you cannot write a test for a weight; here you can write one for a named intermediate of a weight-based model, which is the qualification Opacity is a scale threshold, not a class property already gestures at when it makes structural affordances, not form, set the threshold. Against Towards Faithfully Interpretable NLP Systems and Language Models Don't Always Say What They Think — the KB's two interpretability sources, both failure-side — this one runs the same intervention move constructively, as a correction channel rather than a diagnostic of unfaithfulness. It also sits at the opposite end of one problem from Towards Causal Representation Learning: CRL frames discovering high-level variables from low-level observations as the hard open problem, and this paper takes those variables as given.

Extractable Value

  1. Addressability can be installed at design time rather than recovered at inspection time -- The KB currently frames the route to addressable parametric retention as interpretability tooling applied after the fact, and names that as the falsifier for the explicit-only claim. This paper shows a third route: supervise the interface into existence before training and pay in concept annotation instead of interpretability effort. Highest-reach item, because it generalizes past this architecture to any pipeline where a legible intermediate can be specified in advance — including agent pipelines that could be built to route decisions through named, editable intermediate claims. [deep-dive]
  2. Legibility of an intermediate does not establish its faithfulness, and intervention response is the test that separates them -- The joint model with λ too small has a fully readable concept layer, competitive task accuracy, competitive concept accuracy, and interventions on it make predictions worse. That is a clean worked instance of the split Selective revision needs a faithful rationale, not just a legible one rests on, whose current evidence is entirely failure-side. It also supplies the missing operational test: a rationale is faithful to the degree that editing it moves the decision the way it should, which is cheaper to run than the note's proposed probe-count experiment. [experiment]
  3. Accuracy metrics do not predict the quality of the intervention channel -- The authors' own conclusion, and the strongest transferable warning here. Two models matched on every headline metric diverged sharply once someone tried to correct them; the linear c → y result shows the divergence is not even monotone in model simplicity. The KB analogue is that a review or oversight channel must be exercised to be evaluated, and cannot be certified from the artifact's ordinary quality signals. [quick-win]
  4. A correction channel that is rarely exercised is out of distribution for the system that must consume it -- Sequential and joint models are trained on ĉ → y and face a distribution shift to c → y at the moment of intervention, which is why the independent bottleneck — worse at everything else — gains the most from correction. This rhymes with the pricing failure recorded in The Risks of Invariant Risk Minimization: a mechanism that only operates where the training data is thin is the one least likely to behave. Together they are two instances of one shape. [experiment]
  5. The addressability tax was near zero on these tasks, with a number attached -- Forcing all information through 10 or 112 named concepts cost nothing on OAI (bottlenecks beat the baseline) and about 2–7 error points on CUB. Where the KB's claims about the cost of legible structure are qualitative, this is a measurement — bounded to two vision tasks with dense expert annotation, but a measurement. [just-a-reference]
  6. Per-inference correction is not retention, and the boundary is worth keeping sharp -- Nothing survives the forward pass; the next input gets the same uncorrected g. This is what stops the source from falsifying the explicit-only claim, and it names precisely the property a genuine falsifier would have to add: a durable write back into the parametric commitment. Useful as a boundary marker in Only explicit retention is currently durable, writable, and addressable at once, not as a counterexample to it. [quick-win]

Limitations (our opinion)

Editorial judgment. The concept set is assumed, not discovered: both datasets carry dense expert annotation, which is exactly the input most settings lack, and the paper does not test what happens when the concept vocabulary is incomplete, wrong, or contested — the hard problem causal representation learning names is sidestepped rather than solved, and the cost of annotation is never accounted against the interpretability gained. The bottleneck is asserted to be a bottleneck but not tested as one: the paper reports no check on whether task-relevant information routes around the named concepts through soft concept values, and its own λ-misalignment failure is a symptom that this can happen. Interventions are performed by an oracle supplying ground truth in an ordering fixed on a validation set; there is no fallible human, no online ordering, and no accounting of the human effort the intervention gains are traded against, so "accuracy improves significantly if we can correct model mistakes" is a conditional whose antecedent the paper does not price. The evaluation is two vision tasks from 2020, and reading it as evidence about correctable rationales in language models or agent systems would extend it well past what was tested. Most importantly for this KB, the intervention is per-inference and non-durable: treating it as evidence that parametric commitments can be retained addressably would over-generalize it, and the honest reading is a graded middle — retrieval and correction of individual commitments, without durable write.

Revise Only explicit retention is currently durable, writable, and addressable at once: its "The claim is current, and its falsifier is named" section frames the falsifier as interpretability-grade editing recovered after training, and its first Open Question asks whether probing and steering form a graded middle worth naming. This source answers both — a design-time route to the same interface, landing in the graded middle rather than across the falsifier — so the section should record architecture-installed addressability as a second route to the falsifier and cite this ingest as the worked case, with the non-durability of the correction stated as the reason the row's "no" still stands.