Ingest: The "Problem" of Automation: Inappropriate Feedback and Interaction, Not "Overautomation"

Type: types/ingest-report.md · Tags: observability

Classification

This is a short position paper, not an empirical study. It argues one thesis from three aviation incident reports, two thought experiments and one footnoted informal simulator replication. It cites prior human-factors work but runs no controlled comparison. The captured text is the 1989 UCSD Institute for Cognitive Science report (ICS Report 8904), written for the Royal Society discussion meeting "Human Factors in High-Risk Situations" (June 1989) and later published in Philosophical Transactions of the Royal Society B (1990). This ingest analyses the NASA-hosted report, not the journal version. Author: Donald A. Norman, cognitive scientist at UC San Diego, author of The Psychology of Everyday Things (1988) and originator of the "system image" concept. The work was NASA-funded (grant NCC 2-591, Ames Aviation Safety/Automation Program), and Norman credits Edwin Hutchins's research on crew interaction.

Summary

Norman argues that accidents blamed on "overautomation" come from inappropriate automation design. Current automation sits at an intermediate level of intelligence: it can take over normal control, but it cannot handle all abnormalities, and it cannot provide the continual feedback that human co-workers give each other. In two cases (the 1985 China Airlines 747 engine-power loss and a 1987 fuel leak) the autopilot compensated silently for a growing fault. The crew found out only by chance, or only when the compensator reached its limit and the least recovery time remained. A third case (the 1979 Cape Cod "incapacitated" captain) shows the same detection failure with no automation at all, which Norman uses to argue that the culprit is lack of feedback and interaction, not automation. Two thought experiments make the point: from the captain's view, handing control to the autopilot or to the first officer both "automate" control, but only the human delegate would say "I seem to be correcting this thing more and more." Norman notes that the information was technically present (the control wheel visibly turned) but nobody commented on it. He blames single-threshold alarms and designers' habit of treating a device as finished "when the device functions as requested", since "the automation itself doesn't need" feedback. General anomaly reporting, he says, needs a "monitoring of its own monitoring" and a knowledge base about the system; nobody yet knows how to build that for the general case. Norman's "new irony" is that automation is not too powerful but not powerful enough. His remedy is design that assumes error, gives continual, informative and nonintrusive feedback, and plans for the worst situations.

Quotes

No source quotes have been retained yet.

Connections Found

For the occasion, the source's main role is as a case of function-level automation that leaves a companion function out of the task definition. The automated function (holding attitude) was operationalized and executed correctly. The function the human delegate supplied along with it (continual reporting of state and anomalies) was outside the designer's task definition because the executor did not need it. This is independent, non-LLM evidence for Unattended work needs the functions that interaction supplied, and it also qualifies that note. Norman's missing function is supplied by the delegate, not the operator, so it falls outside the note's five operator-supplied functions (purpose, context, correction, evaluation, stopping). The note's Scope section already allows for that.

The source is also a physical-control anchor case for Final task success does not establish intended-path health. A compensating autopilot produces the same visible outcome, a level aircraft, whether or not a fault is present. Norman's footnoted 727 simulator replication shows that a cue that was technically available (the cocked control wheel) was not salient enough to count as an execution signal. The source adds a dynamic the note does not state: when the masked fault grows and the compensator has a limit, hidden repair turns a recoverable drift into a late, time-compressed failure. That bears on when a synchronous signal is needed rather than an asynchronous one.

On decision autonomy, the source is a counterpoint and comparison for the human-factors automation sources already in the KB. It accepts the analyses in Ironies of Automation (Bainbridge) (out of the loop, deskilling, failure when most needed) and draws the opposite conclusion: make automation more capable, specifically at reporting, rather than less. Keep that tension visible rather than resolve it. Compared with A Model for Types and Levels of Human Interaction with Automation, Norman's claim that an intermediate level "tends to maximize difficulties" suggests that a given level of automation is safe only when the system's monitoring and analysis stages are shown to the operator.

Norman's "less intelligent or more" remedy can be compared with The augmentation-automation boundary is discrimination not accuracy. The shared axis is what a system must know about its own abnormal states before humans can step back. Norman's proposed signal, "compensating more than usual", is a crude self-assessment of that kind. His remark that automation "seems to function best when the workload is light and the task routine" is period illustration, not before-and-after evidence, for Warranted transfer out of the human cut leaves people the hardest-to-warrant decisions. LLM generation can hide a relaxed goal where human writing exposes a stall shares the axis of whether a failure announces itself where it occurs. The mechanisms differ: control saturation in Norman's cases, a plausible continuation in LLM generation.

The analysis is derived from the NASA NTRS original.

Extractable Value

1. Task definitions written for an executor drop functions the executor does not need -- Norman explains why feedback is missing: the designer's task "is completed when the device functions as requested", and reporting is secondary because "the automation itself doesn't need it." For the workshop, this gives a mechanism: operationalizing a function for a defined executor sets the boundary by what that executor consumes. Functions that served the surrounding operator, such as running status and anomaly reports, drop out unless the task definition names them. High reach: it explains a recurring omission rather than one incident. [deep-dive]
2. Same delegation, different delegate: the thought-experiment pair as a decomposition probe -- Holding the captain's delegation fixed and swapping the executor (autopilot or first officer) isolates what the human executor supplied beyond the delegated function. The workshop could reuse this device to separate "automating a function" from "automating everything the prior executor did in that role". It is a conceptual comparison, not a tested one. [quick-win]
3. Specific anomaly monitors are easy to operationalize; the general case needs a system model -- Norman says any competent programmer could write a monitor for yoke compensation or unexpected fuel loss. The general case (the next, slightly different anomaly) needs "a monitoring of its own monitoring" plus a knowledge base about the aircraft's systems. This bears on decision autonomy: escalation is easy to warrant within a defined task and anomaly class, and unsolved beyond it. It matches the task-and-executor scoping in the occasion. [experiment]
4. Silent compensation with a limit turns drift into a late, compressed failure -- A candidate claim that extends the hidden-recovery cluster around final-task-success: the hidden signal arrives exactly when the compensator saturates and the least time remains. The claim would decide when synchronous signals are required. It needs an agent-runtime instance, such as retry budgets or context-pressure workarounds, before it earns a note. [experiment]
5. Feedback should report trends, continually and without intruding -- Single-threshold alarms proliferate and irritate. Norman's alternative is continual feedback about the state of the system and trends in it, adjusting to the participants and the problem, and neither silent nor a "back-seat driver". This is a design requirement for operator-facing status in long-running agent work. Norman admits that he does not know how to meet it. [just-a-reference]
6. Information present but not salient is not feedback -- The wheel position was technically visible, and nobody noted it, even in a replication where the participant had read the accident report. This separates whether a signal exists from whether it reaches the people who need it. Observability claims should not stop at "the data was logged". [quick-win]

Limitations (our opinion)

This is a conceptual essay, and its evidential base is thin for a thesis this broad. Three incident narratives plus one informal, unpublished simulator run cannot separate "lack of feedback" from rivals such as crew-resource-management culture, the salience of particular displays, or training. The incapacitated-pilot case, Norman's key argument that automation is not the culprit, involves an uncommunicative captain and a steep social hierarchy. That shows that missing interaction can fail without automation. It does not show that automation with feedback would avoid the failures blamed on overautomation. The thought experiments assume that a human delegate would talk about increasing correction. Norman's own fuel-leak case and replication show crews missing exactly this kind of cue, so the "first officer would probably have said something" premise is untested. He announces formal studies of the thought experiments, but none appears here. The "new irony" (automation is not powerful enough) is hard to falsify as stated: any failure can be read as insufficient intelligence. The paper concedes that nobody knows how to build the required general self-monitoring or natural communication, so the remedy is a research direction, not a demonstrated design. Its reach to LLM agents rests on analogy. Autopilots are continuous controllers with physical saturation limits, and an agent's hidden repairs may not saturate in the same way. Capture caveat: pdftotext extracted the two-column layout with interleaved fragments, for example in the abstract and in the section on not knowing enough to mimic natural human interaction. Exact quotation needs care in reconstructing sentences across the scrambled order.

Recommended Next Action

In the workshop kb/work/operationalization-executability-automation, add a worked case built on Norman's delegation thought experiment (autopilot versus first officer). Use it to state the distinction between automating a delegated function and automating the full role of the previous executor. Include the mechanism that a task definition written for an executor drops functions the executor does not consume.