Out-of-spec output is a failure of the interpreter, not the spec

Type: kb/types/note.md · Tags: llm-reliability

Real LLMs produce outputs outside the space of valid interpretations. The spec rules them out, but the LLM fails to comply. This is not underspecification (where multiple outputs are valid) and not indeterminism (sampling noise across runs). It is interpreter failure: the gap between what a conforming interpreter of the spec's public meaning would do and what a real LLM does.

Calling it a failure is a claim about a role, not about the computation. The forward pass computes its own function flawlessly, the way a buggy compiler binary executes its own instructions flawlessly; the fault is non-conformance to what the spec's words publicly mean — the same norm a compiler is held to by a language standard. Classical stacks can bracket that norm because compilers honor it reliably; LLM systems cannot. And the norm is the spec's public meaning under a competent reading, never the author's private intent: an unwanted output the words admit is underspecification, the author's problem — not this phenomenon.

Examples: - Constraint violation: "Output JSON only" → LLM produces markdown with a JSON block - Hallucination: "Summarise this document" → LLM includes facts not in the document - Bookkeeping failure: on ConvexBench's mechanically verifiable convexity-classification task, reported performance falls from F1=1.0 at depth 2 to approximately 0.2 at depth 100 - Content bias: reasoning accuracy varies with semantic content rather than logical structure, producing errors on valid syllogisms with unfamiliar premises - Emotional prompt sensitivity: Ma et al. hold functional specifications fixed across semantically equivalent variants under a uniform 16-sample decoding policy. Under valence×arousal conditions, they report that emotional prompting mildly reshapes correctness and calibration for some models, with effects varying by model and family.

In each case, a conforming interpreter given the same spec would not make the error. The spec is sufficient; the interpreter is not.

Not every measured sensitivity establishes interpreter failure. In the pinned Mazur position-bias benchmark, the median of 27 judge models changed its canonical winner in 44.8% of decisive swapped-order sibling-edit pairs. The retained source material establishes order sensitivity, but not a judging contract or public-meaning norm that makes one canonical winner the only valid interpretation. Without that premise, the result could reflect underspecification, sampling variation, or non-conformance; it cannot by itself distinguish them.

Why this matters as a distinct claim

The idealised two-phenomena model implicitly assumes a conforming interpreter — one that always lands within the valid interpretation space. This is a useful simplification for reasoning about system design, but it leaves out a prominent practical failure mode: the LLM just getting it wrong. The examples above establish existence and variety, not a comparative prevalence rate.

The remedy is fundamentally different from the other two phenomena. Narrowing the spec (underspecification remedy) can make things worse by overloading context. Sampling control (indeterminism remedy) is irrelevant — a deterministic LLM still fails as an interpreter. The correct remedies are error detection and correction: validation, oracles, voting, guardrails, and architectural separation that moves error-prone operations to reliable substrates.

This is also one phenomenon that makes discrimination — knowing per-instance whether the output is wrong — the binding constraint on automation. If LLMs were conforming interpreters, the only question would be which valid interpretation they chose. Because they're not, you also need to detect when they've left the valid space entirely.

Relevant Notes: