Out-of-spec output is a failure of the interpreter, not the spec
Type: kb/types/note.md · Tags: llm-reliability
Real LLMs produce outputs outside the space of valid interpretations. The spec rules them out, but the LLM fails to comply. This is not underspecification (where multiple outputs are valid) and not indeterminism (sampling noise across runs). It is interpreter failure: the gap between what a conforming interpreter of the spec's public meaning would do and what a real LLM does.
Calling it a failure is a claim about a role, not about the computation. The forward pass computes its own function flawlessly, the way a buggy compiler binary executes its own instructions flawlessly; the fault is non-conformance to what the spec's words publicly mean — the same norm a compiler is held to by a language standard. Classical stacks can bracket that norm because compilers honor it reliably; LLM systems cannot. And the norm is the spec's public meaning under a competent reading, never the author's private intent: an unwanted output the words admit is underspecification, the author's problem — not this phenomenon.
Examples: - Constraint violation: "Output JSON only" → LLM produces markdown with a JSON block - Hallucination: "Summarise this document" → LLM includes facts not in the document - Bookkeeping failure: tracking compositional depth (fully specified, one correct answer) → F1 collapses from 1.0 to 0.2 at depth 100 despite short context - Content bias: reasoning accuracy varies with semantic content rather than logical structure, producing errors on valid syllogisms with unfamiliar premises - Emotional prompt sensitivity: Ma et al. show that semantically equivalent prompts with different emotional framing produce systematic performance degradation — bias, not noise, since the functional spec is unchanged - Judgment instability under reordering: the Mazur position-bias benchmark shows 27 LLM judges flip their pairwise winner in 44.8% of decisive cases when candidate display order is swapped — identical content in both views, interpretation driven by an ordering cue the spec does not mention
In each case, a conforming interpreter given the same spec would not make the error. The spec is sufficient; the interpreter is not.
Why this matters as a distinct claim
The idealised two-phenomena model implicitly assumes a conforming interpreter — one that always lands within the valid interpretation space. This is a useful simplification for reasoning about system design, but it leaves out the failure mode that dominates practical experience: the LLM just getting it wrong.
The remedy is fundamentally different from the other two phenomena. Narrowing the spec (underspecification remedy) can make things worse by overloading context. Sampling control (indeterminism remedy) is irrelevant — a deterministic LLM still fails as an interpreter. The correct remedies are error detection and correction: validation, oracles, voting, guardrails, and architectural separation that moves error-prone operations to reliable substrates.
This is also the phenomenon that makes discrimination — knowing per-instance whether the output is wrong — the binding constraint on automation. If LLMs were conforming interpreters, the only question would be which valid interpretation they chose. Because they're not, you also need to detect when they've left the valid space entirely.
Relevant Notes:
- agentic systems interpret underspecified instructions — foundation: the idealised two-phenomena model this note extends; covers underspecification and indeterminism in depth
- execution indeterminism is a property of the sampling process — sibling: the second phenomenon, a property of the execution engine
- LLM output deviation has three sources with non-substitutable remedies — synthesis: the three-source taxonomy this note is part of, and why only error detection and correction address this source
- LLM reliability — parent area: deviation sources plus the verification and correction machinery
- traditional software can bracket executor conformance; LLM systems cannot — grounds: why the failure attribution is legitimate — the interpreter is held to a role norm, like a compiler to a language standard, and for LLMs that norm cannot be bracketed
- error-correction-works-above-chance-oracles-with-decorrelated-checks — remedy: the general theory of error correction applicable to interpreter failures
- scheduler-llm-separation-exploits-an-error-correction-asymmetry — architectural remedy: moving error-prone bookkeeping to a reliable substrate
- Ma et al. (Sep 2025) — Prompt Stability in Code LLMs — evidenced-by: emotional prompt variation produces systematic performance degradation (bias) on functionally identical tasks
- Mazur position-bias benchmark — evidence: judge-layer interpreter failure — 27 LLMs flip their pairwise winner in 44.8% of decisive cases under display-order swap alone; parallel peg to Ma et al. at the LLM-as-judge layer (preprint-tier, sibling-edit surface)