Interpretation errors are failures of the interpreter

Type: note · Status: seedling · Tags: llm-interpretation-errors

Real LLMs produce outputs outside the space of valid interpretations. The spec rules them out, but the LLM fails to comply. This is not underspecification (where multiple outputs are valid) and not indeterminism (sampling noise across runs). It's a property of the interpreter itself: the gap between what a perfect interpreter would do and what a real LLM does.

Examples: - Constraint violation: "Output JSON only" → LLM produces markdown with a JSON block - Hallucination: "Summarise this document" → LLM includes facts not in the document - Bookkeeping failure: tracking compositional depth (fully specified, one correct answer) → F1 collapses from 1.0 to 0.2 at depth 100 despite short context - Content bias: reasoning accuracy varies with semantic content rather than logical structure, producing errors on valid syllogisms with unfamiliar premises - Emotional prompt sensitivity: Ma et al. show that semantically equivalent prompts with different emotional framing produce systematic performance degradation — bias, not noise, since the functional spec is unchanged

In each case, a perfect interpreter given the same spec would not make the error. The spec is sufficient; the interpreter is not.

Why this matters as a distinct claim

The idealised two-phenomena model implicitly assumes a perfect interpreter — one that always lands within the valid interpretation space. This is a useful simplification for reasoning about system design, but it leaves out the failure mode that dominates practical experience: the LLM just getting it wrong.

The remedy is fundamentally different from the other two phenomena. Narrowing the spec (underspecification remedy) can make things worse by overloading context. Sampling control (indeterminism remedy) is irrelevant — a deterministic LLM still makes interpretation errors. The correct remedies are error detection and correction: validation, oracles, voting, guardrails, and architectural separation that moves error-prone operations to reliable substrates.

This is also the phenomenon that makes discrimination — knowing per-instance whether the output is wrong — the binding constraint on automation. If LLMs were perfect interpreters, the only question would be which valid interpretation they chose. Because they're not, you also need to detect when they've left the valid space entirely.

Relevant Notes: