LLM reliability

Type: kb/types/tag-readme.md

LLM output deviates from what the user intended for three distinct reasons — underspecification of the spec, error by the interpreter, and indeterminism in sampling — each a property of a different part of the system and each needing a different remedy. This area covers that taxonomy, the detection and correction machinery (oracles, voting, verification), and architectural responses (separation, bounded context) for managing all three.

The Taxonomy

• LLM output deviation has three sources with non-substitutable remedies — the synthesis: why the three are properties of different objects, why prompt narrowing, error correction, and sampling control cannot stand in for each other, and how to tell the three apart empirically
• agentic-systems-interpret-underspecified-instructions — source 1: the spec admits multiple valid interpretations; a property of the specification language that even a perfect interpreter faces
• out-of-spec-output-is-a-failure-of-the-interpreter-not-the-spec — source 2, interpreter failure: the output falls outside what the spec allows; a property of the interpreter, with the worked failure catalogue
• traditional-software-can-bracket-executor-conformance-llm-systems — the foundation under the taxonomy: classical stacks can assume executor conformance and unique meaning, LLM systems can assume neither, so error analysis needs three questions where programming needed one
• execution-indeterminism-is-a-property-of-the-sampling-process — source 3: the same prompt gives different outputs across runs; a property of the sampling process, theoretically eliminable
• llm-debugging-starts-with-retry-versus-rewrite-triage — the operational move: which remedy to reach for first, for the two-source case

Error Correction Theory

• error-correction-works-above-chance-oracles-with-decorrelated-checks — the core theory: error correction is viable when oracles have discriminative power (TPR > FPR) and checks are decorrelated; amplification cost scales with 1/(TPR-FPR)²
• systematic-prompt-variation-serves-verification-and-diagnosis-not-explanatory-reach-testing — controlled framing changes do two different jobs here: decorrelate weak checks for verification and expose brittleness under semantically fixed prompts; distinct from Deutsch's explanatory-reach test

Oracle Theory

• oracle-strength-spectrum — oracle strength as a gradient from hard (deterministic) to no oracle (vibes); the engineering move is to harden oracles progressively
• reliability-dimensions-map-to-oracle-hardening-stages — Rabanser et al.'s four reliability dimensions each target a different oracle question; each can be hardened independently
• the-augmentation-automation-boundary-is-discrimination-not-accuracy — crossing from augmentation to automation requires per-instance discrimination, which is empirically stagnant; external oracle construction is the practical path
• knowledge-storage-does-not-imply-contextual-activation — relevant knowledge can be present but remain unelicited; activation failure appears when probe retrievability is high but spontaneous emergence is low
• elicitation-requires-maintained-question-generation-systems — strategies for closing the activation gap, ordered by expertise required; composes probes into maintained review architectures
• the-boundary-of-automation-is-the-boundary-of-verification — synthesis: three independent lines of evidence (oracle theory, labor economics, frontier-lab predictions) converge on verification cost as the structural determinant of automation
• evaluation automation is phase-gated by comprehension — phase model inside evaluation loops: automation only generalizes after manual comprehension and calibrated specification produce discriminative judges

Aggregation & Correction

• synthesis-is-not-error-correction — merging agent outputs propagates errors; voting discards minorities and corrects them; the aggregation operation must match the decomposition structure

Architectural Responses

• scheduler-llm-separation-exploits-an-error-correction-asymmetry — separation works because bookkeeping admits cheap error correction (hard oracles) while semantic work resists it; mixing forces bookkeeping onto the expensive substrate (also computational-model)
• specification-level-separation-recovers-scoping-before-it-recovers-error-correction — OpenProse-like DSLs recover frame isolation before gaining hard-oracle bookkeeping; an intermediate regime (also computational-model)

Related notes in other areas

• enforcement-without-structured-recovery-is-incomplete (kb-design, learning-theory) — the enforcement gradient covers detection and blocking but not recovery; oracle strength constrains viable recovery strategies
• semantic-review-catches-content-errors-that-structural-validation-cannot (kb-maintenance) — four semantic checks that are decorrelated weak oracles for content errors
• spec-mining-as-codification (learning-theory) — the manufacturing step: extracting deterministic checks from observed behavior to construct oracles
• silent disambiguation is the semantic analogue of tool fallback (observability, computational-model) — adjacent distinction: some bad outcomes come from hidden semantic recovery after an ambiguous spec, not from interpreter failure inside a clear spec

Sources

• Ma et al. (Sep 2025) — Prompt Stability in Code LLMs — empirical evidence: separates all three phenomena methodologically; performance-stability decoupling confirms they are independent

Related Tags

• learning-theory — oracle and verification theory originated there; this area applies it specifically to LLM output deviations
• computational-model — the scheduling architecture that separation notes describe; error correction explains why it works

Other tagged notes

• A bare writing prompt does not determine its intended contribution - Separates the contribution a bare writing prompt leaves underdetermined from empirical claims about how experts and LLMs supply the missing purpose.
• A goal-holding interpreter fails soft, and its workarounds tax a bounded budget - A procedure compiles its goal away, so a blocked step fails loud and hard; an interpreter holds the goal and re-routes, so failures are absorbed as a per-encounter tax on bounded capacity — silent, accumulating, and softly saturating
• An LLM's generation confidence tracks typicality, not soundness - An LLM's next-token confidence measures how typical a continuation is, not whether it's true or valid; the two are decoupled, so soundness can't be read off confidence and needs a separate check
• Brainstorming: how to test whether pairwise comparison can harden soft oracles - Staged test plan for whether pairwise comparison improves soft-oracle properties (discrimination, stability, calibration) in LLM evaluation loops
• Brainstorming: maintainability oracles for agentic development - Explores candidate signals, calibration experiments, authority levels, and workflow placements for evaluating maintainability in agent-generated code
• Context contamination operates below an agent's compliance reasoning - A controlled test found fine-grained stance drift despite explicit detection and refusal; exclusion guarantees non-exposure, while instruction-level mitigation remains an empirical question
• LLM generation relaxes a goal it can't satisfy and hides the constraint a human writer stalls on - A human writer stalls at the constraint they can't satisfy; an LLM instead ships fluent output that looks solved but silently drops it — hiding the error, so the check falls on the reader
• Reasoning production is not reasoning evaluation - Review and critique systems need independent process-validity checks because a model can substitute answer reconstruction for reasoning evaluation
• Review automation should target verifiable subroles before reviewer identity - Scholarly-review automation should decompose reviewer work into separately verifiable subroles before giving an AI system reviewer-level authority
• Topology, isolation, and verification form a causal chain for reliable agent scaling - Topology, isolation, and verification may form a strict dependency chain rather than independent design choices — tested against the simpler account that good decomposition implies the other two
• Weakly discriminated qualities tend to be underselected - Conjecture separating available model capability from selection: qualities weakly distinguished by the actual acceptance oracle lose to strongly verified objectives