LLM reliability

Type: kb/types/tag-readme.md

LLM output deviates from what the user intended for three distinct reasons — underspecification of the spec, error by the interpreter, and indeterminism in sampling — each diagnosed by a different question and each repaired at a different primary surface. This area covers that taxonomy, the detection and correction machinery (oracles, voting, verification), and architectural responses (separation, bounded context) for managing all three.

The Taxonomy

• LLM output deviation requires three-way diagnosis — the synthesis: the three diagnostic questions, why prompt narrowing, error correction, and sampling control target different relations and so cannot completely substitute for one another, and how to tell the three apart empirically
• agentic-systems-interpret-underspecified-instructions — source 1: the spec admits multiple valid interpretations; a property of the specification language that even a perfect interpreter faces
• out-of-spec-output-is-a-failure-of-the-interpreter-not-the-spec — source 2, interpreter failure: the output falls outside what the spec allows; a property of the interpreter, with the worked failure catalogue
• traditional-software-can-bracket-executor-conformance-llm-systems — the foundation under the taxonomy: classical stacks can assume executor conformance and unique meaning, LLM systems can assume neither, so error analysis needs three questions where programming needed one
• execution-indeterminism-is-a-property-of-the-sampling-process — source 3: the same prompt gives different outputs across runs; a property of the sampling process, theoretically eliminable
• llm-debugging-starts-with-retry-versus-rewrite-triage — the operational move: which remedy to reach for first, for the two-source case

Error Correction Theory

• error-correction-works-above-chance-oracles-with-decorrelated-checks — the core theory: error correction is viable when oracles have discriminative power (TPR > FPR) and checks are decorrelated; amplification cost scales with 1/(TPR-FPR)²
• systematic-prompt-variation-serves-verification-and-diagnosis-not-explanatory-reach-testing — controlled framing changes do two different jobs here: decorrelate weak checks for verification and expose brittleness under semantically fixed prompts; distinct from Deutsch's explanatory-reach test

Oracle Theory

• oracle-strength-spectrum — oracle strength as a gradient from hard (deterministic) to no oracle (vibes); the engineering move is to harden oracles progressively
• reliability-dimensions-map-to-oracle-hardening-stages — Rabanser et al.'s four reliability dimensions each target a different oracle question; each can be hardened independently
• the-augmentation-automation-boundary-is-discrimination-not-accuracy — crossing from augmentation to automation requires per-instance discrimination, which is empirically stagnant; external oracle construction is the practical path
• knowledge-storage-does-not-imply-contextual-activation — relevant knowledge can be present but remain unelicited; activation failure appears when probe retrievability is high but spontaneous emergence is low
• elicitation-requires-maintained-question-generation-systems — strategies for closing the activation gap, ordered by expertise required; composes probes into maintained review architectures
• the-boundary-of-automation-is-the-boundary-of-verification — synthesis: three independent lines of evidence (oracle theory, labor economics, frontier-lab predictions) converge on verification cost as the structural determinant of automation
• evaluation automation is phase-gated by comprehension — phase model inside evaluation loops: automation only generalizes after manual comprehension and calibrated specification produce discriminative judges

Aggregation & Correction

• synthesis-is-not-error-correction — merging agent outputs propagates errors; voting discards minorities and corrects them; the aggregation operation must match the decomposition structure

Architectural Responses

• scheduler-llm-separation-exploits-an-error-correction-asymmetry — separation works because bookkeeping admits cheap error correction (hard oracles) while semantic work resists it; mixing forces bookkeeping onto the expensive substrate (also computational-model)
• specification-level-separation-recovers-scoping-before-it-recovers-error-correction — OpenProse-like DSLs recover frame isolation before gaining hard-oracle bookkeeping; an intermediate regime (also computational-model)

Related notes in other areas

• enforcement-without-structured-recovery-is-incomplete (kb-design, learning-theory) — the enforcement gradient covers detection and blocking but not recovery; oracle strength constrains viable recovery strategies
• semantic-review-catches-content-errors-that-structural-validation-cannot (kb-maintenance) — four semantic checks that are decorrelated weak oracles for content errors
• spec-mining-as-codification (learning-theory) — the manufacturing step: extracting deterministic checks from observed behavior to construct oracles
• silent disambiguation is the semantic analogue of tool fallback (observability, computational-model) — adjacent distinction: some bad outcomes come from hidden semantic recovery after an ambiguous spec, not from interpreter failure inside a clear spec

Sources

• Ma et al. (Sep 2025) — Prompt Stability in Code LLMs — empirical evidence: separates all three phenomena methodologically; performance-stability decoupling confirms they are independent

Related Tags

• learning-theory — oracle and verification theory originated there; this area applies it specifically to LLM output deviations
• computational-model — the scheduling architecture that separation notes describe; error correction explains why it works

Other tagged notes

• A bare writing prompt does not determine its intended contribution - Separates the contribution a bare writing prompt leaves underdetermined from empirical claims about how experts and LLMs supply the missing purpose.
• A goal-holding interpreter fails soft, and its workarounds tax a bounded budget - A procedure compiles its goal away, so a blocked step fails loud and hard; an interpreter holds the goal and re-routes, so failures are absorbed as a per-encounter tax on bounded capacity — silent, accumulating, and softly saturating
• Brainstorming: how to test whether pairwise comparison can harden soft oracles - Staged test plan for whether pairwise comparison improves soft-oracle properties (discrimination, stability, calibration) in LLM evaluation loops
• Brainstorming: maintainability oracles for agentic development - Explores candidate signals, calibration experiments, authority levels, and workflow placements for evaluating maintainability in agent-generated code
• Cheap generation breaks text volume as an effort signal - When text is cheap to expand but costly to verify, length stops evidencing author effort and can instead warn that the reviewer inherits unperformed checking
• Context contamination operates below an agent's compliance reasoning - A controlled test found fine-grained stance drift despite explicit detection and refusal; exclusion guarantees non-exposure, while instruction-level mitigation remains an empirical question
• Generation confidence does not by itself certify soundness - Distinguishes next-token probability from factual truth and inferential validity: confidence can support correctness decisions only after task-specific validation, and high-assurance acceptance still needs a separate check
• LLM generation can hide a relaxed goal where human writing exposes a stall - An LLM can ship fluent output after silently relaxing an unmet goal, while human composition may expose the same gap as a stall; a conjectural mechanism for why readers inherit the check
• Memory-backed personalization can look like model improvement - Distinguishes user-specific gains supplied by retained intent from gains in the model that interprets the assembled context.
• Reasoning production is not reasoning evaluation - Review and critique systems need independent process-validity checks because a model can substitute answer reconstruction for reasoning evaluation
• Review automation should target verifiable subroles before reviewer identity - Scholarly-review automation should decompose reviewer work into separately verifiable subroles before giving an AI system reviewer-level authority
• Topology, isolation, and verification form a causal chain for reliable agent scaling - Topology, isolation, and verification may form a strict dependency chain rather than independent design choices — tested against the simpler account that good decomposition implies the other two
• Weakly discriminated qualities tend to be underselected - Statistical conjecture: under named proposal-selection conditions, unequal oracle discrimination yields unequal enrichment; absolute degradation needs an additional directional mechanism