verbatim-citation-gate synthesis
Type: agentic-system-analyses/types/agentic-system-synthesis.md
Bounded synthesis
Verbatim-citation-gate is a returning citation-auditing tool, not an enclosing agent runtime. It checks whether a supplied quotation occurs in supplied documents and optionally asks a caller-selected model whether the quotation supports a claim. This code-grounded account covers the Python library, its exported alternatives, shipped example consumers and evaluation machinery at commit 2113d603031a554ed901a44648307bd3b711ea10, with an applicability cutoff of 2026-11-08. SRC-1 is the frozen repository identified in the boundary and Source register. Retrieval, citation generation, provider internals and downstream answer acceptance are outside this subsystem boundary. Source inspection establishes wiring; no target execution, live-model trace or independently reproduced benchmark was supplied.
From supplied citation to deterministic disposition
The caller supplies a claim, quotation, document key, document dictionary and optional model callback. Python owns the next-step decision. The ordinary audit_citation route normalizes strings and searches the attributed document first. A qualifying match returns found; only if that search fails does occurrence in another supplied document return misattributed. Otherwise it returns not_found. The dispatcher immediately returns any non-found result without invoking the callback. This is an implemented veto over callback invocation on this entry point, not an answer-delivery veto or a universal fabrication detector (RT-RTE-audit-citation, EPI-RTE-quote-check).
The match is contiguous and boundary-checked in normalized text, not byte-identical text. Normalization repairs typography, folds case and selected homoglyphs, and treats quantitative punctuation specially. Those rules protect some distinctions but do not establish semantic preservation. The source-derived example 25-30 mg versus 25 30 mg becomes the same normalized string; the encoded occurrence predicate cannot distinguish their meanings. A whole-word one-word quotation can also pass because the gate has no minimum evidence-length or relevance criterion. A match establishes an algorithmic relationship among supplied strings, not source authenticity, source truth or evidential sufficiency (EPI-RTE-quote-check).
Public alternatives have different contracts. Direct judge_support calls bypass occurrence checking; direct deterministic checks omit semantic judgment. The offset API validates half-open Python character indices and, only when supplied, chunk containment. Its no-quote branch accepts a valid contentful slice even if it cuts through a word or number. With a quotation, boundaries are checked relative to the slice, not the whole document. Neither the ordinary quotation gate nor the combined auditor enforces chunk bounds. These are reachable optional routes, not violations of a mandatory system-wide chunk policy (RT-RTE-span-gate, RT-RTE-judge-support).
From occurrence to requested support judgment
Without a callback, the dispatcher stops at found, which must remain distinct from supports. With a callback, it passes a fixed skeptical system instruction and a user message containing the original claim, original quote and entire original cited document. It does not send normalized cache values, a selected evidence window, other documents or batch history. There is no library token budget, compaction or separate worker conversation. Provider-side context handling remains unknown (RT-RTE-judge-support, MEM-RTE-normalization-reuse).
The instruction asks for source-relative support at the claim's full strength: the same population, outcome and direction, with certainty no stronger than the source. Outside knowledge is inadmissible, and uncertainty should favor the weaker verdict. This is substantive evaluation policy delivered to a callback, not a deterministic semantic constraint. The package supplies neither a provider transport nor an exact model pin, and its synchronous callback has no library timeout, retry or cancellation control. Callback side effects and provider retention are not constrained by this small interface (EPI-CLM-citation-support, RT-CMP-llm-call, RT-BAP-judge-instruction).
A second code-level decision admits the callback's representation. The parser accepts the first parseable JSON object with one of four verdict labels. Caught callback exceptions and recognized parsing failures return unrelated, without retrying for a more favorable answer. But {"verdict":"supports"} passes despite missing reasoning and confidence; defaults fill those fields, and no confidence threshold controls acceptance. Non-string returns violate the documented callback contract and can fail during parsing outside the callback exception handler. Thus the source's blanket missing-field fail-closed claim exceeds its implementation. Vocabulary acceptance is not evidence-consuming acceptance of the underlying claim (EPI-RTE-verdict-admission, RT-CLM-closed-judge).
The direct judge returns reasoning, label and confidence. The ordinary dispatcher returns only the label, discarding the other fields. Neither persists semantic decisions for later audits. A consuming application therefore needs its own admission and delivery policy; the returned label alone supplies neither a retained rationale nor demonstrated semantic reliability (RT-RTE-audit-citation, EPI-OBJ-reasoning).
Shipped delivery and later use
The representative workflow is the four-item demo batch. It submits fixed citations, receives deterministic rejection or conditional callback judgment, prints each result and continues. A rejected item is not repaired or resubmitted. The callback is an offline constant supports stand-in that ignores the evidence strings. Its counter illustrates intended delegation boundaries, not observed model accuracy or provider token accounting. Delivery is stdout, not suppression of unsupported material in a generated answer. The workflow therefore establishes a multi-step tool integration, not a demonstrated autonomous agent workflow (WF-RTE-demo-batch).
Retained runtime state is chiefly a process-local LRU cache of raw string keys and normalized values, bounded to 4096 entries. Cache misses automatically admit computed values; later exact-key normalization requests reuse them. These symbolic values supply validation references to deterministic matching, not model-visible memories or accumulated support judgments. LRU eviction forgets entries, and changed text obtains another key. Wiring alone establishes neither measured acceleration nor improved decisions (MEM-OBJ-normalized, MEM-RTE-normalization-reuse).
Evaluation has a separate durable route. A builder imports arXiv abstracts, unwraps whitespace, filters entries and overwrites a JSON corpus. A later benchmark invocation requests that file and constructs labeled cases. Its expected answers come from case-generation rules, not independent semantic annotation. Expected-negative cases are dropped when their normalized text occurs anywhere in the corpus; that filter shares the tested normalizer but omits the production boundary predicate. Results therefore characterize the surviving constructed population, not arbitrary false quotations or production traffic (MEM-RTE-corpus-reload, EPI-RTE-benchmark-check).
The strongest improvement contribution is an author-reported reduction in suite false positives from 0.4254 to 0.1721 after normalization fixes, with detection reported unchanged at 1.0 on 1,221 bad citations. The retained output agrees with the reported totals. This is a bounded reported before/after result, not an independently authenticated execution or identified effect of each fix. Stage two is explicitly outside the benchmark. A frozen pre-change comparison and generating traces would be needed to strengthen the improvement attribution; a live-model evaluation would address a different, presently unresolved semantic question (EPI-CLM-benchmark-detection).
Reports retain aggregate counts and a truncated failure sample, not a complete execution history. Human inspection is afforded, but no executable report-to-policy update and later consumer is established. Trace-derived report construction is not itself trace learning or improved capacity for future action. Product changes instead have declared human merge/release authority and conditional automated publication; the separate tests workflow is not a declared dependency of publishing. These records do not establish an exercised self-improvement loop, while leaving human development and opaque provider changes as separate questions (MEM-OBJ-results, RT-RTE-release-admission).
For auditing supplied quote-style citations, the discriminating capacity is the deterministic pre-model rejection route followed by an optional, explicitly source-relative judgment request. Its limits follow the same progression: normalized occurrence is weaker than literal identity; requested judgment is weaker than verified support; and a returned classification is weaker than enforced user protection. Evidence that would change that assessment includes authenticated live-model judgments against independently labeled support cases, downstream traces showing how verdicts govern delivery, and code changes enforcing stricter response validation or a mandatory combined route. None follows from the present implementation alone (EPI-CLM-citation-support, RT-RTE-audit-citation).
Limitations
| Limitation | Affected SRC IDs and record citations | Inspected boundary | Conclusion prevented | Evidence that would resolve it |
|---|---|---|---|---|
| No independently observed execution or live-model evaluation | SRC-1; RT-RTE-audit-citation, RT-RTE-judge-support, WF-RTE-demo-batch | Library source and shipped constant-callback consumers | Exercised vetoes, actual semantic accuracy, prompt compliance, token savings and end-to-end user protection | Authenticated invocation traces, independently labeled live-model evaluations and downstream delivery traces |
| External provider and host responsibilities are unresolved | SRC-1; RT-CMP-llm-call, RT-BAP-judge-instruction, EPI-OBJ-docs | Callback interface and supplied current inputs; production retrieval, provider internals and answer acceptance excluded | Exact model stability, provider persistence or truncation, source authenticity and enclosing-application enforcement | A separately bounded capture of provider configuration, document provenance and host consumers |
| Benchmark generation and historical artifact provenance are not authenticated | SRC-1; MEM-OBJ-corpus, MEM-OBJ-results, EPI-CLM-benchmark-detection | Builder, driver, transformations and saved aggregates; corpus sampled rather than fully audited | Reproduced rates, complete corpus provenance, production recall or component-specific causal improvement | Code/corpus-bound generating traces, frozen pre-change results, controlled interventions and representative sampling |
| Later report-driven change and deployed release controls are not established | SRC-1; MEM-OBJ-results, RT-RTE-release-admission | Report output, declared governance and CI/publication definitions | A trace-fed durable behavior update, exercised learning or self-improvement, and actual deployment admission guarantees | Evidence-to-update lineage, retained changed artifact and later causal consumer; deployed approval and release records |
| Raw cache-key lineage is not separately inventoried in the partial lineage classification | SRC-1; MEM-OBJ-normalized, MEM-RTE-normalization-reuse; profile verification limit | Raw input keys and derived normalized values in the process-local cache | Complete part-by-part cache lineage comparison; extending compiled-value lineage to raw-key provenance | Separate raw-key provenance and admission inventory supported by caller-origin evidence |