Self-Training-LLM report verification

Type: agentic-system-analyses/types/agentic-system-verification.md

Verification

The pinned reports support a bounded, source-grounded account of an operator-mediated model-building workflow. The reference-as-chosen scope defect identified previously is repaired in both runtime and epistemic. No material blocker remains. The boundary correctly separates the shipped improvement plane from external provider/trainer internals and from an autonomous conversational runtime.

I independently read the complete generation, training, response-generation and pairwise-evaluation drivers, trainer implementation, scorer, topic helper and utility module; document chunking, section/join helpers, dataset loaders, exact-instruction mapping and length normalization; prompt formatters and self-rating prompt; all four launchers; TinyLlama, Wikipedia and trainer configurations; README, requirements and ignore declarations. I inspected the frozen tracked tree and searched Python, scripts and configurations for alternate producers, consumers, evaluators and persistence paths. These reads supply SRC-1 implementation evidence; README and requested model behavior supply doctrine/design. Checkout HEAD matched the reviewed revision and tracked status was clean. No target code, tests, examples, provider calls or services were executed. No supplied run or experiment establishes successful operation or benefit.

Runtime records and correction. I checked all five model components, four stage routes, the knowledge-filtering claim and checkpoint behavioral-authority path against their anchors and downstream consumers. The scorer spelling mismatch, existing-question requirement and empty-test indexing support the conditional account in RT-RTE-data-generation. The positive-threshold branch and full-dataset fallback support RT-RTE-model-training. Direct model loading versus an independently selected TGI endpoint supports RT-RTE-response-generation and RT-BAP-checkpoint-consumption as wired rather than observed. Two-order parsing, exact cache reuse, omitted failures and counted-result division support RT-RTE-pairwise-evaluation. Model identifiers are not immutable weight pins; the classifier and provider adapters do not implement parameter training for those roles. The separate implementation, operation and benefit assessments remain supported.

The runtime correction answer agrees with SRC-1, uncertainty/scorer.py and uncertainty/train.py: reference override supplies the single pair and first multiple-preference pair, while the later constructor uses selected gold samples and the driver admits the returned list through extend. The guarantee paragraph, selection predicate and detailed route account all preserve that exception. Their load-bearing passages are retained in RT-RTE-model-training. The numerical admission mechanism survives the correction; neither reference naming nor preferred-role assignment certifies truth.

Memory records. I checked the topic embedder and every retained-object declaration through its producer, storage, transformation and consumer: document cache, question/context/test files, answer content and scores, known subset, responses, judge cache, results and the explicitly separated alternative topic-cache parts. MEM-OBJ-answer-content and MEM-OBJ-answer-scores distinguish generated payload/text from derived selection metadata. DPO conversion delivers prompt/chosen/rejected, not document/score columns; auxiliary SFT context is embedded separately. The parameter annotation correctly distinguishes durable numerical behavior from an inspectable rationale interface. MEM-OBJ-topic-caches preserves distinct text, vector, count and topic-list parts, unresolved caller/initialization facts and alignment limits. Existence-based reuse and exact cache keys establish neither authenticated lineage nor freshness. The training annotation defers reference-choice scope to the corrected runtime rather than asserting an all-pairs invariant.

Epistemic records and correction. I checked all six object declarations, four subroutes, the clustering claim and their ledger uses. EPI-RTE-document-transform follows imported assertions through deletion and selection without certifying source truth. EPI-RTE-answer-cleaning bounds its examples as source deductions and distinguishes lexical admission from factual acceptance. EPI-RTE-score-evidence separates NLI relations, self-rating and clustering; its noncontradiction mask is weaker than the entailment docstring. Early access to all_hallu_scores supports its alternative-scorer compatibility qualification. The score container explicitly delegates different evaluator domains rather than asserting a combined factuality measurement.

The epistemic correction answer is borne out by the same source branch and training consumer. EPI-RTE-preference-selection now restricts reference enforcement in its selection predicate, activation/effect and override explanation. The dependent ledger mismatch and bounded conclusion also state the later-pair exception. The previous epistemic blocker is resolved, not moved into a publication caveat. Candidate generation, retention, numerical selection and parameter adaptation still do not establish evidence-consuming acceptance or lifecycle integration of warranted factual assertions. The corrected account preserves the supported override and parameter-update pathway while withholding exercised self-improvement and learning benefit.

Workflow. The workflow account and its three annotations trace a conditional DPO experiment, not an observed successful run. The reference override statement is scoped to the single-preference SelfCheckGPT branch. Instruction-only DPO prompts, EOS handling, separately aligned serving, response-file delivery, retry exhaustion and terminal aggregate reporting have source support. The checkpoint completes training but is not an approved deployment; the evaluator neither supervises training nor vetoes checkpoint saving.

Relations and reconciliation. I checked every record's source anchors and evidence strength, annotation referents, the four explicit epistemic Part of: relations and the overlap dispositions in reconciliation. Document cleanup, answer cleaning and scoring are suboperations of generation; preference selection is a suboperation of training. Their fields and statuses remain local. Cache mappings versus article occurrences, persisted question records versus instruction text, and answer/score containers versus separately assessed parts do not require identity supersession. Preferences, parameters, model roles and the two design claims are distinct. No supersession amendment is asserted or needed.

Reconciliation's reference-choice convergence matches the current reports; its previous cross-report conflict no longer applies. Workflow's single-pair scope and memory's delegation are compatible with both corrected records. The _qf producer/consumer path mismatch and exact-instruction remapping loss qualify dataset continuity rather than contradict a completed-run claim. Alternative scorer compatibility and inactive topic helpers remain distinct qualifications. The clustering mismatch is an internal doctrine-versus-implementation finding, not an unresolved analyst disagreement. No unresolved conflict lacks disposition. The supplied structural check reports none, so there is no structural failure to route. Runtime and epistemic supplied correction answers; memory and workflow answers were empty. Reconciliation's statements about what answers it received do not substitute for this independent review of the supplied answers.

Omission questions and stopping decision. The frozen tree, four launchers and producer/consumer dependencies motivated three consequential challenges: whether another shipped caller closes an automatic judge-to-trainer loop; whether the topic-vector family supplies active retrieval to generation; and whether alternate preference construction preserves uniform reference targets. Searches across SRC-1, uncertainty/, script/ and configs/, followed by direct driver/helper reads, support separate stage launches and the predefined-topic dispatch. The multiple-preference path is a shipped training branch, not an external hypothetical consumer; it supplies the exception now retained in both corrected records. The relevant passages are already available through RT-RTE-model-training, MEM-OBJ-topic-caches and EPI-RTE-preference-selection.

Further optional omission expansion stopped because these questions have supported dispositions and no identified material challenge remains unresolved. Example-list bodies, benchmark dataset-class internals, other model configurations and external provider/framework internals were not comprehensively audited here. Those alternatives do not establish absence or equivalence; the reports already withhold exhaustive prompt-content, alternative-pipeline, factuality, operational-success and causal-benefit conclusions. Their scoped uncertainty does not invalidate the predefined-topic/SelfCheckGPT builder account and is not an additional publication limit.

Blockers

none

Limits

none