KBLaM report verification

Type: agentic-system-analyses/types/agentic-system-verification.md

Verification

The supplied reports support a bounded implementation account, with no remaining publication blocker. All supplied members, correction answers, structural findings and the previous verification were read completely. Each of the four correction answers was checked against the current text and frozen source rather than accepted as proof of repair. The structural report-check says none; there is no structural failure to route.

SRC-1's checkout HEAD matches the boundary, and its working tree is clean. The whole-system classification remains supported: shipped producers, training, retained-state consumers and evaluators are included rather than treated as external applications. This is a research knowledge/context system, not an enclosing agent runtime. No evidence requires changing the frozen target or boundary kind. Inspection was source-only; no target code, tests, examples, providers or benchmarks were executed. Stored notebook outputs were not used as run evidence.

Runtime and workflow dispositions

  • RT-CMP-llama3-model, RT-CMP-phi3-model, RT-CMP-kb-encoder and RT-CMP-gpt-session distinguish numerical model state, projections, provider messages and configuration. Independent reads of SRC-1, src/kblam/kb_encoder.py, src/kblam/gpt_session.py and the training parameter-selection bodies in experiments/train.py support the mutability qualifications. The inspected driver freezes model parameters before selectively enabling query heads; this is not a restriction on arbitrary library users. External asset revisions and mutable deployment identities remain unresolved as stated.
  • RT-RTE-kb-attention and RT-BAP-knowledge-attention are supported by independent inspection of SRC-1, src/kblam/models/llama3_model.py and src/kblam/models/phi3_model.py. Configured layers concatenate compatible KB keys/values, normalize competing attention scores and aggregate values. Llama's optional pruning gathers selected vectors but returns a leading score slice; the reports correctly withhold equivalent-output and relevance guarantees. Phi's inspected attention has no corresponding pruning branch. The guarantee is conditional insertion, not truth or successful operation under every configuration.
  • RT-RTE-direct-inference remains correctly qualified as an afforded interface with a defective convenience-loader connection, distinct from the directly wired tensor consumer. SRC-1, src/kblam/utils/model_utils.py passes four positional arguments to the processor, while src/kblam/models/kblam_processor.py accepts only tokenizer and encoder arguments positionally. The processor's tensor-list recognition and the encoder's key/value layout and NumPy conversion also support the memory annotation's compatibility limit. Neither report asserts successful public-helper execution.
  • The runtime correction resolves the earlier central reachability defect. RT-RTE-accuracy-generation, the ordinary invocation account, RT-RTE-evaluation-modes, training read-back and checkpoint consumption now explicitly require source repair before unchanged evaluator dispatch can occur. Independent full inspection of SRC-1, src/kblam/utils/data_utils.py confirms augment_row and no aug_row alias; experiments/eval.py imports and internally calls the latter. The retained minimum passages establish the mismatch. A compatible external setup alone is no longer represented as sufficient. The wired statuses describe inspected internal progression, with operation separately uninspected.
  • The workflow correction carries the same prerequisite through its account and accuracy annotation. Independent inspection of SRC-1, experiments/eval.py's loader, accuracy function and CLI, together with src/kblam/utils/train_utils.py, confirms the subsequent sequence and limits: sampled row order supplies question/attention labels; generation is bounded to 60 new tokens; text precedes metric persistence; the CLI discards the return; oversized requests raise; uncached equal-size range input conflicts with indices.shape; fixed layer-file expectations, minimum top-five population and batch-size assumptions constrain completion. This is a multi-step computation, not evidence of child-agent delegation, review or automated recovery.
  • RT-RTE-evaluation-modes, EPI-RTE-answer-cleanup and EPI-RTE-metric-inputs accurately separate numerical KB, textual ICL and zeroshot inputs from downstream extraction and measurement. SRC-1, experiments/eval.py and src/kblam/utils/eval_utils.py independently confirm the unsupported topk_size argument, 150-token helper bound, global marker replacement, question splitting, regex extraction, optional sorry omission and refusal substring classification. Their source deductions are not observed error rates. Attention rank, text similarity and provider judgments do not constitute one factual oracle or an answer-release gate.

Production, retained objects and epistemic dispositions

  • RT-RTE-dataset-production, EPI-RTE-template-qa and EPI-RTE-augment-qa are supported by SRC-1, dataset_generation/gen_synthetic_data.py and src/kblam/utils/data_utils.py. JSON/dataclass construction admits structurally usable entities; templates preserve property strings under aligned inputs; model rewrites have output-dependent semantics. Existing-file reload skips, append persistence, caught item failures and the misspelled perturbe_names call are source-visible. The two epistemic routes are genuine parts of the dataset-production container, not replacements for its embedding/split responsibilities. The remaining guidance disagreement has the local disposition below.
  • MEM-OBJ-entity, MEM-OBJ-data-point, MEM-OBJ-embd-key and MEM-OBJ-embd-value preserve different retained inputs and actual consumers. Independent reads of both embedding/split scripts, the generator, encoder and retriever substantiate their text versus numerical forms, separate key/value sources, positional joining and cached versus uncached projection. Editing text need not change consumed cached vectors. JSONL output and downstream JSON-array input do not establish a turnkey pipeline. Historical correspondence and factual correctness are not inferred from filename or length agreement.
  • RT-RTE-adapter-training, MEM-OBJ-encoder, MEM-OBJ-kblam-config and RT-BAP-checkpoint-consumption have positive source support in SRC-1, experiments/train.py, experiments/eval.py, src/kblam/utils/train_utils.py and src/kblam/models/kblam_config.py. Independent inspection traced supervised targets through weighted loss, gradients, AdamW and periodic parameter/configuration saves to explicit later loads. The optimizer admits steps without validation-selected checkpoint promotion. Sibling encoder storage and differing configuration names support the resume mismatch. Independent outlier resampling does not enforce missing-answer membership. Prefix truncation removes tokens rather than preserving a reloadable summary. Numerical updates support the marked dispositional adaptation account, not demonstrated capacity improvement, formulated criticism, reflection or exercised self-improvement.
  • MEM-OBJ-attention-weights and MEM-RTE-result-cleanup correctly distinguish raw snapshots, reduced metrics and destructive retention control. SRC-1, both attention writers, experiments/eval.py's accuracy sweep and src/kblam/utils/convert.py independently confirm conditional snapshot writes, metric reload, whole-directory deletion/recreation and later JSON-to-CSV consumption. Cleanup is distinct from answer generation and does not withdraw a theory or KB row. Output-path aliasing and failure before final summary persistence remain explicit, not hidden completion guarantees.
  • EPI-OBJ-entity, EPI-OBJ-data-point, EPI-OBJ-extended-qa and EPI-OBJ-model-output preserve the differences between source assertions, supervision, unchecked rewrites and candidate answers. Source passages and independently inspected consumers support their qualified content relations. Neither dense encoding nor generated fluency establishes checked entailment, accepted factual novelty or lifecycle integration. The inventory and ledger do not erase the numerical machinery merely because its proposition-level contents are opaque.
  • The memory and epistemic judgment-part corrections resolve the earlier missing comparison units. MEM-OBJ-eval-example and EPI-OBJ-eval-example remain valid containers, now with separately declared MEM-OBJ-reference-judgment, MEM-OBJ-open-judgment, EPI-OBJ-reference-judgment and EPI-OBJ-open-judgment. Complete independent reads of SRC-1, experiments/output_scorer.py and experiments/output_scorer_open_ended.py confirm differing fields, evaluator domains, admission, persistence timing, rationale and reporting consumers. Reference judgments return as a list, save after the batch and feed a mean; they retain no provider reason. Open-ended judgments retain parsed score/reason per successful iteration; normal completion implicitly returns None, whereas caught failure returns a prefix list. The current memory text now correctly withholds normal completed-batch aggregate reporting. The epistemic inventory and ledger connect each evaluator to its corresponding operative object rather than one uniform judgment identity.
  • EPI-RTE-open-judgment correctly separates rubric-based checking and retention from acceptance. Its source passages and the complete scorer inspection establish evidence-relative/common-sense guidance, regex/float admission without range enforcement, per-record append and the exception-only return. GPT4 defaults fail the inspected helper allowlist; constructor initialization overwrites subclass system-message/seed assignments, while user-prompt rubrics remain supplied. A stored reason is model-produced assessment content, not independent warrant or demonstrated later inference influence.
  • RT-CLM-knowledge-augmentation and RT-CLM-base-preservation accurately attribute SRC-1, README.md, independently read in full. Conditional KB insertion/bypass supports narrower implementation findings, not measured scaling, guaranteed grounding or universal exact-output equality. The README's research scope and incorrect-answer warning are preserved. No analyst declares an evidenced absence or promotes implementation into observed or causal evidence.

The reconciliation is supported as identity mapping, not value arbitration. Its three amendments concern the same Entity and the two same script-local judgment referents across lenses. Their schemas, storage and consumer paths substantiate supersession without erasing epistemic qualifications. All eight explicit Part of: relations were checked: the four judgment parts belong to their respective containers; template/augmentation and cleanup/metric routes are contained portions of the two runtime route containers. Neither key/value arrays nor saved parameters are duplicate live components. The broader DataPoint and answer/judgment containers overlap without identical scope; no split supersession is needed because their findings remain valid alongside separately assessed parts.

The prior CLI and judgment-part objections are repaired, not merely declined. The prior open-ended aggregate limit is also resolved in the current memory wording, so it need not remain an unresolved finding. Reconciliation retains the extension-guidance disagreement with affected records, both positions, source evidence and prevented conclusion. Independent inspection confirms the specialists' attribute mismatch. Runtime literally says the guidance requests evidence-only QA; it does not assert successful delivery or compliance. This supports a local declared-versus-delivered limit rather than a blocker against an unsupported truth-control value. No conflict remains without a disposition.

Consequential omission questions were whether shipped alternate callers repair or bypass the identified entry defects; whether score/reason artifacts feed automatic knowledge revision or checkpoint adoption; and whether broad containers hide judgment parts with different authority or rationale. The search basis was SRC-1's frozen tracked tree, script dispatch and producer/consumer dependencies, including searches across dataset_generation/, src/, experiments/ and tests/, not just listed anchors. Focused Makefile inspection confirms shipped evaluator callers use the same module; initial notebook source cells also import it and reference stale names. These callers do not establish an operational repair. Separate direct-model consumers preserve a bounded numerical inference mechanism despite wrapper defects.

Independent inspection of training optimizer/save bodies, dataset admission, both complete scorer files and the result converter supports the bounded descriptions of score reporting and parameter admission. No score-to-KB or score-driven successor decision is wired in those inspected consumers. That is not a repository-wide or provider-internal absence claim. The rationale-bearing open-ended path is now represented by separate objects and its own evaluator route, so it is no longer concealed by the scalar-reference container.

Further optional omission expansion stopped because these selected challenges have supported dispositions, and no identified material challenge remains unassigned. Additional model internals, corpus rows and external assets are not reasonably likely to change the retained narrow implementation conclusions enough to justify broader inspection. This does not establish exhaustive semantic coverage, successful notebook operation, complete corpus lineage or comparative benefit. Missing execution evidence constrains operation, not the positive source-wired mechanisms.

Blockers

none

Limits

  • RT-RTE-dataset-production's evidence-only extension guidance is supported as declared intent, not delivered control. EPI-RTE-augment-qa, the memory dataset annotation and the reconciliation conflict retain the differing qualifications. SRC-1, dataset_generation/gen_synthetic_data.py assigns system_prompt, while src/kblam/gpt_session.py consumes system_msg; their minimum passages establish the mismatch. Withhold a conclusion that this system constraint reaches the extension call or guarantees semantic preservation. The literal declared request, structural admission and field persistence remain supported; actual model compliance is unobserved rather than disproved.
  • Exact lineage, completed pipeline operation and benefit remain outside the bounded account. MEM-OBJ-embd-key, MEM-OBJ-embd-value, MEM-OBJ-encoder and the Source register identify unverified positional correspondence, historical checkpoint/provider identity and unavailable assets. RT-RTE-accuracy-generation and the workflow account explicitly require source repair before unchanged evaluator entry. These constraints preserve the implementation transformation/consumer account but prevent corpus-correctness, historical-reconstruction, reliable unchanged-execution, exercised-improvement or causal-benefit comparisons.