KBLaM synthesis
Type: agentic-system-analyses/types/agentic-system-synthesis.md
Bounded synthesis
From knowledge records to answering machinery
KBLaM is a research knowledge-augmentation system, not an enclosing autonomous-agent runtime. It represents external knowledge as numerical keys and values supplied to modified Llama or Phi transformer attention. This account covers the repository's shipped producers, retained records and embeddings, adapter training, reload paths, generation and evaluators at commit 69683ad096e022b58854fd88c0c58968c7d8c470, with an applicability cutoff of 2026-10-15. The evidence is SRC-1 implementation and repository-native design declarations. No target execution, benchmark, provider call or causal experiment supports this account; stored notebook outputs were not used. The whole-system boundary includes the shipped research loop, not external provider implementations or unshipped deployment hosts (boundary and evidence, Source register).
The write side begins with supplied records or GPT-generated entities. Entity properties become knowledge keys, descriptions and question/answer supervision. Templates preserve supplied property strings under aligned inputs; they do not independently verify those properties. Model-written question perturbations and extensions have output-dependent semantics. JSON construction and regex extraction admit structurally usable content without per-record human approval or factual acceptance. The extension's evidence-only system guidance exists as declared intent, but the helper assigns a different attribute from the one the chat client consumes. It must not be treated as delivered control or a semantic-preservation guarantee. This is the remaining cross-report qualification carried by report verification (MEM-OBJ-entity, MEM-OBJ-data-point, EPI-RTE-template-qa, EPI-RTE-augment-qa, verification limits).
Knowledge content and learned consumption machinery remain separate. An embedding command persists key and description vectors in paired NumPy files. Later drivers select rows and apply a loaded encoder's projections. Positional correspondence joins records and arrays; the loader does not authenticate their semantic alignment. Consequently editing record text need not change numerical knowledge when cached arrays remain selected. Encoder parameters persist separately from model parameters and symbolic KB configuration. None of these files establishes its own historical provenance or freshness (MEM-OBJ-embd-key, MEM-OBJ-embd-value, MEM-OBJ-encoder, MEM-OBJ-kblam-config).
Selection, attention and terminal delivery
Experiment code owns progression: it loads artifacts, selects rows, formats questions and invokes generation. The model owns token computation, not tool dispatch or research scheduling. Requested artifact loading is distinct from the driver's automatic delivery of selected context to the model. Driver selection is coarse—configured sizes, boundaries and random indices—rather than external semantic search. At configured layers, compatible KB keys and values are concatenated with text attention state and compete through normalized numerical scores. This establishes conditional insertion, not truth, relevance or observed answer dependence (RT-RTE-accuracy-generation, RT-RTE-kb-attention, RT-BAP-knowledge-attention).
Llama can additionally select top-k rows using current query/key scores. This is internal embedding-based selection, not a separate retrieval service or durable deletion of KB rows. Phi has no corresponding pruning branch in the inspected attention path. Llama's pruning gathers vectors but returns a leading score slice rather than the selected scores, so equivalent-output claims remain unsupported. Other routes serialize selected knowledge into ordinary text context for in-context learning, omit it for zeroshot generation, or accept compatible tensors directly. Processor loading is not a universal gate because direct callers can bypass it; the convenience loader and tensor-list interface have their own compatibility defects. Numerical KB input avoids text tokenization but remains subject to row budgets, layer frequency, numerical scale and attention competition (RT-RTE-kb-attention, RT-RTE-evaluation-modes, RT-RTE-direct-inference, memory's direct-inference annotation).
The representative workflow is the connected internal sequence behind the accuracy command: load artifacts, select a KB, project it, generate with a 60-new-token bound, save attention and decoded text, then compute attention-index metrics. The frozen CLI cannot enter this sequence unchanged: it imports aug_row, while the shipped utility defines augment_row without that alias. Source repair is required before dispatch, not merely compatible external assets. The reconstructed sequence therefore supports an implementation account conditional on repair, not a successful or reachable unchanged invocation (RT-RTE-accuracy-generation, workflow account).
Within that sequence, question positions supply expected KB-row labels. Top-one and top-five scores measure attention ranking, not decoded-answer truth. No score threshold gates delivery. Text precedes metric persistence, writes are sequential, and the CLI discards the returned dictionaries; its durable results are files. Oversized KB requests raise to the launcher without automatic retry. The uncached full-dataset branch has a range/index-shape mismatch, and expected layer files, sufficient KB rows and question-batch size constrain completion. This is a multi-step computation, not child-agent delegation, review or recovery. File presence alone cannot certify completed delivery (workflow accuracy annotation).
What evaluation and retained changes establish
The evaluator family has several distinct targets. Attention metrics assess row rank. ROUGE/BERTScore assess extracted text against supplied references. Refusal detection tests the substring sorry; an optional filter removes such outputs before metrics and export. Marker replacement, question splitting and regex extraction can change what is measured. Reference GPT judgments assess agreement with supplied answers, while open-ended GPT judgments assess evidence-relative quality and common-sense reasonableness. These checks do not supply one independent factual oracle or an evidence-consuming acceptance decision for newly produced claims (EPI-RTE-answer-cleanup, EPI-RTE-metric-inputs, RT-RTE-evaluation-modes, EPI-RTE-open-judgment).
The two judgment artifacts also have different horizons. Reference judgments retain a scalar and comparison inputs, with post-batch saving and mean reporting. Open-ended judgments append parsed scores and reasons per record, but normal batch completion returns None, preventing the normal aggregate mean; caught failure can return a prefix list. Their defaults conflict with the client's model allowlist. Neither inspected consumer route feeds judgments or reasons into answering, KB revision or successor-checkpoint admission. Retaining a reason is a positive artifact property, not demonstrated later behavioral use. Accuracy sweeps additionally delete raw attention directories after deriving metrics, so reduced reporting can outlive its raw evidence (MEM-OBJ-reference-judgment, MEM-OBJ-open-judgment, MEM-RTE-result-cleanup).
The strongest improvement-directed contribution is implemented supervised adaptation of reusable numerical machinery. Dataset targets shape weighted loss, gradients and optimizer updates to encoder projections and optionally selected query heads. Periodic saves retain those changes without a validation-improvement gate. The training driver freezes the other model parameters; arbitrary library callers are not bound by that policy. Later load wiring gives saved parameters a route to projection and generation, but the evaluator entry defect and save/resume layout mismatch qualify that route. Improved capacity for future action is unestablished: no before/after outcome or later operation causally depending on the retained change is supplied (RT-RTE-adapter-training, RT-BAP-checkpoint-consumption).
These findings leave independent questions open rather than assigning a grade. The accepted runtime does not establish a formulated-theory route meeting theory-builder conditions: localized theory content (condition 1), content-dependent consumption of that theory (2), stated content-directed criticism with changed reliance or revision (3), and retained criticism shaping another round (4) are each unestablished for the supervised pathway. Numerical loss is not itself formulated criticism. Criticism of a consumed theory improving future capacity is likewise unestablished. Reflection is not established by parameter updates, and automatic bounded optimization does not establish autonomous conduct of the research workflow. The source supports a marked dispositional, improvement-directed self-change pathway in the training-to-inference machinery, not exercised or successful self-improvement: later causal exercise is missing. These are scoped findings, not claims about inaccessible provider processing (RT-RTE-adapter-training, epistemic bounded conclusion).
Scenario-relative assessment and changes that would matter
For the scenario of answering from selected external records, KBLaM supplies a distinctive numerical context path without an external search service. For the scenario of reproducing the shipped evaluation unchanged, the known entry defect prevents completion. For the scenario of evidence-driven improvement, supervised parameter adaptation is implemented, while judgment reporting remains separate from successor admission. The README's linear-overhead and exact-base-output claims remain declarations beyond the narrower insertion/bypass wiring; no measured scaling, guaranteed grounding or universal exact-output equivalence follows (RT-CLM-knowledge-augmentation, RT-CLM-base-preservation, RT-RTE-adapter-training).
A repaired, version-pinned execution trace could establish completed delivery. Content-addressed record/vector/checkpoint lineage and explicit cache invalidation would change the reconstruction and freshness assessment. Controlled KB/no-KB, pruning and before/after-training comparisons could establish effects only at the level actually varied. A trace linking saved updates to later behavior, with outcomes against the independent answering objective, would address exercised change and benefit separately. A documented score-consuming successor decision would change the current reporting-only assessment, but would not by itself prove factual warrant or improvement. The profile verification supports the local classifications while explicitly withholding complete inventories and trace-learning positives or negatives; ordinary supervised adaptation is not thereby denied.
Limitations
| limitation | affected SRC IDs and record citations | inspected boundary | conclusion prevented | evidence that would resolve it |
|---|---|---|---|---|
| Declared versus delivered extension guidance remains the reconciled qualification. Runtime records the evidence-only request; specialists establish the attribute mismatch. | SRC-1; RT-RTE-dataset-production, EPI-RTE-augment-qa, memory dataset annotation, report-verification Limits | Generator extension call and inherited chat-message construction | That the evidence-only system constraint reaches this call, guarantees semantic preservation, or that model compliance is disproved | Corrected message construction plus captured messages and checked input/output examples; reliable compliance needs broader evaluation |
| Historical lineage and unavailable assets constrain reconstruction. | SRC-1; Source register, MEM-OBJ-embd-key, MEM-OBJ-embd-value, MEM-OBJ-encoder, report-verification Limits | Shipped record/array/checkpoint producers and loaders; synthetic LFS payload and exact external assets unavailable | Corpus-wide correctness, authenticated positional correspondence, historical reconstruction and freshness guarantees | Full payload, immutable provider/model and checkpoint identities, per-artifact provenance and verified record/vector manifests |
| Internal wiring is not unchanged end-to-end executability or observed benefit. | SRC-1; RT-RTE-accuracy-generation, RT-RTE-dataset-production, workflow accuracy annotation, report-verification Limits | Evaluator entry and selected internal workflow; generator and downstream formats | Reliable unchanged execution, completed delivery or comparative operational performance | Repair import/name and generator defects, reconcile JSONL/array formats and branch contracts, then supply pinned execution traces |
| Alternate consumers and pruning retain compatibility limits. | SRC-1; RT-RTE-direct-inference, memory direct-inference annotation, RT-RTE-kb-attention | Processor constructor/type/layout paths and Llama optional pruning; distinct Phi attention | Universal processor usability or pruning equivalence; no generalization from direct compatible tensors to every wrapper | Corrected interfaces and score gathering, shape/version contracts and controlled path comparisons |
| Measurements and judgments have bounded licenses and incomplete reporting paths. | SRC-1; EPI-RTE-answer-cleanup, EPI-RTE-metric-inputs, MEM-OBJ-reference-judgment, MEM-OBJ-open-judgment | Extracted metric population and both scorer scripts | Independent truth verification, unbiased all-attempt metrics, calibrated judgments or normal completed open-ended aggregate reporting | Full attempted-output traces, validated extraction, independent references, judge calibration, valid provider selection and repaired normal return |
| Raw evidence retention and delivery are nontransactional. | SRC-1; MEM-RTE-result-cleanup, MEM-OBJ-attention-weights, workflow accuracy annotation | Accuracy sweep cleanup and sequential result writes | File presence proving completion, indefinite raw-evidence retention or automatic rollback | Path-separation enforcement, durable intermediate summaries, retained snapshots and completion/recovery traces |
| Adaptation is not demonstrated capacity improvement or later causal exercise. | SRC-1; RT-RTE-adapter-training, RT-BAP-checkpoint-consumption, report-verification Limits | Supervised optimizer, periodic saves and later load wiring; mismatched resume layout and independently resampled outlier rows | Exercised or successful self-improvement, improved future capacity, exact recovery, or guaranteed missing-answer supervision | Aligned save/resume contracts, verified outlier membership, and controlled outcomes linking retained updates to later operation |
| Opaque retained-state alternatives and missing bounded absence warrants remain unresolved. | SRC-1; RT-CMP-gpt-session, RT-RTE-adapter-training, MEM-OBJ-open-judgment, profile-verification | Local client, supplied artifact histories and inspected reporting/update consumers; SDK/provider internals uninspected | Complete memory inventories, provider retention/grant claims, or a system-wide trace-learning yes/no and trace-source classification | Scoped SDK/provider evidence, authenticated original update inputs and complete searched warrants for any proposed negative |