fragility-grid synthesis
Type: agentic-system-analyses/types/agentic-system-synthesis.md
Bounded synthesis
Scope and progression
Fragility-grid is a model-evaluation workflow, not an autonomous agent runtime. It asks how multiple-choice correctness and model rankings vary with prompt format, option order and scoring method. This account covers the selected collection-to-analysis subsystem in SRC-1 at commit 3f51444ead009d8351de1b6b19bf901c4da3d420, with an applicability cutoff of 2026-10-18. It includes the shipped sweep, collector, inference adapter, retained measurements and CPU consumer, but not the repository's other experiments. The evidence is inspected implementation and attributed design/results, not observed execution or a causal experiment. No target tests or model calls were run, and released numerical values were not independently checked (boundary, Source register).
An operator selects model keys, benchmarks, item count, permutations, backend and GPU resources. In the representative run_fragility.sh workflow, shell workers take model keys from a locked queue and launch one collector subprocess per assignment. Fixed shell and Python rules own progression; models return answer text or continuation scores. Default scope is eight model keys, four benchmarks, 300 items and six permutations, not the release's reported twelve-model, 3,679-item experiment. Two additional shipped launchers vary scheduling and defaults, so this representative trace is not a complete launcher comparison (RT-RTE-sweep-driver, workflow account, report verification limits).
The collector loads benchmark items separately for each model. It uses fixed shuffle seeds and reserves the first five items without using them as demonstrations. Imported dataset labels supply the expected-answer oracle; the workflow does not independently establish their truth. TruthfulQA is reduced to four options before evaluation. At default settings each item crosses 24 generation configurations and two likelihood configurations. This implements same-loaded-item comparison within an invocation; equality across model invocations additionally depends on stable external data (EPI-OBJ-benchmark-items, RT-OBJ-config, RT-CLM-grid-purpose).
Answer production and measurement
Each generation call receives an independent question/options frame and format instructions. Earlier answers, aggregate results and gold labels are not inserted into that frame. Likelihood scoring instead evaluates one continuation against a stem and selects the largest mean token score. This is fixed experimental enumeration, not model-led delegation or conversational memory retrieval. Model weights are unchanged in the inspected inference paths, but checkpoint and tokenizer loading do not pin immutable revisions (RT-RTE-generation-grid, RT-RTE-likelihood-grid, RT-CMP-roster-model).
Generated labels are parsed, mapped through the option permutation and compared with gold. Unparsed responses become zero bits rather than requests for correction. These checks establish dataset-relative agreement conditional on gold and extraction fidelity, not answer truth. Excluding fuzzy restatement parsing does not exclude parser noise: the inspected first-match parser selects A from Not A; B is correct. This is a source-level counterexample, not an observed response or an estimate of error frequency. Likewise, generation assumes backend return cardinality through zip; a short return could leave missing bits, but no such backend failure was observed (EPI-RTE-label-extraction, EPI-RTE-gold-comparison, RT-RTE-generation-grid).
Backend controls are route-specific. Chat rendering retries without the thinking-disable flag on a tokenizer TypeError; likelihood scoring bypasses chat rendering. HF generation truncation does not establish the same guard for HF likelihood scoring. CUDA OOM recovery halves HF generation chunks down to one item, then propagates failure; it covers neither vLLM nor likelihood calls. These are conditional controls and best-effort recovery, not backend equivalence, universal context clipping or guaranteed completion (RT-CMP-engine, RT-RTE-backend-recovery).
Retention, later consumers and delivery
After all requested benchmarks for a model finish evaluation, fixed software writes per-item JSONL records and a model summary. Records retain correctness bits, question text, gold and instrumentation, but not raw answers or complete rendered prompts. Later CPU analysis loads only item IDs and bits, derives model/benchmark identities from filenames and reconstructs configuration keys using the legend's permutation count. It does not recheck answer extraction or authenticate common question identity. Matching files can therefore carry stale or manually changed inputs into analysis; a newly written legend can coexist with older records (RT-OBJ-records, RT-RTE-record-analysis, MEM-OBJ-config-legend).
The strongest supported trace-fed behavioral contribution is a durable completion marker: automatic summary production follows model-call outcomes, and a later shell worker skips the model when that summary path exists. This qualifies as wired trace learning in the memory comparison's behavior-shaping-write sense. It is not evidence of improved future capacity, parameter training or scientific acceptance. The consumer reads existence, not summary explanations, numerical quality or request compatibility. CPU analysis separately pulls bit mappings; reporting artifacts afford human inspection but do not enter subsequent model context through the inspected collector (RT-OBJ-summary, RT-BAP-summary-skip, MEM-OBJ-analysis, profile verification).
Delivery means local artifact publication, not certification of a complete sweep. The collector frees its engine on evaluation failure, but writes no completed-model result from that failed evaluation. Later persistence can fail partway through multiple files. Workers continue after failed children. A caught gated-access skip can also end normally without new artifacts; report verification establishes this branch through the HF missing-token check, not as a guarantee about vLLM access errors. CPU analysis processes available records, writes analysis.json, then attempts figures independently. Empty inputs return early, and figure failures are tolerated. Neither a child OK nor the final shell message establishes all-model or all-figure success (workflow account, RT-RTE-record-analysis, report verification).
What the analysis warrants
The CPU stage computes descriptive bands, ranks, gap decompositions, bootstrap intervals and discrimination-based subset comparisons over retained bits. Pairwise analysis intersects two models' IDs; compression uses items shared by all models and skips insufficient item/model counts. Seeded cutoff tie-break draws and champion margins qualify simplistic winner claims. They remain implementation safeguards, not verified empirical results (EPI-RTE-statistical-check).
A robust-subset interval containing zero licenses the program's tied label, not affirmative equivalence. Combining that interval with a nonzero reference gap automatically emits a manufactured-gap label; the numerical premises do not establish causal manufacture or that fragile items represent mental uncertainty. Similarly, the plotted any-item/all-item envelope need not be attained by any single configuration. The artifact retains these interpretations, but publication is not an evidence-consuming acceptance of their broader explanatory truth (EPI-RTE-result-labeling, EPI-OBJ-analysis-output).
For the scenario of comparing fixed harness configurations on stable benchmark inputs, the implementation supports conditional measurement and descriptive comparison. It does not independently establish the release's magnitudes, finer component causes, behavior of all compression methods or byte-identical reproduction. Separate seeded generators contradict the literal single-generator description without proving that fixed-environment reruns differ (EPI-CLM-headline-results, EPI-CLM-reproducible-analysis).
These findings do not establish a theory-building or model-led revision cycle. The accepted runtime does not separately assess theory-builder conditions 1–4; configuration scores and retained outcomes alone cannot substitute for localized theory consumption, content-directed criticism and its iteration. Criticism of a consumed theory improving future capacity is not established. Model-internal reflection remains uninspected; fixed orchestration is not autonomous theory building. The summary-marker pathway changes later scheduling, but its observed exercise and improvement-directed attribution are unestablished, so exercised self-improvement is not demonstrated. These are separate limits, not a negative inferred from fixed weights or from one another (RT-RTE-record-analysis, RT-CMP-roster-model, RT-BAP-summary-skip, epistemic bounded conclusion, profile verification).
The assessment would change with provenance-linked run traces and raw answers, immutable item/model/tokenizer manifests, completeness and compatibility checks at reload/resume, controlled repeatability comparisons, and experiments addressing the specific explanatory claims. A claimed improvement cycle would additionally need a retained update, its later exercised consumer and objective-relative evidence; the current scheduling marker alone does not supply those findings. No additional source reads were made for this synthesis.
Limitations
| Limitation | Affected SRC IDs and record citations | Inspected boundary | Conclusion prevented | Evidence that would resolve it |
|---|---|---|---|---|
| Launcher inventory omits two shipped variants | SRC-1; RT-RTE-sweep-driver, workflow account, report verification limits | Representative driver plus verifier inspection of repro/run_big.sh and repro/run_modern.sh |
Complete scheduling/default/resource-policy comparison: big reserves a tensor-parallel pair alongside workers; modern uses single/paired phases and defaults purge on | Accepted route-level accounts of both variants and their differing resource/default policies |
| Operation and released results are unverified | SRC-1; RT-CLM-grid-purpose, EPI-CLM-headline-results, RT-RTE-record-analysis | Source inspection; no executed tests, model calls or authenticated result-value reproduction | Actual activation, empirical magnitudes, successful fresh reproduction and benefit | Provenance-linked runs, test results and independent recomputation under recorded conditions |
| External bytes and cache internals are unresolved | SRC-1; RT-CMP-engine, RT-CMP-roster-model, EPI-OBJ-benchmark-items, profile verification | Adapter/loaders and exposed retained outputs, not dependency internals | Immutable cross-run identity, backend equivalence or complete memory/cache classifications | Pinned dependency/data/model/tokenizer identities and inspected cache admission/consumption mechanisms |
| File provenance, item identity and grid compatibility are not authenticated | SRC-1; RT-OBJ-records, RT-OBJ-summary, RT-RTE-record-analysis, MEM-OBJ-config-legend | Automatic writes and bit-only reload; initialization/manual replacement unresolved | All admitted files came from compatible collection, shared IDs denote identical questions, or resume is request-valid | Content-addressed item/configuration manifests, recorded admission lineage and enforced reload/resume compatibility checks |
| Lossy extraction and missing-output assumptions limit measurement | SRC-1; EPI-RTE-label-extraction, EPI-RTE-gold-comparison, RT-RTE-generation-grid | First-match parsing, gold equality and assumed backend cardinality | Universal exclusion of parser noise, reconstruction of answer extraction, or demonstrated completeness under short backend returns | Raw response/prompt retention, extraction-fidelity evidence and output-cardinality/completeness enforcement; observed traces to measure frequency |
| Statistical labels and display bounds exceed narrower numerical warrant | SRC-1; EPI-RTE-statistical-check, EPI-RTE-result-labeling, EPI-OBJ-analysis-output | Arithmetic, bootstrap, subset contrasts and figure labels over supplied bits | Population equivalence, mental uncertainty, causal manufacture, universal compression effects or a fixed configuration attaining oracle bounds | Appropriate equivalence/causal designs, explicit sampling assumptions and corrected distinctions between itemwise bounds and attained configuration accuracies |
| Repeatability is conditional, not demonstrated byte identity | SRC-1; EPI-CLM-reproducible-analysis | Separate seeded generators and output code, without repeated runs | Literal single-generator architecture or environment-independent byte-identical output | Corrected architecture description and controlled repeat-run comparisons with frozen inputs/environment |
| Behavioral adaptation is wired, but improved capacity and exercised self-improvement are unestablished | SRC-1; RT-OBJ-summary, RT-BAP-summary-skip, RT-RTE-record-analysis, profile verification | Automatic outcome-to-summary-to-later-skip route; no observed later exercise or method-revision cycle | Improvement, content-directed critical iteration, autonomous theory building or exercised self-improvement; wired trace learning remains supported | Property-specific records of theory/criticism where claimed, objective-bearing update determination, retained consumer lineage and subsequent causally dependent operation |