fragility-grid report verification

Type: agentic-system-analyses/types/agentic-system-verification.md

Verification

The reports support a bounded implementation account of the selected evaluation workflow. I independently traced the consequential behavior in SRC-1 at the registered boundary, rather than treating analyst agreement or their validation statements as evidence of support. The checkout HEAD matched the frozen revision and its status was clean. I inspected source text only; no target execution, tests, provider calls or empirical reproduction was performed. The supplied structural report says none; the four correction-answer files are empty. There are no structural failures or correction arguments to route.

Runtime and authority. I checked every runtime declaration against the driver, collector, adapter, configuration logic and relevant shared loaders: SRC-1 repro/run_fragility.sh, repro/fragility_grid.py, repro/inference.py, repro/fragility_lib.py, repro/prompt_format_lib.py, repro/prompt_format.py, repro/brittleness.py and repro/common.py. The model supplies text or likelihoods, while fixed code owns iteration, parsing, permutation reversal, gold comparison and persistence. The ordinary sweep defaults, dataset pool exclusion and independent inference inputs are supported. RT-CMP-roster-model correctly separates unchanged inference parameters from mutable checkpoint-name resolution; the loading paths do not pin a revision. RT-BAP-prompt-answer describes conditioning, not reliable instruction compliance. The alternate template render, raw likelihood path, generation-only truncation and backend-specific quantization substantiate the qualified account of RT-CMP-engine.

I checked both backend recovery and branches outside it. RT-RTE-backend-recovery correctly limits chunk-halving to HF generation and propagates one-item OOM. The collector's evaluation finally precedes record writes; persistence can still fail partway through subsequent writes. Summary existence, not contents or request compatibility, controls RT-RTE-sweep-driver and RT-BAP-summary-skip. Direct collection bypasses that skip. The workflow's scheduling annotation correctly adds caught gated access: this normal-exit skip is reachable through the HF loader's missing-token check for gated roster entries. It is not a guarantee about how vLLM reports access failure. JSON-before-figures, empty-record return and tolerated figure exceptions support the workflow's bounded delivery account. Neither the final shell echo nor child exit authenticates complete requested output.

Memory and consumer transformations. I checked all memory declarations and annotations against collector writes, loader projection, configuration reconstruction, plotting and output helpers, including SRC-1 repro/predownload_data.py. RT-OBJ-records retains derived correctness, not raw answer traces; the later loader keeps only item IDs and bits. The summary's separate scheduling consumer is supported. MEM-OBJ-config-legend is genuinely part of RT-OBJ-config: analysis reads its permutation count but reconstructs keys from current code. MEM-OBJ-analysis and MEM-OBJ-fragility-figures correctly distinguish serialized aggregates, current-process plotting and display exports. No inspected collector input reads these outputs as subsequent model context. This supports the stated consumer wiring, not a negative over external backend caches or the repository's other experiments. Unknown manual initialization does not erase known automatic writes.

Epistemic claims and checks. I checked every epistemic object, route and claim, including complete SRC-1 repro/fragility_analysis.py and repro/stats.py. EPI-OBJ-benchmark-items correctly treats gold as imported rather than independently verified; TruthfulQA reduces its option domain before evaluation. EPI-RTE-label-extraction has a valid first-match counterexample to the universal source claim in EPI-CLM-parser-noise. The input is supported by the parser's string interface; no assumption that a model actually emitted it is needed or made. The fixed grid excludes the fuzzy parser, but that exclusion cannot exclude first-match mistakes. EPI-RTE-gold-comparison therefore warrants only dataset-relative agreement conditional on extraction and gold fidelity. The reports appropriately treat short backend returns as a conditional cardinality-contract issue, not an observed failure or demonstrated ordinary backend behavior.

The arithmetic decomposition, conditional robust-subset mean, percentile bootstrap, seeded cutoff tie-breaks and random subsets support EPI-RTE-statistical-check. Inclusion of zero is consumed to label a pair tied; it is not an equivalence test. EPI-RTE-result-labeling preserves the distinction between programmatic reporting disposition and warranted explanatory acceptance. Its non-entailment argument is sound: a finite-sample interval containing zero does not entail population equality or mental uncertainty. The oracle-envelope counterexample in EPI-OBJ-analysis-output also follows from the any/all definitions and the figure label. EPI-CLM-reproducible-analysis correctly challenges the literal single-generator account: bootstrap calls initialize separate generators from the one inside compression. This does not establish that fixed-environment reruns differ. EPI-CLM-headline-results remains attributed, without promoted empirical or causal status. Source-level tests were read as specifications only.

Identity and reconciliation. I checked all source anchors and evidence-status meanings, all four Part of: relations, and the one supersession. The label extractor belongs to generation; statistical evidence production and result labeling are distinct parts of the analysis route; gold comparison overlaps two runtime routes and correctly avoids claiming one parent. The superseded epistemic aggregate identifies the same out mapping as the memory aggregate. Its additional findings remain available; the amendment does not transfer or overwrite values. Figure exports remain distinct. The remaining declarations identify different objects, parts, routes or claims. The memory integration issues are disposed without inventing provenance. I found no unresolved inter-report conflict: the parser and generator challenges concern source assertions, not contradictory analyst findings. Workflow's gated skip supplements, rather than contradicts, runtime's process-exit account. All route wired statuses describe inspected implementation; none is promoted to observed operation by convergence.

Omission search and stopping decision. I used the frozen tracked tree, searches for collector/consumer callers and output references, and the producer-to-consumer chain to ask whether another shipped launcher changes scheduling or purge behavior, whether saved explanatory fields reach later inference, and whether numerical or display consumers strengthen the scientific claims. The latter two questions have supported dispositions above. The caller search found SRC-1 repro/run_big.sh and repro/run_modern.sh; I read both completely. They belong to the selected experiment, not an external host. They call the same collector and CPU consumer, retain the summary-existence predicate and add multi-GPU scheduling. The modern script also enables purge by default. Their omission leaves the selected representative workflow valid but the launcher inventory incomplete, as retained below. These reads supply implementation evidence only; their comments do not establish successful operation. Further optional expansion stopped because the selected challenges now have dispositions, no identified material contradiction remains, and additional result files cannot by presence alone establish execution provenance or causal warrant.

Blockers

none

Limits

• The launcher inventory around RT-RTE-sweep-driver and the selected workflow omits two shipped variants: SRC-1 repro/run_big.sh reserves a tensor-parallel GPU pair alongside queue workers, and repro/run_modern.sh runs single-GPU and paired-GPU phases and enables weight purge by default. Readers should withhold a complete comparison of launch scheduling, launcher defaults and resource policy across this subsystem. The selected run_fragility.sh trace, common collector/consumer account, summary-skip authority and lack of inferred model learning remain supported. This is a local inventory omission, not evidence that those variants are external or absent. The modern driver's operative default is:

  export PURGE_WEIGHTS="${PURGE_WEIGHTS:-1}" --- repro/run_modern.sh