Assay 1 generated evidence

This directory preserves the exact inputs and outputs for the first description-length assay. It is generated by ../run_assay.py, except that model outputs necessarily come from the recorded evaluator calls.

Contents

  • manifest.json — corpus size, target hashes, batch definitions, and per-trial pointer costs.
  • prompts/ — the exact 30 retrieval prompts shown to the evaluator.
  • results/ — raw structured selection results for those prompts.
  • logs/ — evaluator traces establishing runner/model provenance; each ends with the structured result and runner-reported token use.
  • scored-results.json — retrieval decisions joined to gold targets and aggregate metrics.
  • fidelity/, fidelity-logs/, and fidelity-review.json — ten split semantic-fidelity reviews and their 50 aggregated judgments.
  • downstream/ — exact prompts, selected-body excerpts, raw answers, logs, and deterministic rubric scores for the 40-trial downstream subset.

Reproduction

From the repository root:

python3 kb/work/description-length-optimization/run_assay.py generate
python3 kb/work/description-length-optimization/run_assay.py evaluate --jobs 4
python3 kb/work/description-length-optimization/run_assay.py score
python3 kb/work/description-length-optimization/run_assay.py fidelity --jobs 4
python3 kb/work/description-length-optimization/run_assay.py downstream-generate
python3 kb/work/description-length-optimization/run_assay.py downstream-evaluate --jobs 4
python3 kb/work/description-length-optimization/run_assay.py downstream-score

generate ranks distractors against the live corpus, so an exact rerun after corpus changes requires the preserved prompts rather than regeneration. Target SHA-256 hashes are in manifest.json; the prompt files freeze every distractor pointer and randomized order used in this run.

The model-running commands require an authenticated Codex CLI. The 2026-07-27 run used Codex CLI 0.145.0, whose default model resolved to gpt-5.6-sol at high reasoning effort. The explicitly requested luna alias was rejected by the account before inference, so it contributed no judgments.