127 — Review runs as workflows over result files

Type: reference/types/adr.md · Status: accepted

Status: accepted; implemented on the operator's direction, which also ordered the old system's retirement without compatibility or data migration. Date: 2026-11-10 Supersedes: ADR 010 (review state in SQLite), ADR 035 (queued jobs and their finalization), ADR 043 (the store's completion, outcome and baseline model), ADR 052 (the general freshness store), ADR 065 (the store's published transitions) and ADR 067 (the prompt-path worker contract). It also supersedes three earlier decisions about the same store: ADR 029 (one packing protocol for pairs), ADR 032 (freshness from SQLite snapshots under a model partition) and ADR 036 (acceptance rows as current state). Amends: ADR 038 and ADR 041: the type spec and the COLLECTION.md stay the criterion; store identity, model partitions and acknowledgement go. ADR 051 and ADR 080: packets, pending hand-backs and the bounded closing stay; captures and the guard command become run records and recorded hashes. ADR 039: commonplace-review invokes Git for the repository root and the checkout guard.

Context

Review applies a criterion file to a note and keeps the judgment while both texts are unchanged. The store-based system did this with an agent in a conversation coordinating workers through files and a database: a selector, queued jobs, a finalizer, model partitions, baselines with acknowledgement, and a command per transition, over a 50 MB SQLite file. Most of that machinery existed because no code owned the run. The parent agent scheduled workers, so job state, transport files and all-or-nothing finalization had to make its handoffs safe.

The headless runtime (ADR 118, ADR 119) now supplies a workflow written in Python with what the store built by hand: one agent call per request, a run lock, durable launch records, replay of answered calls, deterministic checks with retry, worker problems and usage-limit stops. It does not keep state across runs: Workjig's records belong to one run and promise no format compatibility. Freshness across runs therefore still needs a durable record of what text a judgment was produced from.

Phase 0 of the design work measured the store on 2026-11-09: 9,217 baselines and 6,947 recorded jobs, the last completed 2026-09-26. 37 baselines stood on an acknowledgement (note side only); that count is a lower bound, because a later review replaces an acknowledged baseline. Outside the review and freshness packages, only status reporting and a store health check read the store.

Decision

A result is a file. Each accepted review writes kb/reports/state/review/results/<note-path>/<criterion-id>.md. Its frontmatter names the note and the criterion with the SHA-256 of the texts the worker was given, the outcome (pass, warn, fail, or report for a report criterion), the worker profile, the model the harness recorded, and the run and launch; the body is the reviewer's text. A pair is fresh when both hashes match the current files. Status, staleness and orphan detection are pure functions over these files. Freshness depends on neither Git nor a store. One result exists per pair, whichever profile produced it.

The note hash covers what the criterion watches. A criterion that declares watches: (title, description, body) is hashed against those note parts, normalized so a whitespace-only edit changes nothing. A criterion without a valid declaration watches the whole note. This replaces acknowledgement of edits outside the watched parts.

Review is workflows, under one command. commonplace-review runs run gates (the gate sweep), run full-pass (the full improvement pass) and run revise (the simplification passes), with status, packet, retry and acknowledge, plus approve for operator answers. A workflow is a Python function over ask, checked, gather and decide. Each run has one directory under kb/reports/state/review/runs/. A sweep opens a new directory, and --resume continues it and replays answered calls. A full pass or revision derives its directory from the workflow, the note's path and the hash of the note's text, so repeating the command continues the run by replay, and an edited note opens a new run. Where a workflow needs the operator, it proposes a file in the run and stops. Explicit approval captures the edited or unedited file as an answer, bound to that stop, proposal and its inputs. Repeating never approves; selection-list edits require approval again. Changed proposal inputs invalidate the approval. Options such as the profile and the full pass's direction are recorded with the run.

Planning is an explicit mode, not the default for editing. A new revision run requires --plan FILE for fixed work or --assess to choose work through assessment. Repeats retain their mode; changed artifact text requires a new choice. Fixed-plan execution constructs no assessment request. Editors receive an execution instruction, not the planning procedure. Its publication context excludes unused planner inputs. One fixed pass and one report-only criterion are the independent evaluation baselines; assessment-led revision and full-pass composition need separate evaluation.

Approval protects unsettled selection and final publication, not routine planning. Within the explicitly chosen mode, ordinary intermediate work is preauthorized. Revision retains the generated plan for inspection and executes it after validation. --plan is an explicit validated input or override, including replacement of a stopped plan; neither route requires plan approval. Editing the retained plan alone is not an override.

Plain-wording and abstractions retain explicit approval per selection proposal as temporary procedure-development points. Retain proposed, accepted and rejected choices with reasons, revise selection rules and checks, and evaluate them on retained and fresh examples before separately authorizing removal. This calls for procedure development, not new automatic learning infrastructure. Fewer proposals alone do not establish reliability.

For changed final text, both workflows generate artifact/final.md, artifact/final.diff and artifact/publication.json. The operator inspects the text and diff and approves the exact generated manifest through commonplace-review approve RUN FILE --key KEY. An edited manifest cannot authorize publication. The candidate and manifest are outputs, not revision inputs: use --plan for revision, change workflow inputs or start a new run. Instructions own judgment; prompts name inputs only.

Workers never write the checkout. A worker receives captured inputs and writes only its output files. Code checks each output (the strict result line, a decision schema, the bounds an edit must keep). A workflow that edits a note carries each edited text forward in memory and in the run directory, and writes the note once, at its last step, only after exact final approval and with the original note and captured brief unchanged. A change to either during the run is a conflict, and nothing is written. A checkout guard reports paths that changed while a run was open.

A criterion is any markdown file. Catalog gates, a type spec, a COLLECTION.md and the critique instruction all enter as the criterion input. The workflow names the output contract, verdict or report.

The full pass keeps its packet. It asks the gates over each text it reviews and writes result files once, at its end, for the text the note then holds. When the final text is edited, the pass stops before publishing the candidate packet, results or note until exact final approval. No-edit dispositions may publish their report, packet and results automatically. The packet is markdown whose frontmatter code writes: the note, the start and final text hashes, the decision, the closing status and the resolution fields. Its start text lives in the run; commonplace-review packet compares the recorded hashes with the files.

Considered alternatives

One long-lived run as the store. Workjig keeps every call of a run in one record file; thousands of pairs would make that file the database this decision removes. Set aside for one run per invocation, with result files carrying freshness across runs.

Migrating the baselines. The operator judged on 2026-10-24 that the stored reviews were outdated. The new system started empty; the first sweeps produce its results. The store and its job directories are abandoned local data.

Implicit approval on repeat. Rejected: retrying after interruption must not make an operator decision. A separate approval command captures the answer; a stale proposal key is refused. The command does not authenticate the human: operating instructions permit an agent to approve only on the operator's stated word for that proposal.

Routine plan approval. Rejected: assessment, planning and settled intermediate work are already authorized. Retaining the plan for inspection and accepting a validated --plan override preserves operator control without a routine stop. This does not authorize unsettled selections or publication of changed final text.

Editable final approval payload. Rejected: approving a replacement manifest would detach consent from the checked candidate. Final approval must match the generated manifest exactly; revisions enter through workflow inputs instead.

Keeping acknowledgement. Dropped. It covered 37 of 9,217 baselines, and the watched-parts hash absorbs its commonest case. A changed pair is re-reviewed. An acknowledgement command returns only when sweeps show the re-review cost.

Keeping model partitions. Dropped. The profile and the recorded model are provenance, not identity, and status shows them. What would reopen it: an operator who wants two models' verdicts on one pair kept side by side.

One call per pair or per bundle. Both shipped. pair asks one worker per stale pair, so a criterion edit re-asks exactly its pairs; bundle asks one worker per note and bundle of catalog gates for that bundle's stale gates, and each gate still gets its own result. A launch's fixed context dominates its cost: one live pair cost $0.095, nearly all of it fixed context, and one bundle launch reviewed nine prose pairs on one short note for $0.172 in 31 s. Each figure is a single observation. The sweep defaults to pair until a comparison shows whether grouping changes verdict quality; the full pass uses bundle.

Decision models for bounded gates. Deferred. A judgment-specialized model could serve closed-ended gates, but its cost is calibration per gate, under the gate calibration proposal. The result file names an executor rather than a launch, so such a result fits it.

A directory type for a full-pass run. Deferred until a run shows what validating a run directory as one unit would catch that the packet type and the result-file parser do not.

Consequences

The operativity path: the commonplace-review command and its library modules enforce result files, freshness, the checkout bounds and packets in code. commonplace-status --review reads the result files. The review-system guide, the full-pass and disposition instructions and the fix procedure bind agents operating review. kb/instructions/review-profiles.yaml sets what a worker may do. A review gate's watches: now decides which edits stale its pairs, so gate authors set staleness.

Easier: adding a workflow is one Python function plus its criterion files. A stopped run continues where it stopped without repeating a paid answer. Freshness is inspectable as files. The full pass and the revision apply an edit only through code that checks it, and change the note at most once per run. A hand-back or an unfinished run leaves the note untouched, so nothing needs restoring.

Harder: every changed pair is re-reviewed, and a type-spec or COLLECTION.md edit re-reviews its whole cohort. A wrong watches: declaration leaves pairs falsely fresh. A note move orphans its results; they are deleted and the pairs reviewed again. Results stay machine-local and untracked. commonplace-review needs a Git working tree: it finds the repository root with Git and reads git status for the checkout guard, against ADR 039's rule that commands never invoke Git. Freshness itself does not depend on Git.

Dropped without replacement: acknowledgement, model partitions, link-consumption and availability telemetry, the reviewer's self-reported model, relocation's rekeying of review state and the generic freshness store for non-review targets.

Open:

  • the default grouping, pending a quality comparison of pair and bundle verdicts;
  • the checkout guard cannot tell a worker's change from another session's, so concurrent work in the checkout makes a run exit nonzero with paths to inspect;
  • the Claude Code profiles' write denial outside the run directory has not been seen to refuse a write: the guard is the only verified control;
  • the codex profile cannot be sandboxed read-only, so it relies on the prompt and the guard.

Limits: decided and run in this checkout, with the sonnet profile for the live runs. One full pass on a short note took 27 launches and $4.47 and ended in a hand-back; that is one observation, not an estimate of pass cost or outcome. The opus and codex profiles and the revision workflow have no live measurement here. The prompt around the inputs is outside the freshness hashes, so a prompt change re-asks calls on a resume or repeat but stales no result; judgment belongs in criterion files.

Relevant Notes: