Review system

Type: kb/types/note.md

The review system runs snapshot-anchored assays over KB notes and records their results. Verdict-kind gate reviews produce pass, warn, or fail; the opt-in critique assay is report-kind and completes without an outcome. Notes and assay criteria stay as markdown files in the repo; review state lives in a local SQLite database.

Two properties shape everything else:

• Freshness is independent of Git. Assay creation, finalization, and acknowledgement store DB-owned snapshots of the exact note and criterion text that form the current baseline. Staleness is decided by comparing current file text against those snapshots, not by inspecting Git history.
• It is experimental and opt-in. Reviewing is not part of the default note-writing flow, and reviews are not always-on checks. You invoke the system deliberately when you want a note judged.

Storing review state in SQLite is a scoped exception to the repo's file-first design; ADR-010 records why. This document covers how to operate the system; for how it is built, see review architecture. For the calibrated one-cycle review that follows a transforming note workflow, see full improvement pass closure.

How a review flows

A full review runs as a short pipeline, from a stale-pair query to a durable freshness baseline:

1. Select — the selector lists (note, criterion) pairs that are stale or unreviewed for a given model partition, and says why. Persisted and JSON identities are (note_path, criterion_path).
2. Create jobs — selected pairs are grouped into one or more queued review jobs, each with its own rendered prompt.
3. Review — a worker (typically a sub-agent) reads a job's prompt and writes a single sentinel-delimited job-output.md.
4. Finalize — the parent parses that output against each pair's persisted result contract and, only if every expected pair is present and well-formed, records completion and creates or replaces freshness baselines. Soft consumption metadata is recorded separately and cannot change this decision.

A freshness baseline is the current snapshot-pinned applicability boundary for completed evidence. It is not a universal endorsement: a verdict baseline retains a closed-ended gate outcome, while a report baseline records that the retained report matches the pinned inputs. When a change does not invalidate that evidence, acknowledgement (ack) advances the existing baseline without changing its evidence pair.

Concepts

Assay. Any snapshot-anchored LLM evaluation executed through selector → job → worker → finalizer. Question shape and result kind are separate axes: a closed-ended assay asks a fixed question, while an open-ended assay samples a space of possible findings; independently, the persisted result contract is verdict or report. Shipped gates are closed-ended and verdict-kind. Critique is open-ended and report-kind.

Gate. A closed-ended, verdict-kind assay criterion. Catalog gates are markdown files at kb/instructions/review-gates/{lens}/{name}.md in a source checkout, or under the installed framework gate catalog in generated projects. The {lens}/{name} shorthand is the gate id used at the CLI boundary (for example prose/source-residue). Type- and collection-conformance pairs are virtual gates whose criterion files are the type spec and COLLECTION.md respectively.

Criterion. The criterion-side text applied to the note, sometimes through a mechanical wrapper. It occupies the criterion_path side of every pair, including report assays.

Bundle. A directory of gates sharing a lens. semantic means all gate files under kb/instructions/review-gates/semantic/.

Type-conformance pair. A review of a note against the type spec named by its frontmatter type: — the type spec is the gate (ADR 038). Request it with the virtual type lens: type derives one pair per typed note in scope, type/definition narrows to one type's cohort. The criterion id is type/{name}; the persisted criterion identity is the type-spec path (for example kb/types/definition.md). Because the type spec sits on the criterion side, editing it stales exactly the notes of that type as criterion-changed, and a trivial type edit is acknowledged like any other gate edit. --all-gates includes type-conformance pairs alongside the catalog; request them alone via the type lens. The rendered prompt carries a fixed conformance instruction followed by the type-spec snapshot captured at job creation; if a type spec carries an authored ## Review section, reviewers treat it as the operative test.

Collection-conformance pair. A review of a note against the COLLECTION.md contract of the collection it lives in — the contract is the gate (ADR 041). Request it with the virtual collection lens: collection derives one pair per in-collection note in scope, collection/notes narrows to one collection's cohort (the path segment is the collection directory relative to kb/, so a collection under a namespace directory reads collection/commonplace/notes). The persisted criterion identity is the COLLECTION.md path (for example kb/notes/COLLECTION.md). Editing a collection contract stales exactly that collection's notes as criterion-changed; --all-gates includes collection-conformance pairs. The prompt mechanics mirror the type pair — the captured contract snapshot follows a fixed conformance instruction, an authored ## Review section in the COLLECTION.md is the operative test, and the reviewer judges only what the collection contract asks beyond the type contract, so the two conformance pairs do not double-flag.

Review job. One review invocation: one rendered prompt, one output artifact directory, and one job-level status. A job is queued until finalization marks it completed or failed.

Review pair. One requested (note_path, criterion_path) pair inside a review job. This is the unit of output and freshness baseline. Its persisted result_kind is verdict or report; report pairs complete with outcome = NULL.

Result kind. The persisted output contract fixed when the pair is created. A verdict pair completes with completed_at plus one outcome (pass, warn, or fail). A report pair completes with completed_at, a null outcome, and ## Result: REPORT. Completion and outcome are separate facts. ERROR means the worker could not produce a result and fails the whole job.

Critique pair. Request the virtual critique lens for one report-kind pair per explicitly targeted note. Its persisted criterion identity is kb/instructions/critique-note.md (or the installed framework equivalent), so editing that instruction yields criterion-changed. It is intentionally excluded from --all-gates.

Freshness baseline row. The current record that a (note_path, criterion_path, model_partition) key completed against specific note and criterion text. Freshness baseline makes the evidence fresh; it does not imply that report findings were handled or that the note is globally approved.

Model partition. Reviews are partitioned by model. A review or freshness baseline under one model does not satisfy freshness for another. A partition is a named bucket (claude-sonnet-5, claude-opus-4.8, codex) that groups concrete model IDs judged equivalent for review freshness — it is the review-identity key, not the literal model that ran. Commonplace designates no repository-wide default review model or partition; the orchestrator chooses a partition at selection that maps to a model the current harness can actually run, while a gate criterion may declare a required partition (see run review batches). Switching the selected partition starts a fresh evidence namespace: baselines under other partitions remain valid under their own keys but do not satisfy the newly selected partition's freshness. Concrete model IDs map into partitions through MODEL_PARTITION_REGISTRY in src/commonplace/review/review_model.py (build_model_partition). The CLI flag names carry the distinction: every partition-valued flag is --model-partition; the only --model flag in the review CLI is commonplace-finalize-review-job's, which records the concrete worker model and is validated to map into the job's partition.

Human-readable review output — the per-pair result files and each job's MANIFEST.json — is for inspection. SQLite result_kind, outcome, completed_at, freshness baseline, and job status are canonical state; see review architecture for storage details.

Criterion-derived local paths. A reviewer normally reads only the note, the criterion, and links that appear in the note. One exception is generic and worth knowing when authoring a gate: an active criterion may instruct the reviewer to derive exactly one local path from a link in the target and read it. The allowance lives in the shared prompt scaffolding, not in any one gate, so any criterion can use it — and a gate author who assumes reviewers can reach only linked material will design around a restriction that no longer holds.

The criterion owns the derivation rule and its failure behavior; the scaffolding only permits the dereference. semantic/grounding-alignment is the first user: an ingest link marked (snapshot required) sends the reviewer to kb/sources/.snapshots/<slug>.md, which is deliberately not linkable from a note (ADR 073). The route accounts for both physical inputs: the linked ingest supplies the canonical source and expected snapshot hash, while the derived snapshot supplies the source bytes being checked. Note what this does not loosen — one path, derived from a link that is already in the note, named by the criterion. It is not a licence to search the corpus.

Semantic boundaries

Do not conflate	Distinction
assay and gate	An assay is any snapshot-anchored evaluation; a gate is the closed-ended, verdict-kind subset.
question shape and result kind	Closed-ended/open-ended describes what is asked; verdict/report describes the persisted completion protocol.
completion and outcome	Both result kinds complete; only verdict pairs carry an outcome.
freshness baseline and endorsement	Freshness baseline pins current evidence to snapshots. It does not approve a note or resolve report findings.
freshness and handling	Fresh means the evidence matches current inputs. Handling findings is a separate downstream action.
REPORT and a verdict	REPORT is a protocol completion marker, not a fifth outcome.

Reading freshness

The selector reports each requested (note, criterion) pair as fresh or, if not, why it is stale. JSON and schema fields use criterion_* names. Selection compares the current note file, current criterion file, and current freshness baseline for the persisted key.

Catalog gates declare staleness: changed. This is a validated statement of the system-wide exact-text rule, not a per-gate tuning surface: gate resolution rejects missing or different values, and no rewrite threshold is supported.

• no freshness baseline for the pair → missing-baseline
• malformed, missing, or inconsistent data inside a present baseline → store integrity error
• baseline criterion text differs from the current criterion → criterion-changed
• baseline note text differs from the current note → note-changed
• otherwise the pair is fresh

JSON targets carry reasons as an array, so a joint edit reports both note-changed and criterion-changed. Each entry in changed_inputs names the role, path, accepted snapshot and hash, current hash, and—under --json—its diff. The target also carries baseline_revision. --reason tests membership: a joint edit is returned by either changed-input filter without hiding its other change.

Repository-wide status over registered targets is separate from applicable-pair discovery. commonplace-freshness-status reports every migrated review-pair baseline, with --json, --diff, --all, and partition filters. It does not emit missing-baseline for pairs that were never reviewed. See freshness architecture.

Running a review batch

The full procedure is in run review batches. In outline:

1. Select stale pairs and create jobs:

commonplace-review-target-selector ... --json | commonplace-create-review-jobs --input - --grouping {note|criterion}

--grouping note groups one note's pairs by bundle; --grouping criterion groups one shared criterion across notes. Jobs are always result-kind homogeneous.

1. For each job, launch a sub-agent that reads the job's prompt_path and writes its review to the job's job_output_path.

2. Finalize each completed output with commonplace-finalize-review-job (signature under Command reference).

Finalization is all-or-nothing. If any expected pair is missing, duplicated, unexpected, malformed, or lacks a valid result line, the job fails and writes no freshness baseline rows — a failed job accepts nothing. On success, every pair records per-kind completion and the current freshness baseline row for each pair is upserted. The review-consumption measurement is the exception to structural strictness: missing, partial, or malformed measurement metadata is recorded as such and never fails the job.

Every finalization response includes completed_pair_count and an ordered pairs array. Successful entries project the committed pair ID, note and criterion identities, result kind, canonical outcome, and derived result path. Verdict outcomes are pass, warn, or fail; a completed report carries an explicit null outcome. Failed and no-state-change responses carry an empty array. A completed job therefore means every pair satisfied its result protocol, not that every verdict passed.

Finalization-time provenance is optional. When supplied, --model (with optional --effort) is validated against the job's model partition before any state changes, and the runner/model/effort are recorded. --effort requires --model. --telemetry-json stores opaque harness telemetry without making it review identity. Commonplace's code-generated availability and reviewer-reported consumption measurements occupy versioned keys beside that opaque value in the same review_jobs.telemetry_json column.

Acknowledging trivial changes

For a changed pair, inspect every changed input first. If a change does not invalidate the existing evidence, acknowledge it instead of rerunning. For a closed-ended verdict pair this carries an outcome; for an open-ended report pair it only reuses the report as current evidence and endorses nothing.

commonplace-ack-review consumes inspected JSON from commonplace-review-target-selector. Keep only the targets and changed-input roles authorized for acknowledgement, with each retained role's matching reason. The command advances those exact observed hashes while preserving evidence_review_pair_id; omitted roles keep their accepted snapshots and remain stale. It does not create evidence or rewrite any file or assay text.

commonplace-ack-review --input {inspected-selector-json}

Ack fails when the baseline revision changed, any retained input's live hash no longer matches the inspected hash, or its accepted identity does not match the named baseline. Output lines have the form acked: <note_path> <criterion_id>.

commonplace-ack-trivial-note-changes may select type- and collection-conformance pairs, but never auto-acks them. Their criterion documents declare no watches:, so the command treats the entire artifact as watched. For ordinary gates, trivial qualification parses watches: from the accepted criterion snapshot and checks the exact current note version selected for acknowledgement. An edit between selection and qualification fails closed, including an edit that later restores the same bytes.

Building a warn/fix queue

commonplace-warn-selector builds a fixing queue from the current review state — the actionable findings from reviews whose outcome is warn. It is the entry point to the fix system, which turns these findings into applied corrections and fix reports.

• it reads current evidence review pairs across all models
• it queues findings only when both the live note and live criterion match the freshness baseline
• it reports skipped stale WARN residue separately as stale_pairs, using note-changed and criterion-changed reasons rather than presenting those findings as current
• for a fresh WARN, it loads the rationale from the evidence pair's result file; a missing result file prevents actionable finding extraction, but it cannot hide a stale-pair advisory
• it collapses model partitions to one current entry per (note_path, criterion_path), choosing the latest baseline-backed warn review

Interpreting review output

A pass records that a closed-ended gate found nothing to flag. It is not certification that the note has no problems. REPORT is not a fifth outcome: it says the report protocol completed. A fresh critique means the critique matches the current inputs, not “critiqued and handled.”

The same principle applies once a warn finding is delegated to the fix system. When the fix produces a diff, reviewing that diff is the judgment step: it checks the resulting artifact directly, rather than trusting the fixing agent's own report of success. Reading the agent's self-report still matters when no diff results — there is no artifact to substitute for it, so that case still needs the claim read directly.

Open-ended methods therefore remain report-shaped rather than being forced into gate decisions. The anchored critique uses report-kind completion; the unanchored composition friction gate routes attention without entering the freshness baseline store.

Command reference

Selector — commonplace-review-target-selector:

• positional gate ids, bundle names, conformance requests, and/or critique (e.g. prose, semantic/grounding-alignment, type, collection/notes, critique)
• --all-gates selects every applicable verdict criterion in place of naming ids/bundles (mutually exclusive with them): all catalog gates plus each note's type- and collection-conformance pairs. It intentionally excludes report assays. Like every selector flag it only chooses pairs — the selected pairs still run through create-jobs → worker → finalize; it is not a one-shot "run all gates" command
• --note to filter to specific note paths or directories
• --user-verified to filter exactly to notes with committed user-verified: true
• --model-partition {model-partition} selects the review model partition to inspect or write; omit it only for model-agnostic missing-baseline coverage
• --json
• --reason {missing-baseline,criterion-changed,note-changed}; changed-input filters match membership in reasons

With --model-partition omitted, the selector reports only model-agnostic missing-baseline coverage: a pair is missing-baseline only when there is no freshness baseline under any model partition. It does not classify criterion-changed or note-changed in that mode, because those need a chosen baseline.

Create jobs — commonplace-create-review-jobs --input - --grouping {note|criterion}. Consumes selector JSON.

Finalize — commonplace-finalize-review-job --review-job-id {id} [--runner {worker}] [--model {model} [--effort {effort}]] [--telemetry-json {json}]. Returns committed per-pair outcomes and result paths in pairs.

Ack — commonplace-ack-review --input {inspected-selector-json}.

Warn queue — commonplace-warn-selector.

Global freshness — commonplace-freshness-status [--json] [--diff] [--all] [--model-partition {partition}]. commonplace-freshness-ack --input - and commonplace-freshness-retire --input - consume JSON manifests. Review finalization owns creation and replacement of review-pair baselines; no generic initial-acceptance or refresh command ships.

The reviewer output contract

A worker writes one sentinel-delimited job-output.md covering every pair in the result-kind-homogeneous job. Verdict blocks end in an outcome marker; report blocks end in REPORT:

## Result: PASS
## Result: WARN
## Result: FAIL
## Result: ERROR
## Result: REPORT

Before the first pair block, a worker whose environment states its exact model may include one optional self-reported-model: <model-id> line. Omit it when unavailable; never guess. Finalization returns it as self_reported_model and copies it into finalized result frontmatter under the original name. It does not populate runner_model, validate the job's partition, or affect freshness.

Inside every pair block, the worker reports linked-material consumption before the result line:

review-consumption: {"opened_paths": ["kb/notes/example.md"], "stop_reason": "sufficiency"}

opened_paths contains each distinct repo-relative consumption target used from that target note's pre-resolved link table; it excludes the target note and criterion. An empty list means no linked artifact was used. For an ordinary route, the resolved link target and consumption target are the same. For (snapshot required), the table names and prices both physical inputs: the linked ingest metadata and the derived snapshot bytes. The reviewer reports each one actually opened. stop_reason is budget when a reading limit prevented further inspection and sufficiency when the reviewer had enough evidence.

Finalization strips this line from the per-pair result artifact and records it under commonplace.review_link_consumption beside commonplace.review_link_availability in job telemetry. Each consumption row carries report_status; missing, partial, malformed, or unpriced reports retain diagnostics but never affect the pair's result, job completion, or freshness. Only a complete row with no unpriced paths supplies a comparable consumed byte total.

REPORT is a completion marker, not an outcome. ERROR reports inability to produce the contracted result: finalization fails the whole job, completes no pairs, and advances no baselines. INFO may label a non-actionable finding inside verdict text but is not a valid final outcome. Aliases such as Verdict, Outcome, OK, and UNKNOWN are invalid. The generated prompt is the complete job-specific worker contract: dispatch starts a fresh context without the parent conversation and supplies only the prompt path. The worker writes only the output file named there. It does not touch notes, criteria, indexes, or the review database; its conversational response is not a pipeline input.

Authoring a gate

Gates are typed kb/types/review-gate.md. See that type spec for the frontmatter and body contract, and kb/instructions/review-gates/ for examples. This document covers runtime concepts (bundles, freshness, freshness baseline, the batch workflow); the type owns the authored shape of a gate.