Why directories despite their costs

Type: kb/types/note.md · Tags: document-system

The KB keeps authored files canonical while meanings and write rules remain unsettled, which also keeps the content directly navigable by humans. But a flat directory with hundreds of files isn't really navigable — you're back to needing tooling (search, indexes) to find anything, which is what a database gives you. Directories preserve the human-navigability guarantee at scale. Not at every scale — but for one or two orders of magnitude more files before the same problem recurs.

What directories give us

Scale without tooling. A notes/ directory with 30 files is browsable. With 300, it isn't. Splitting into notes/, claw-design/, sources/, adr/ keeps each directory in the browsable range. This isn't infinite — at thousands of notes you'd need deeper nesting or actual search infrastructure — but it extends the file-substrate sweet spot considerably.

Local conventions per subsystem. Some directories benefit from their own rules that don't need to generalise. Tasks have lifecycle subdirectories (backlog/, active/, completed/). Sources have an ingest pipeline. These are what the workshop layer note calls "small, self-contained subsystems with their own conventions." Directories are natural boundaries for them.

Different metabolic rates. Sources churn fast (new ones arrive, get ingested, some get pruned). ADRs are nearly permanent. Notes are somewhere in between. Separating by metabolic rate lets you apply different lifecycle expectations — a 200-file sources/ directory is normal; 200 files in adr/ would signal something wrong.

Types and directories are orthogonal

Types assert structural properties of individual documents — what sections are expected, what metadata is required, what's checkable. Directories group documents by topic, lifecycle, provenance, or whatever convention the user finds useful. These are independent axes.

A task in kb/tasks/active/ can use type: kb/tasks/types/task-active.md to define its Goal, Tasks checklist, and Current State structure. Its directory carries a separate lifecycle signal.

A collection-local type works throughout its owning collection, not in arbitrary peer collections. For example, type: kb/notes/types/structured-claim.md has the same contract at the root of kb/notes/ and in its subdirectories. The type path carries structural expectations; subdirectories may carry another organization axis.

If types depended on arbitrary content directories, you'd need to redefine types whenever someone creates a new subdirectory. If those directories encoded type information, you'd lose the freedom to organise by whatever dimension matters — topic, lifecycle, provenance, project area. In the current type system, an artifact points to its contract directly; subdirectories inside the owning collection do not determine its type.

Operational costs of directories

Each new top-level directory imposes a registration tax across multiple places:

  1. CLAUDE.md routing table — the "Where Things Go" table maps content types to directories. New directory = new row + routing heuristic prose explaining when to use it vs neighbours.
  2. qmd-collections.yml — each directory needs its own collection entry for search indexing. Currently 11 entries.
  3. Skills hardcode directory lists — /connect searches across three hardcoded directories (notes/, claw-design/, sources/). /validate only knows about notes/. /convert only knows about notes/. Adding a directory means auditing every skill.
  4. WRITING.md — the "Where It Goes" section duplicates routing guidance. The templates table lists directory-specific templates.
  5. generate_notes_index.py — needs to be invoked per directory (the script itself is directory-agnostic, but someone has to know to run it).

Softer costs:

  1. Cross-directory links — relative path depth varies. notes/foo.md links to ../claw-design/bar.md, but notes/subdir/foo.md needs ../../claw-design/bar.md. More directories = more relative-path arithmetic.
  2. Agent routing decisions — every new directory is a classification decision an agent has to make. The notes/ vs claw-design/ heuristic is already non-trivial ("Is it about general systems connecting LLMs and software, or about a specific genre — the KB genre?"). More directories = more routing errors.

Current stance

The registration tax is real but manageable at the current scale (~6 top-level directories). The benefits — human navigability, local conventions, metabolic separation — outweigh the costs. The danger point is if we keep adding directories without noticing the cumulative tax: each one is small, but the aggregate burden on routing, skills, and search config grows linearly.

Mitigations to consider if directory count grows: - A single registry file that skills and config derive from (instead of N hardcoded lists) - Convention that subdirectories within a collection (like notes/definitions/) don't need registration — only top-level collections do - Tooling to validate that all collections are consistently registered

For now, the right default is: be reluctant to create new top-level directories. Subdirectories within existing collections are cheap. New collections are expensive.

Relevant Notes: