Add persistent project state scaffolding

AGENTS.md

@@ -2,6 +2,72 @@ Use the `voidbot` MCP server whenever the task touches Discord history, archived
 
 Commit completed work at the end of each pass unless the user explicitly asks to leave changes uncommitted or the work is clearly mid-surgery. Prefer small, intentional commits over letting a pile of unrelated edits rot in the worktree.
 
+## Canonical Project State
+
+- Treat `state/map.yaml` as the canonical slow-changing project map.
+- Treat `state/scratch.md` as disposable working memory for one bounded subgoal.
+- Treat `state/evidence.jsonl` as the distilled durable ledger of what changed future belief.
+- Treat `notes/fresh-workspace-handoff.md` as the compact re-entry packet.
+- Treat `notes/voidbot-current-system-map.md` as the source-grounded explanation of the live control flow.
+- Treat `notes/voidbot-implementation-plan.md` as the current larger-organ sequence.
+- Update `state/map.yaml` when project understanding changes.
+- Add evidence after meaningful research, implementation, verification, or rejected paths, but keep it distilled. Routine “I just did this” proof belongs in git history, commits, smoke artifacts, or targeted logs unless it changes what the next session should believe.
+
+## Important Paths
+
+- Project root: `E:\Projects\VoidBot`
+- Canonical map: `E:\Projects\VoidBot\state\map.yaml`
+- Scratch surface: `E:\Projects\VoidBot\state\scratch.md`
+- Distilled evidence ledger: `E:\Projects\VoidBot\state\evidence.jsonl`
+- Branch ledger: `E:\Projects\VoidBot\state\branches.json`
+- Handoff summary: `E:\Projects\VoidBot\notes\fresh-workspace-handoff.md`
+- System map: `E:\Projects\VoidBot\notes\voidbot-current-system-map.md`
+- Implementation plan: `E:\Projects\VoidBot\notes\voidbot-implementation-plan.md`
+- State CLI: `E:\Projects\VoidBot\tools\voidbot_state.ts`
+- Pre-compaction helper: `E:\Projects\VoidBot\tools\voidbot_prepare_compaction.ts`
+
+## Useful Commands
+
+```powershell
+npm run state:status
+npx tsx .\tools\voidbot_state.ts add-evidence --type research --status ok --note "..."
+npx tsx .\tools\voidbot_state.ts add-branch --id branch-id --hypothesis "..."
+npm run state:prepare-compaction
+```
+
+## Session Bootstrap And Re-entry Protocol
+
+On fresh session load, do this before wandering into implementation:
+
+1. read:
+   - `state/map.yaml`
+   - `notes/fresh-workspace-handoff.md`
+   - `notes/voidbot-current-system-map.md`
+   - `notes/voidbot-implementation-plan.md`
+2. run:
+   - `npm run state:status`
+3. restate the current next action from the persisted state before starting edits
+
+After compaction, resume, or suspicious continuity loss:
+
+1. rerun `npm run state:status`
+2. reread `state/map.yaml` and `notes/fresh-workspace-handoff.md`
+3. treat the persisted next action as authoritative unless fresh repo evidence contradicts it
+
+When context pressure is clearly rising:
+
+1. stop broad exploration
+2. narrow the active move to one bounded organ
+3. persist map or handoff updates, plus distilled evidence only when the lesson changes future belief, before the blackout lands
+
+When the user says to prepare for imminent compaction:
+
+1. run `npm run state:prepare-compaction` before editing persistence surfaces
+2. use its warnings as the checklist for map, handoff, scratch, evidence, and git hygiene
+3. update only the state that actually needs to change
+4. rerun `npm run state:prepare-compaction` after edits
+5. fix errors, address warnings, and commit the completed persistence pass unless the work is deliberately mid-surgery
+
 Prefer these tools over raw file inspection when they can answer the question:
 - `list_indexed_repos` to discover valid indexed repository names before narrowing a source search
 - `search_history` for semantic retrieval across archived Discord messages
@@ -13,3 +79,13 @@ Prefer these tools over raw file inspection when they can answer the question:
 When searching GameCult repos or lore, start with `voidbot` retrieval instead of `rg --files` plus file-by-file reads unless you need exact edit targets or the indexed results are clearly insufficient.
 
 Do not inspect `.voidbot/rag/messages.json`, `.voidbot/rag/source-documents.json`, `.voidbot/history-vector-store.json`, or `.voidbot/source-vectors/` directly unless the MCP tools are unavailable or clearly insufficient.
+
+## Operating Discipline
+
+- Before substantial edits, restate the current mechanism and intended change.
+- Prefer one clear hypothesis per iteration.
+- Verify with checks that reflect the real goal, not just proxy success.
+- Revert or discard changes that do not clearly improve the target.
+- If the diff grows while understanding shrinks, stop implementation and switch to diagnosis.
+- Keep maps and prose together; do not replace useful maps with prose-only explanations.
+- Before handoff, compaction, or phase boundaries, sync `state/map.yaml`, add distilled evidence when the lesson changes future belief, refresh `notes/fresh-workspace-handoff.md`, and make the next action explicit.

README.md

@@ -49,6 +49,8 @@ The repo is split into a bot, a worker, and a handful of focused packages:
 
 The durable local state is split on purpose: Postgres holds jobs, audit events, and interaction memory, while `.voidbot/` keeps artifacts, archives, logs, and detached indexing status.
 
+There is also a project-memory spine now for future Codex sessions: `state/map.yaml`, `state/scratch.md`, `state/evidence.jsonl`, `notes/fresh-workspace-handoff.md`, and the helper CLIs under `tools/`. The point is to rehydrate from canonical files instead of pretending the transcript will stay coherent forever.
+
 ## Quick Start
 
 ### 1. Configure `.env`
@@ -509,6 +511,32 @@ Defaults come from `.env` / `.env.example`:
 
 The watchdog task also uses `Interactive` logon because the remote freshness check relies on the same local SSH key material as the offsite sync path.
 
+## Persistent Project State
+
+VoidBot is now large enough that transcript memory alone is a stupid way to steer it. The repo carries a small canonical project-state scaffold for future sessions:
+
+- `state/map.yaml`: slow-changing canonical project map and current next action
+- `state/scratch.md`: disposable one-subgoal working memory
+- `state/branches.json`: hypothesis / branch ledger
+- `state/evidence.jsonl`: distilled belief-changing evidence, rejections, and scars
+- `notes/fresh-workspace-handoff.md`: re-entry packet
+- `notes/voidbot-current-system-map.md`: source-grounded control-flow explanation
+- `notes/voidbot-implementation-plan.md`: larger-organ build order
+
+Useful commands:
+
+```bash
+npm run state:status
+npm run state:prepare-compaction
+```
+
+For direct edits to the ledger surfaces:
+
+```bash
+npx tsx tools/voidbot_state.ts add-evidence --type research --status ok --note "..."
+npx tsx tools/voidbot_state.ts add-branch --id branch-id --hypothesis "..."
+```
+
 ## Commands
 
 Current slash commands:
@@ -538,6 +566,11 @@ Useful local state and docs:
 - `.voidbot/status/operations-health.json`: latest watchdog health report
 - `.voidbot/status/operations-watchdog.json`: live watchdog run-state plus last notification metadata
 - `.voidbot/status/operations-dashboard.html`: auto-refreshing local operations dashboard
+- `state/map.yaml`: canonical project map and current next action
+- `state/evidence.jsonl`: distilled project-memory ledger
+- `notes/fresh-workspace-handoff.md`: re-entry packet for future sessions
+- `notes/voidbot-current-system-map.md`: source-grounded system map
+- `notes/voidbot-implementation-plan.md`: larger-organ plan
 - `config/system-messages.json`: rotating stock system messages
 - `styles/void-default.md`: public default style pack
 - `docs/architecture/overview.md`: higher-level architecture notes

notes/fresh-workspace-handoff.md

@@ -0,0 +1,74 @@
+# VoidBot Fresh Workspace Handoff
+
+VoidBot is no longer small enough to trust chat scrollback as its only memory. Use the canonical state files first, then touch code.
+
+## What To Trust First
+
+For the current project shape and next move, trust these in order:
+
+1. `state/map.yaml`
+2. `notes/fresh-workspace-handoff.md`
+3. `notes/voidbot-current-system-map.md`
+4. `notes/voidbot-implementation-plan.md`
+5. `state/evidence.jsonl`
+
+Git history and smoke artifacts carry routine proof. `state/evidence.jsonl` should carry only belief-changing records, not every tiny triumph with a timestamp stapled to it.
+
+## Current Shape
+
+- Discord bot and worker are live.
+- Postgres owns jobs, audit events, interaction memory, and rate-limit state.
+- Qdrant owns history and source vectors.
+- `.voidbot/` owns archives, artifacts, logs, status files, and backups.
+- `owner_codex` and `local_llm` are the active reply lanes.
+- Ops health, backup verification, offsite sync, and dashboard surfaces already exist.
+- The new state/notes surfaces are now the continuity spine for future nontrivial work.
+
+## Important Paths
+
+- Project root: `E:\Projects\VoidBot`
+- Canonical map: `E:\Projects\VoidBot\state\map.yaml`
+- Scratch surface: `E:\Projects\VoidBot\state\scratch.md`
+- Distilled evidence ledger: `E:\Projects\VoidBot\state\evidence.jsonl`
+- Branch/hypothesis ledger: `E:\Projects\VoidBot\state\branches.json`
+- Handoff summary: `E:\Projects\VoidBot\notes\fresh-workspace-handoff.md`
+- System map: `E:\Projects\VoidBot\notes\voidbot-current-system-map.md`
+- Implementation plan: `E:\Projects\VoidBot\notes\voidbot-implementation-plan.md`
+- State CLI: `E:\Projects\VoidBot\tools\voidbot_state.ts`
+- Pre-compaction helper: `E:\Projects\VoidBot\tools\voidbot_prepare_compaction.ts`
+
+## Useful Commands
+
+```powershell
+npm run state:status
+npx tsx .\tools\voidbot_state.ts add-evidence --type research --status ok --note "..."
+npx tsx .\tools\voidbot_state.ts add-branch --id branch-id --hypothesis "..."
+npm run state:prepare-compaction
+```
+
+## Guardrails
+
+- Do not continue implementation automatically from a rehydrate-only request.
+- Do not trust this file for the exact live HEAD.
+- Do not let `state/evidence.jsonl` become an activity feed.
+- Do not let `state/scratch.md` linger with stale subgoals after the bounded move is done.
+
+## Immediate Re-entry Instruction
+
+On fresh session load or after suspicious continuity loss:
+
+1. read `state/map.yaml`
+2. run `npm run state:status`
+3. reread `notes/fresh-workspace-handoff.md`
+4. read `notes/voidbot-current-system-map.md`
+5. read `notes/voidbot-implementation-plan.md`
+6. restate the current next action from `state/map.yaml` before editing
+
+When context pressure is rising:
+
+1. stop broad exploration
+2. narrow the active move to one bounded organ
+3. persist map or handoff changes only if understanding actually changed
+4. add distilled evidence only when the lesson changes future belief
+
+The point is to bank coals, not leave the next waking thing a pile of ash and a browser tab graveyard.

notes/voidbot-current-system-map.md

@@ -0,0 +1,89 @@
+# VoidBot Current System Map
+
+This note is the source-grounded description of how the live VoidBot stack is shaped right now. It is not a prose replacement for the code, and it is not a second canonical map. Its job is to explain the major organs and their data flow in plain language with concrete path anchors.
+
+## Main Organs
+
+- `apps/bot/src/discord-bot.ts`
+  - Discord gateway, commands, mention handling, permissions, provider selection, rate-limit gating, and context assembly entrypoint.
+- `apps/worker/src/index.ts`
+  - owner-job poller, provider execution, handoff packaging, and MCP wiring for the worker-side lane.
+- `apps/worker/src/mcp-server.ts`
+  - read-only retrieval tools and resource surfaces for archived history, indexed repos, lore, and owner notifications.
+- `packages/core/src/state-storage.ts`
+  - Postgres-backed durable state for jobs, audit events, interaction memory, and Void usage rate-limit counters.
+- `packages/core/src/context-builder.ts`
+  - request context assembly, including interaction-memory/profile attachment.
+- `packages/providers/src/owner-codex-provider.ts`
+  - Discord-safe `codex exec` lane, read-only tool policy, source-grounding enforcement, traces, handoff behavior.
+- `packages/providers/src/local-llm-provider.ts`
+  - Ollama chat lane with a bounded host-managed read-only tool loop.
+- `packages/rag/src/retrieval-service.ts`
+  - high-level history/source retrieval API used by bot, worker, and provider tool loops.
+- `packages/rag/src/qdrant-vector-store.ts`
+  - Qdrant vector persistence for history and source collections.
+- `packages/rag/src/message-archive.ts`
+  - archived Discord message store under `.voidbot/rag/messages.json`.
+- `packages/rag/src/source-document-archive.ts`
+  - archived source/lore document store under `.voidbot/rag/source-documents.json`.
+
+## Flow 1: Discord Request To Provider
+
+1. `apps/bot/src/discord-bot.ts` receives a slash command or mention-driven request.
+2. Permission checks run through `packages/core/src/permission-engine.ts`.
+3. Void usage limits are applied through `packages/core/src/void-usage-rate-limiter.ts` backed by `packages/core/src/state-storage.ts`.
+4. `packages/core/src/context-builder.ts` assembles request context, including recent interaction profile and any retrieval hints.
+5. Provider selection goes through `packages/providers/src/index.ts`.
+6. The bot either:
+   - answers directly through `local_llm`, or
+   - queues an owner job for the worker / `owner_codex` path.
+
+## Flow 2: Owner Job Execution
+
+1. `apps/worker/src/index.ts` polls approved jobs from durable state.
+2. The worker claims a job and dispatches it to the configured provider.
+3. `packages/providers/src/owner-codex-provider.ts` runs `codex exec` in the bounded lane, exposes the VoidBot MCP server, records traces, and enforces source grounding for repo/lore/project questions.
+4. If the answer fits the Discord-safe lane, the worker posts it back.
+5. If the task needs deeper work, the worker writes a handoff bundle under `.voidbot/artifacts/<job-id>/` and posts the handoff response.
+
+## Flow 3: Retrieval And Indexing
+
+1. `packages/rag/src/message-archive.ts` and `packages/rag/src/source-document-archive.ts` keep the raw corpora.
+2. `packages/rag/src/retrieval-service.ts` translates history/source queries into vector lookups plus metadata filters.
+3. `packages/rag/src/qdrant-vector-store.ts` executes the live vector lookups against separate history and source collections.
+4. `scripts/index-source-repos.ts` and `scripts/git-post-push-index.mjs` drive detached source/lore reindex work.
+5. `apps/worker/src/mcp-server.ts` exposes retrieval to Codex and other sessions through `search_history`, `get_message_context`, `search_sources`, `get_source_context`, and `list_indexed_repos`.
+
+## Flow 4: Ops And Recovery
+
+- `scripts/start-voidbot-stack.ps1`
+  - stack bootstrap, health checks, fresh build, stale-process cleanup, bot/worker restart, runtime status emission.
+- `scripts/check-voidbot-operations.ps1`
+  - watchdog for process liveness, Qdrant, Postgres, Ollama, Discord auth, backup freshness, and offsite sync freshness.
+- `scripts/backup-voidbot-state.ps1`
+  - local backup of Postgres, Qdrant snapshots, and RAG archives.
+- `scripts/restore-voidbot-state.ps1`
+  - local restore path.
+- `scripts/sync-voidbot-backup-offsite.ps1`
+  - verified offsite backup sync to the Qwen box.
+- `scripts/voidbot-operations-dashboard-lib.ps1`
+  - dashboard rendering for the local ops surface.
+
+## Storage Boundaries
+
+- Postgres:
+  - jobs
+  - audit events
+  - interaction-memory events
+  - provider run records
+  - usage rate-limit state
+- Qdrant:
+  - Discord history vectors
+  - source/lore vectors
+- `.voidbot/`:
+  - archived raw corpora
+  - job artifacts and traces
+  - logs and status files
+  - backups and snapshots
+
+The important scar is that these stores are split on purpose. Do not casually weld them back into one convenient blob and act surprised when it becomes a moon.

notes/voidbot-implementation-plan.md

@@ -0,0 +1,38 @@
+# VoidBot Implementation Plan
+
+This is the current forward plan for the next larger organs. It is not a changelog and it is not allowed to become a duplicate brain for the canonical map.
+
+## Current Aim
+
+Use the new persistent project-state surfaces to keep future work source-grounded and rehydratable while the stack keeps getting more interconnected.
+
+## Near-Term Organs
+
+### 1. Public-Lane Hardening
+
+- tighten moderation and abuse handling for non-owner traffic
+- make budgets and rate-limit policy easier to inspect/administer
+- preserve the current owner/public provider boundaries instead of letting them blur
+
+### 2. Recovery That Is More Adult Than Hope
+
+- add a real restore-drill cadence instead of stopping at backup verification
+- reduce dependence on interactive workstation logon where practical
+- keep watchdog signals and dashboard diagnostics honest when dependencies are degraded
+
+### 3. Admin And Forensics Surfaces
+
+- richer inspection around jobs, provider runs, tool usage, interaction memory, and rate-limit denials
+- better “what happened?” answers without spelunking raw artifacts by candlelight
+
+### 4. The Still-Scaffolded Paths
+
+- replace the remaining scaffolded provider paths with funded or production-grade implementations
+- move sandbox execution from policy theater to real constrained runners when the policy story is ready
+
+## Rules For Editing This Plan
+
+- update this note when the larger sequence of work changes
+- do not dump volatile minute-by-minute status here
+- if a lesson changes future belief, record it in `state/evidence.jsonl`
+- if the current next action changes, update `state/map.yaml` and `notes/fresh-workspace-handoff.md`

package.json

@@ -13,6 +13,8 @@
     "dev:mcp": "tsx apps/worker/src/mcp-server.ts",
     "dev:worker": "tsx apps/worker/src/index.ts",
     "stack:start": "powershell -ExecutionPolicy Bypass -File scripts/start-voidbot-stack.ps1",
+    "state:status": "tsx tools/voidbot_state.ts status",
+    "state:prepare-compaction": "tsx tools/voidbot_prepare_compaction.ts",
     "state:backup": "powershell -ExecutionPolicy Bypass -File scripts/backup-voidbot-state.ps1",
     "state:restore": "powershell -ExecutionPolicy Bypass -File scripts/restore-voidbot-state.ps1",
     "state:verify-backup": "powershell -ExecutionPolicy Bypass -File scripts/verify-voidbot-backup.ps1",

state/branches.json

@@ -0,0 +1,3 @@
+{
+  "branches": []
+}

state/evidence.jsonl

@@ -0,0 +1,3 @@
+{"ts":"2026-04-24T16:30:00+01:00","type":"architecture","status":"ok","note":"Operational state now lives in Postgres while vector retrieval lives in Qdrant; archives, artifacts, logs, and backups remain under .voidbot so operational truth, semantic retrieval, and raw evidence no longer fight in one store."}
+{"ts":"2026-04-25T13:00:00+01:00","type":"architecture","status":"ok","note":"Tool-driven retrieval is now the default for owner_codex and local_llm; source and lore answers are expected to touch source-side lookup before sounding confident."}
+{"ts":"2026-04-26T03:05:00+01:00","type":"process","status":"ok","note":"Canonical project-state surfaces were added so future sessions can rehydrate from map, handoff, system map, implementation plan, and distilled evidence instead of relying on transcript memory alone."}

state/map.yaml

@@ -0,0 +1,85 @@
+experiment:
+  name: VoidBot
+  status: active
+  phase: persistent_state_adoption
+  premise: >
+    Keep VoidBot operable as a Discord-native assistant with real retrieval,
+    durable ops, and explicit project memory surfaces instead of letting
+    continuity dissolve into transcript fog.
+
+objective:
+  primary: >
+    Maintain and extend VoidBot as a Discord-native assistant for GameCult with
+    grounded answers from archived Discord history, indexed source trees, and
+    Aetheria lore, while keeping the system understandable enough to modify
+    safely.
+  current: >
+    Land persistent project-state scaffolding so future sessions can rehydrate
+    from canonical files before touching the increasingly interconnected stack.
+
+constraints:
+  - Public traffic still rides one remote Qwen box plus local embedding and ops infrastructure.
+  - Owner flow and public flow share infrastructure but have different provider and safety boundaries.
+  - Postgres, Qdrant, Ollama, Docker, and Discord failures are operationally real; observability matters as much as features.
+  - Persistent project state must stay small enough to rehydrate from without becoming a second swamp.
+
+invariants:
+  - The map is canonical and slow-changing.
+  - Scratch is temporary and disposable.
+  - Evidence is durable and distilled; routine activity proof belongs in git history, commits, smoke artifacts, or targeted logs.
+  - Jobs, audit events, interaction memory, and rate-limit state live in Postgres; vectors live in Qdrant; archives, artifacts, logs, and backups remain under `.voidbot/`.
+  - Source, repo, and lore answers must be grounded through source-side retrieval instead of vibes.
+  - When GameCult history, repo, or lore retrieval can answer the question, use the VoidBot MCP surfaces before raw repo spelunking.
+
+data_flow:
+  observe:
+    - user request
+    - repository state
+    - canonical project state files
+    - provider and runtime health
+    - verifier results and smoke outputs
+  process:
+    - rehydrate from map, handoff, system map, and implementation plan
+    - reason over one bounded organ
+    - modify only the surface that deserves the change
+    - verify the seam that matters
+    - record only durable evidence that changes future belief
+    - commit or backtrack
+  cut:
+    - revert failed hypotheses instead of preserving churn
+    - distill obsolete state instead of accumulating transcript-shaped sludge
+    - keep persistent state lean enough that re-entry sharpens the next turn instead of drowning it
+
+active_subgoals:
+  - Make the new project-state surfaces the default continuity path for nontrivial VoidBot work.
+  - Keep public-lane limits, moderation, and provider behavior constrained enough that one 4060 Ti box does not get flayed for sport.
+  - Preserve boring observability around Postgres, Qdrant, Ollama, Docker, backups, and watchdogs.
+
+accepted_design:
+  - Bot, worker, providers, and RAG remain the main structural seams.
+  - State storage stays intentionally split: Postgres for operational state, Qdrant for vectors, `.voidbot/` for archives, artifacts, logs, and backup material.
+  - `owner_codex` and `local_llm` remain distinct lanes with different tool-execution and safety posture.
+  - The VoidBot MCP server is the preferred search plane for archived Discord, indexed GameCult repos, Aetheria lore, and owner notifications.
+  - Long-running work should expose durable logs or status files instead of relying on one attached session and hope.
+  - Persistent project memory must stay ruthlessly distilled; git history and artifacts carry routine proof.
+
+landed_baseline:
+  phases:
+    - Discord bot, worker, provider registry, and retrieval stack are live.
+    - Postgres-backed jobs, audit events, interaction memory, and rate-limit state are live.
+    - Qdrant-backed history and source retrieval are live.
+    - Local and offsite backups, watchdogs, and the operations dashboard are live.
+    - Interaction memory and behavioral profiling are live.
+    - Source-grounding enforcement and tool-driven retrieval are live.
+
+rejected_paths:
+  - name: transcript_only_continuity
+    reason: repo complexity is now too high to trust chat fog as the only memory surface
+  - name: monolithic_local_state_blobs
+    reason: giant shared files for unrelated state hide costs and rot the next rehydrate
+  - name: source_answers_without_retrieval
+    reason: vibes produce plausible slop that sounds smart and lies with a straight face
+
+current_status:
+  summary: VoidBot now spans Discord routing, background jobs, Postgres state, Qdrant retrieval, source and lore indexing, provider-specific tool loops, ops automation, and a growing social-memory layer. The system works, but it is too interconnected to steer safely from transcript memory alone, so canonical project-state surfaces are being added to keep future work source-grounded and rehydratable.
+  next_action: After this persistence pass, rehydrate from the canonical state files before the next nontrivial change and keep the system map and evidence ledger fresh instead of letting continuity fall back to chat scrollback.

state/scratch.md

@@ -0,0 +1,10 @@
+# VoidBot Scratch
+
+## Current Subgoal
+
+No active scratch subgoal.
+
+## Working Notes
+
+- Replace this file freely while working one bounded move.
+- Distill any lasting lesson into `state/map.yaml`, `state/evidence.jsonl`, or `notes/fresh-workspace-handoff.md` before clearing it.

tools/voidbot_prepare_compaction.ts

@@ -0,0 +1,349 @@
+import { existsSync, readFileSync, statSync } from "node:fs";
+import { resolve } from "node:path";
+import { spawnSync } from "node:child_process";
+
+type FindingLevel = "ok" | "warn" | "error";
+
+type Finding = {
+  level: FindingLevel;
+  message: string;
+};
+
+type BranchRecord = {
+  id?: string;
+  status?: string;
+};
+
+const root = process.cwd();
+const stateDir = resolve(root, "state");
+const notesDir = resolve(root, "notes");
+
+const mapPath = resolve(stateDir, "map.yaml");
+const scratchPath = resolve(stateDir, "scratch.md");
+const branchesPath = resolve(stateDir, "branches.json");
+const evidencePath = resolve(stateDir, "evidence.jsonl");
+const handoffPath = resolve(notesDir, "fresh-workspace-handoff.md");
+const systemMapPath = resolve(notesDir, "voidbot-current-system-map.md");
+const planPath = resolve(notesDir, "voidbot-implementation-plan.md");
+const agentsPath = resolve(root, "AGENTS.md");
+
+function readText(path: string): string {
+  return readFileSync(path, "utf8");
+}
+
+function runGit(args: string[]): string {
+  const completed = spawnSync("git", args, {
+    cwd: root,
+    encoding: "utf8",
+  });
+  if (completed.status !== 0) {
+    throw new Error((completed.stderr || completed.stdout || "git command failed").trim());
+  }
+  return completed.stdout.trim();
+}
+
+function extractMapField(name: string): string | undefined {
+  const prefix = `  ${name}:`;
+  for (const line of readText(mapPath).split(/\r?\n/)) {
+    if (line.startsWith(prefix)) {
+      return line.split(":", 2)[1]?.trim();
+    }
+  }
+  return undefined;
+}
+
+function extractActiveSubgoals(): string[] {
+  const lines = readText(mapPath).split(/\r?\n/);
+  const results: string[] = [];
+  let inSection = false;
+  for (const line of lines) {
+    if (!inSection) {
+      if (line.trim() === "active_subgoals:") {
+        inSection = true;
+      }
+      continue;
+    }
+    if (line.startsWith("  - ")) {
+      results.push(line.slice(4).trim());
+      continue;
+    }
+    if (line && !line.startsWith(" ")) {
+      break;
+    }
+  }
+  return results;
+}
+
+function currentScratchSubgoal(): string | undefined {
+  const lines = readText(scratchPath).split(/\r?\n/);
+  for (let index = 0; index < lines.length; index += 1) {
+    if (lines[index]?.trim() !== "## Current Subgoal") {
+      continue;
+    }
+    for (let offset = index + 1; offset < lines.length; offset += 1) {
+      const stripped = lines[offset]?.trim();
+      if (stripped) {
+        return stripped;
+      }
+    }
+  }
+  return undefined;
+}
+
+function loadEvidence(): Record<string, unknown>[] {
+  const records: Record<string, unknown>[] = [];
+  const lines = readText(evidencePath).split(/\r?\n/);
+  lines.forEach((line, index) => {
+    if (!line.trim()) {
+      return;
+    }
+    let record: unknown;
+    try {
+      record = JSON.parse(line);
+    } catch (error) {
+      throw new Error(
+        `evidence line ${index + 1} is invalid JSON: ${error instanceof Error ? error.message : String(error)}`,
+      );
+    }
+    if (!record || Array.isArray(record) || typeof record !== "object") {
+      throw new Error(`evidence line ${index + 1} is not a JSON object`);
+    }
+    records.push(record as Record<string, unknown>);
+  });
+  return records;
+}
+
+function loadBranches(): { branches: BranchRecord[] } {
+  const data = JSON.parse(readText(branchesPath)) as { branches?: unknown };
+  if (!data || typeof data !== "object" || !Array.isArray(data.branches)) {
+    throw new Error("branches.json must contain a branches array");
+  }
+  return { branches: data.branches as BranchRecord[] };
+}
+
+function addRequiredFileChecks(findings: Finding[]): void {
+  for (const path of [
+    mapPath,
+    scratchPath,
+    branchesPath,
+    evidencePath,
+    handoffPath,
+    systemMapPath,
+    planPath,
+    agentsPath,
+  ]) {
+    if (existsSync(path)) {
+      findings.push({ level: "ok", message: `found ${path.slice(root.length + 1)}` });
+    } else {
+      findings.push({ level: "error", message: `missing ${path.slice(root.length + 1)}` });
+    }
+  }
+}
+
+function addContentChecks(findings: Finding[]): Record<string, unknown>[] {
+  const summary = extractMapField("summary");
+  const nextAction = extractMapField("next_action");
+  if (summary) {
+    findings.push({ level: "ok", message: "state/map.yaml has current_status.summary" });
+  } else {
+    findings.push({ level: "error", message: "state/map.yaml is missing current_status.summary" });
+  }
+  if (nextAction) {
+    findings.push({ level: "ok", message: "state/map.yaml has current_status.next_action" });
+  } else {
+    findings.push({ level: "error", message: "state/map.yaml is missing current_status.next_action" });
+  }
+
+  const subgoals = extractActiveSubgoals();
+  if (subgoals.length > 0) {
+    findings.push({ level: "ok", message: `state/map.yaml has ${subgoals.length} active subgoal(s)` });
+  } else {
+    findings.push({ level: "warn", message: "state/map.yaml has no active_subgoals entries" });
+  }
+
+  const scratchSubgoal = currentScratchSubgoal();
+  if (scratchSubgoal === "No active scratch subgoal.") {
+    findings.push({ level: "ok", message: "state/scratch.md has no stale active scratch subgoal" });
+  } else if (scratchSubgoal) {
+    findings.push({ level: "warn", message: `state/scratch.md has active scratch subgoal: ${scratchSubgoal}` });
+  } else {
+    findings.push({ level: "warn", message: "state/scratch.md has no Current Subgoal value" });
+  }
+
+  const evidence = loadEvidence();
+  findings.push({ level: "ok", message: `state/evidence.jsonl parses (${evidence.length} record(s))` });
+  if (statSync(evidencePath).size > 25_000) {
+    findings.push({ level: "warn", message: "state/evidence.jsonl is larger than 25 KB; consider distillation" });
+  }
+
+  const branches = loadBranches();
+  const active = branches.branches.filter((branch) => branch?.status === "active");
+  findings.push({ level: "ok", message: `state/branches.json parses (${active.length} active branch(es))` });
+
+  return evidence;
+}
+
+function addHandoffChecks(findings: Finding[]): void {
+  const text = readText(handoffPath);
+  if (/Current branch before .*ahead \d+/i.test(text) || /Current HEAD before .*\b[0-9a-f]{7,40}\b/i.test(text)) {
+    findings.push({ level: "error", message: "handoff embeds an exact branch or HEAD snapshot; use git commands instead" });
+  } else {
+    findings.push({ level: "ok", message: "handoff avoids exact branch or HEAD snapshots" });
+  }
+
+  const requiredPhrases = [
+    "Do not continue implementation automatically from a rehydrate-only request.",
+    "Do not trust this file for the exact live HEAD.",
+    "Immediate Re-entry Instruction",
+  ];
+  for (const phrase of requiredPhrases) {
+    if (text.includes(phrase)) {
+      findings.push({ level: "ok", message: `handoff contains: ${phrase}` });
+    } else {
+      findings.push({ level: "warn", message: `handoff missing: ${phrase}` });
+    }
+  }
+}
+
+function addAgentsChecks(findings: Finding[]): void {
+  const text = readText(agentsPath);
+  if (text.includes("voidbot_prepare_compaction.ts")) {
+    findings.push({ level: "ok", message: "AGENTS.md tells agents to use the compaction helper" });
+  } else {
+    findings.push({ level: "error", message: "AGENTS.md does not mention tools/voidbot_prepare_compaction.ts" });
+  }
+  if (text.toLowerCase().includes("prepare for imminent compaction")) {
+    findings.push({ level: "ok", message: "AGENTS.md names the imminent-compaction trigger" });
+  } else {
+    findings.push({ level: "warn", message: "AGENTS.md does not name the imminent-compaction trigger phrase" });
+  }
+}
+
+function addGitChecks(findings: Finding[]): { status: string; log: string } {
+  const status = runGit(["status", "--short", "--branch"]);
+  const log = runGit(["log", "--oneline", "-5"]);
+  const dirtyLines = status
+    .split(/\r?\n/)
+    .slice(1)
+    .filter((line) => line.trim().length > 0);
+  if (dirtyLines.length > 0) {
+    findings.push({ level: "warn", message: "git worktree has uncommitted changes; commit or explain before compaction" });
+  } else {
+    findings.push({ level: "ok", message: "git worktree is clean" });
+  }
+  return { status, log };
+}
+
+function renderReport(
+  findings: Finding[],
+  status: string,
+  log: string,
+  evidence: Record<string, unknown>[],
+): string {
+  const summary = extractMapField("summary") ?? "(missing)";
+  const nextAction = extractMapField("next_action") ?? "(missing)";
+  const subgoals = extractActiveSubgoals();
+  const counts = {
+    ok: findings.filter((finding) => finding.level === "ok").length,
+    warn: findings.filter((finding) => finding.level === "warn").length,
+    error: findings.filter((finding) => finding.level === "error").length,
+  };
+
+  const lines = [
+    "VoidBot pre-compaction persistence check",
+    `Workspace: ${root}`,
+    `Findings: ${counts.ok} ok, ${counts.warn} warn, ${counts.error} error`,
+    "",
+    "Git status:",
+    status,
+    "",
+    "Recent commits:",
+    log,
+    "",
+    `Summary: ${summary}`,
+    `Next action: ${nextAction}`,
+  ];
+
+  if (subgoals.length > 0) {
+    lines.push("Active subgoals:");
+    lines.push(...subgoals.map((item) => `- ${item}`));
+  }
+
+  if (evidence.length > 0) {
+    const latest = evidence[evidence.length - 1] as Record<string, unknown>;
+    lines.push(
+      "",
+      "Latest distilled evidence:",
+      `- ${String(latest.ts ?? "(missing ts)")} ${String(latest.type ?? "(missing type)")}/${String(latest.status ?? "(missing status)")}: ${String(latest.note ?? "(missing note)")}`,
+    );
+  }
+
+  lines.push("", "Findings:");
+  for (const finding of findings) {
+    lines.push(`[${finding.level.toUpperCase()}] ${finding.message}`);
+  }
+
+  lines.push(
+    "",
+    "Pre-compaction checklist:",
+    "- Update state/map.yaml only if current understanding changed.",
+    "- Refresh notes/fresh-workspace-handoff.md if re-entry instructions changed.",
+    "- Add distilled evidence only for a belief-changing lesson, verification, rejected path, or scar.",
+    "- Keep exact branch or HEAD out of handoff prose; git commands own volatile truth.",
+    "- Commit completed persistence changes, or state why the worktree must stay dirty.",
+    "- Re-run this helper after edits before yielding to compaction.",
+  );
+
+  return lines.join("\n");
+}
+
+function main(argv: string[]): number {
+  const strict = argv.includes("--strict");
+  const findings: Finding[] = [];
+  let evidence: Record<string, unknown>[] = [];
+
+  addRequiredFileChecks(findings);
+  try {
+    evidence = addContentChecks(findings);
+  } catch (error) {
+    findings.push({
+      level: "error",
+      message: `state content check failed: ${error instanceof Error ? error.message : String(error)}`,
+    });
+  }
+  try {
+    addHandoffChecks(findings);
+  } catch (error) {
+    findings.push({
+      level: "error",
+      message: `handoff check failed: ${error instanceof Error ? error.message : String(error)}`,
+    });
+  }
+  try {
+    addAgentsChecks(findings);
+  } catch (error) {
+    findings.push({
+      level: "error",
+      message: `AGENTS check failed: ${error instanceof Error ? error.message : String(error)}`,
+    });
+  }
+
+  let status = "(git status unavailable)";
+  let log = "(git log unavailable)";
+  try {
+    ({ status, log } = addGitChecks(findings));
+  } catch (error) {
+    findings.push({
+      level: "error",
+      message: `git check failed: ${error instanceof Error ? error.message : String(error)}`,
+    });
+  }
+
+  console.log(renderReport(findings, status, log, evidence));
+
+  const hasError = findings.some((finding) => finding.level === "error");
+  const hasWarning = findings.some((finding) => finding.level === "warn");
+  return hasError || (strict && hasWarning) ? 1 : 0;
+}
+
+process.exitCode = main(process.argv.slice(2));

tools/voidbot_state.ts

@@ -0,0 +1,220 @@
+import { appendFileSync, readFileSync, writeFileSync } from "node:fs";
+import { resolve } from "node:path";
+
+type Json = null | boolean | number | string | Json[] | { [key: string]: Json };
+
+type BranchRecord = {
+  id: string;
+  hypothesis: string;
+  status: "active" | "accepted" | "rejected" | "archived";
+  artifacts?: string[];
+  notes?: string;
+};
+
+type BranchStore = {
+  branches: BranchRecord[];
+};
+
+const root = process.cwd();
+const stateDir = resolve(root, "state");
+const mapPath = resolve(stateDir, "map.yaml");
+const branchesPath = resolve(stateDir, "branches.json");
+const evidencePath = resolve(stateDir, "evidence.jsonl");
+
+function readText(path: string): string {
+  return readFileSync(path, "utf8");
+}
+
+function writeText(path: string, content: string): void {
+  writeFileSync(path, content, "utf8");
+}
+
+function loadBranches(): BranchStore {
+  return JSON.parse(readText(branchesPath)) as BranchStore;
+}
+
+function saveBranches(data: BranchStore): void {
+  writeText(branchesPath, `${JSON.stringify(data, null, 2)}\n`);
+}
+
+function appendEvidence(record: Record<string, Json>): void {
+  appendFileSync(evidencePath, `${JSON.stringify(record)}\n`, "utf8");
+}
+
+function utcStamp(): string {
+  return new Date().toISOString().replace(/\.\d{3}Z$/, "Z");
+}
+
+function extractMapField(name: string): string | undefined {
+  const prefix = `  ${name}:`;
+  for (const line of readText(mapPath).split(/\r?\n/)) {
+    if (line.startsWith(prefix)) {
+      return line.split(":", 2)[1]?.trim();
+    }
+  }
+  return undefined;
+}
+
+function extractActiveSubgoals(): string[] {
+  const lines = readText(mapPath).split(/\r?\n/);
+  const results: string[] = [];
+  let inSection = false;
+  for (const line of lines) {
+    if (!inSection) {
+      if (line.trim() === "active_subgoals:") {
+        inSection = true;
+      }
+      continue;
+    }
+    if (line.startsWith("  - ")) {
+      results.push(line.slice(4).trim());
+      continue;
+    }
+    if (line && !line.startsWith(" ")) {
+      break;
+    }
+  }
+  return results;
+}
+
+function parseOptions(args: string[]): Map<string, string[]> {
+  const options = new Map<string, string[]>();
+  for (let index = 0; index < args.length; index += 1) {
+    const token = args[index];
+    if (!token?.startsWith("--")) {
+      throw new Error(`Unexpected argument '${token}'.`);
+    }
+    const key = token.slice(2);
+    const value = args[index + 1];
+    if (!value || value.startsWith("--")) {
+      throw new Error(`Missing value for --${key}.`);
+    }
+    const existing = options.get(key) ?? [];
+    existing.push(value);
+    options.set(key, existing);
+    index += 1;
+  }
+  return options;
+}
+
+function getRequired(options: Map<string, string[]>, key: string): string {
+  const value = options.get(key)?.[0];
+  if (!value) {
+    throw new Error(`Missing required option --${key}.`);
+  }
+  return value;
+}
+
+function cmdStatus(): number {
+  const branches = loadBranches().branches ?? [];
+  const active = branches.filter((branch) => branch.status === "active");
+  const summary = extractMapField("summary") ?? "(missing)";
+  const nextAction = extractMapField("next_action") ?? "(missing)";
+  const subgoals = extractActiveSubgoals();
+
+  console.log(`Workspace: ${root}`);
+  console.log(`Summary: ${summary}`);
+  console.log(`Next action: ${nextAction}`);
+  console.log(`Active branches: ${active.length} / ${branches.length}`);
+  if (subgoals.length > 0) {
+    console.log("Active subgoals:");
+    for (const item of subgoals) {
+      console.log(`- ${item}`);
+    }
+  }
+  return 0;
+}
+
+function cmdAddEvidence(args: string[]): number {
+  const options = parseOptions(args);
+  const record: Record<string, Json> = {
+    ts: utcStamp(),
+    type: getRequired(options, "type"),
+    status: getRequired(options, "status"),
+    note: getRequired(options, "note"),
+  };
+  const branch = options.get("branch")?.[0];
+  if (branch) {
+    record.branch = branch;
+  }
+  appendEvidence(record);
+  console.log("Appended evidence record.");
+  return 0;
+}
+
+function cmdAddBranch(args: string[]): number {
+  const options = parseOptions(args);
+  const data = loadBranches();
+  const id = getRequired(options, "id");
+  if (data.branches.some((branch) => branch.id === id)) {
+    throw new Error(`Branch '${id}' already exists.`);
+  }
+  data.branches.push({
+    id,
+    hypothesis: getRequired(options, "hypothesis"),
+    status: "active",
+    artifacts: options.get("artifact"),
+    notes: options.get("note")?.[0],
+  });
+  saveBranches(data);
+  console.log(`Added branch '${id}'.`);
+  return 0;
+}
+
+function cmdCloseBranch(args: string[]): number {
+  const options = parseOptions(args);
+  const id = getRequired(options, "id");
+  const status = getRequired(options, "status");
+  if (!["accepted", "rejected", "archived"].includes(status)) {
+    throw new Error("--status must be one of accepted, rejected, archived.");
+  }
+  const data = loadBranches();
+  const branch = data.branches.find((item) => item.id === id);
+  if (!branch) {
+    throw new Error(`Branch '${id}' was not found.`);
+  }
+  branch.status = status as BranchRecord["status"];
+  const note = options.get("note")?.[0];
+  if (note) {
+    branch.notes = note;
+  }
+  saveBranches(data);
+  console.log(`Updated branch '${id}' to status '${status}'.`);
+  return 0;
+}
+
+function usage(): never {
+  throw new Error(
+    [
+      "Usage:",
+      "  npx tsx tools/voidbot_state.ts status",
+      "  npx tsx tools/voidbot_state.ts add-evidence --type TYPE --status STATUS --note NOTE [--branch ID]",
+      "  npx tsx tools/voidbot_state.ts add-branch --id ID --hypothesis TEXT [--artifact PATH ...] [--note TEXT]",
+      "  npx tsx tools/voidbot_state.ts close-branch --id ID --status accepted|rejected|archived [--note TEXT]",
+    ].join("\n"),
+  );
+}
+
+function main(argv: string[]): number {
+  const [command, ...args] = argv;
+  switch (command) {
+    case "status":
+      return cmdStatus();
+    case "add-evidence":
+      return cmdAddEvidence(args);
+    case "add-branch":
+      return cmdAddBranch(args);
+    case "close-branch":
+      return cmdCloseBranch(args);
+    default:
+      usage();
+  }
+}
+
+try {
+  process.exitCode = main(process.argv.slice(2));
+} catch (error) {
+  const message = error instanceof Error ? error.message : String(error);
+  console.error(message);
+  process.exitCode = 1;
+}