VoidBot/notes/voidbot-current-system-map.md

VoidBot Current System Map

This note is the source-grounded description of how the live VoidBot stack is shaped right now. It is not a prose replacement for the code, and it is not a second canonical map. Its job is to explain the major organs and their data flow in plain language with concrete path anchors.

Main Organs

• apps/bot/src/discord-bot.ts
  • Discord gateway shell: client bootstrap, provider wiring, event registration, and top-level orchestration.
• apps/bot/src/discord-bot-handlers.ts
  • Prompt/command handling, provider dispatch, rate-limit gating, job approval/rejection, reindexing, and slash-command registration helpers.
• apps/bot/src/discord-bot-support.ts
  • Discord-shape adapters, archive/source conversion helpers, ambient-memory ingestion helpers, source-grounding hint logic, and bot-side message formatting/rendering helpers.
• apps/worker/src/index.ts
  • owner-job poller, provider execution, handoff packaging, and MCP wiring for the worker-side lane.
• apps/worker/src/mcp-server.ts
  • read-only retrieval tools and resource surfaces for archived history, indexed repos, lore, and owner notifications.
• packages/core/src/state-storage.ts
  • Thin state-storage factory that chooses file or Postgres backends and wires the domain stores together.
• packages/core/src/state-storage-postgres-bootstrap.ts
  • Postgres schema bootstrap plus one-time legacy file-state import.
• packages/core/src/state-storage-postgres-job-queue.ts
  • Postgres-backed job queue plus job row persistence helpers.
• packages/core/src/state-storage-postgres-audit-log.ts
  • Postgres-backed audit log plus audit-event upsert helper.
• packages/core/src/state-storage-postgres-interaction-memory.ts
  • Postgres-backed interaction-memory event storage and profile reconstruction.
• packages/core/src/state-storage-rate-limit-stores.ts
  • File and Postgres implementations for Void usage rate-limit state.
• packages/core/src/interaction-memory-analysis.ts
  • Event construction, tone/tag analysis, repetition detection, and persistence gating for remembered interactions.
• packages/core/src/interaction-memory-profile.ts
  • Profile synthesis, disposition, psychological read, inferred traits, interaction dimensions, and response-guidance construction.
• packages/core/src/context-builder.ts
  • request context assembly, including interaction-memory/profile attachment.
• packages/providers/src/owner-codex-provider.ts
  • thin orchestration layer for the Discord-safe codex exec lane.
• packages/providers/src/owner-codex-runtime.ts
  • Codex process execution, stdout/stderr parsing, trace-event normalization, history-tool loop requests, and owner-DM intent parsing.
• packages/providers/src/owner-codex-render.ts
  • prompt assembly, bundle rendering, and trace/debug transcript rendering for the owner lane.
• packages/providers/src/owner-codex-shared.ts
  • owner-lane constants, shared types, request-payload shaping, trace formatting helpers, and interaction-memory rendering.
• packages/providers/src/local-llm-provider.ts
  • Ollama chat lane with a bounded host-managed read-only tool loop.
• packages/rag/src/retrieval-service.ts
  • high-level history/source retrieval API used by bot, worker, and provider tool loops.
• packages/rag/src/qdrant-vector-store.ts
  • Qdrant vector persistence for history and source collections.
• packages/rag/src/message-archive.ts
  • archived Discord message store under .voidbot/rag/messages.json.
• packages/rag/src/source-document-archive.ts
  • archived source/lore document store under .voidbot/rag/source-documents.json.

Flow 1: Discord Request To Provider

1. apps/bot/src/discord-bot.ts receives a slash command or mention-driven request and delegates prompt/command work into apps/bot/src/discord-bot-handlers.ts.
2. Permission checks run through packages/core/src/permission-engine.ts.
3. Void usage limits are applied through packages/core/src/void-usage-rate-limiter.ts backed by packages/core/src/state-storage.ts.
4. apps/bot/src/discord-bot-support.ts adapts Discord message/interaction shapes, ambient-memory events, and source-grounding hints.
5. packages/core/src/interaction-memory-analysis.ts turns direct prompts or ambient mentions into normalized remembered events; packages/core/src/interaction-memory-profile.ts distills remembered events into a reusable social read.
6. packages/core/src/context-builder.ts assembles request context, including recent interaction profile and any retrieval hints.
7. Provider selection goes through packages/providers/src/index.ts.
8. The bot either:
   • answers directly through local_llm, or
   • queues an owner job for the worker / owner_codex path.

Flow 2: Owner Job Execution

1. apps/worker/src/index.ts polls approved jobs from durable state.
2. The worker claims a job and dispatches it to the configured provider.
3. packages/providers/src/owner-codex-provider.ts runs codex exec in the bounded lane, exposes the VoidBot MCP server, records traces, and enforces source grounding for repo/lore/project questions. Runtime and parsing live in packages/providers/src/owner-codex-runtime.ts; prompt and artifact rendering live in packages/providers/src/owner-codex-render.ts.
4. If the answer fits the Discord-safe lane, the worker posts it back.
5. If the task needs deeper work, the worker writes a handoff bundle under .voidbot/artifacts/<job-id>/ and posts the handoff response.

Flow 3: Retrieval And Indexing

1. packages/rag/src/message-archive.ts and packages/rag/src/source-document-archive.ts keep the raw corpora.
2. packages/rag/src/retrieval-service.ts translates history/source queries into vector lookups plus metadata filters.
3. packages/rag/src/qdrant-vector-store.ts executes the live vector lookups against separate history and source collections.
4. scripts/index-source-repos.ts and scripts/git-post-push-index.mjs drive detached source/lore reindex work.
5. apps/worker/src/mcp-server.ts exposes retrieval to Codex and other sessions through search_history, get_message_context, search_sources, get_source_context, and list_indexed_repos.

Flow 4: Ops And Recovery

• scripts/start-voidbot-stack.ps1
  • stack bootstrap, health checks, fresh build, stale-process cleanup, bot/worker restart, runtime status emission.
• scripts/check-voidbot-operations.ps1
  • watchdog for process liveness, Qdrant, Postgres, Ollama, Discord auth, backup freshness, and offsite sync freshness.
• scripts/backup-voidbot-state.ps1
  • local backup of Postgres, Qdrant snapshots, and RAG archives.
• scripts/restore-voidbot-state.ps1
  • local restore path.
• scripts/sync-voidbot-backup-offsite.ps1
  • verified offsite backup sync to the Qwen box.
• scripts/voidbot-operations-dashboard-lib.ps1
  • dashboard rendering for the local ops surface.

Storage Boundaries

• Postgres:
  • jobs
  • audit events
  • interaction-memory events
  • provider run records
  • usage rate-limit state
• Qdrant:
  • Discord history vectors
  • source/lore vectors
• .voidbot/:
  • archived raw corpora
  • job artifacts and traces
  • logs and status files
  • backups and snapshots

The important scar is that these stores are split on purpose. Do not casually weld them back into one convenient blob and act surprised when it becomes a moon.

Within the Postgres path, the implementation is split on purpose now too:

• state-storage.ts chooses the backend
• state-storage-postgres-bootstrap.ts handles schema/bootstrap/import work
• per-domain modules own queue, audit, interaction-memory, and rate-limit behavior