Expose exact source documents through MCP

README.md

@@ -274,6 +274,7 @@ The `local_llm` provider uses Ollama chat completion plus a bounded host-managed
 - `get_message_context`
 - `search_sources`
 - `get_source_context`
+- `get_exact_source_document`
 - `list_indexed_repos`
 
 Unlike the owner Codex lane, this is not MCP. It is a tighter host-controlled loop and deliberately excludes side-effecting tools.

apps/worker/src/mcp-server-shared.ts

@@ -105,6 +105,10 @@ export const sourceContextInputSchema = {
   after: z.number().int().min(0).max(20).optional(),
 };
 
+export const exactSourceDocumentInputSchema = {
+  sourceId: z.string().min(1),
+};
+
 export interface SearchHistoryArgs {
   query: string;
   limit?: number;
@@ -210,6 +214,10 @@ export interface SourceContextArgs {
   after?: number;
 }
 
+export interface ExactSourceDocumentArgs {
+  sourceId: string;
+}
+
 export function formatArchivedMessage(
   message: ArchivedMessageRecord,
   anchorMessageId: string,

apps/worker/src/mcp-server-tools.ts

@@ -46,6 +46,7 @@ import {
   type SearchHistoryArgs,
   type SearchSourcesArgs,
   type SourceContextArgs,
+  type ExactSourceDocumentArgs,
   type UpdateSharedDocumentArgs,
   formatArchivedMessage,
   formatHistoryResults,
@@ -67,6 +68,7 @@ import {
   searchHistoryInputSchema,
   searchSourcesInputSchema,
   sourceContextInputSchema,
+  exactSourceDocumentInputSchema,
   updateSharedDocumentInputSchema,
 } from "./mcp-server-shared";
 
@@ -554,6 +556,45 @@ export function registerVoidbotTools(
     },
   );
 
+  registerIfAllowed(
+    "get_exact_source_document",
+    {
+      title: "Get Exact Source Document",
+      description:
+        "Fetch one complete indexed source document by its exact sourceId. This is the canonical read-only witness path for consumers that must hash and receipt the full source rather than a retrieval chunk.",
+      inputSchema: exactSourceDocumentInputSchema,
+      annotations: READ_ONLY_ANNOTATIONS,
+    },
+    async (input: ExactSourceDocumentArgs): Promise<CallToolResult> => {
+      const document = await context.sourceArchiveRepository.get(input.sourceId);
+
+      if (!document) {
+        return {
+          content: [{ type: "text", text: `No indexed source document with ID ${input.sourceId} was found.` }],
+          structuredContent: { found: false, sourceId: input.sourceId },
+          isError: true,
+        };
+      }
+
+      const exactDocument = {
+        found: true,
+        sourceId: document.id,
+        repoName: document.repoName,
+        path: document.path,
+        language: document.language,
+        title: document.title,
+        content: document.content,
+        lastModifiedAt: document.lastModifiedAt,
+        indexedAt: document.indexedAt,
+        metadata: document.metadata ?? {},
+      };
+      return {
+        content: [{ type: "text", text: renderJsonBlock(exactDocument) }],
+        structuredContent: exactDocument,
+      };
+    },
+  );
+
   registerIfAllowed(
     "get_message_context",
     {

notes/fresh-workspace-handoff.md

@@ -217,6 +217,7 @@ Git history and smoke artifacts carry routine proof. `state/evidence.jsonl` shou
 - Revision `f97e67b` adds the portable moderation-heartbeat coordinator and enforcement organ behind a separate disabled-by-default 15-minute cadence. The model returns an exact review ledger plus urgent IDs and typed case operations; missing evidence or unaccounted urgent evidence blocks cursor advancement. The attempt is receipted by latest message before inference, so one unchanged archive window cannot burn tokens repeatedly. Case enforcement is separate, scans typed pending debt on every cadence, and resolves a case only after a required Discord ban succeeds. Ygg's first `log_only` window at `0629fc7` failed safely with one 401, zero cases, zero cursor movement, and zero sanctions; both supported auth-header shapes prove the mounted MiMo credential itself is stale. `VOID_MODERATION_DAEMON_ENABLED` is false again pending a refreshed provider credential or deliberate alternate model port.
 - The shared Bifrost Discord mouth now lives in `packages/core/src/bifrost-discord-command.ts`. Worker repo-Face speech and resident Void candidate delivery use the same stable-id typed command, provider-store write, optional Bifrost pump, and completed-receipt check. The candidate organ selects at most one targeted queued candidate and writes its speech receipt/`spoken` transition only after transport completes. Ygg mounts the Bifrost provider store and CultLib runtime into the scheduler; `VOID_CANDIDATE_DELIVERY_ENABLED` controls this transport-only organ independently from the disabled moderation/rumination model cadence.
 - Discord archive/source retrieval is a resident worker organ on Yggdrasil. `apps/worker/src/mcp-http-server.ts` publishes stateless streamable HTTP at `127.0.0.1:17875/mcp` and receives the worker's existing archive repositories, source ingester, and retrieval service by injection. The canonical workstation SSH tunnel maps the same Starfire loopback port to Ygg. Codex attaches to `scripts/serve-voidbot-mcp-bridge.mjs`, a task-lifetime stdio transport bridge with no retrieval authority; each tool request uses HTTP to the daemon, and a deployment/tunnel gap returns one MCP tool error without closing the outer transport. It starts no SSH, Docker, or remote Node process. `scripts/smoke-voidbot-mcp-bridge-recovery.mjs` proves one outer client succeeds before an endpoint outage, receives an error during it, and succeeds again after recovery.
+- Source retrieval now also exposes read-only `get_exact_source_document`. It resolves an exact source id through the canonical sharded archive and returns the complete archived document plus provenance fields, allowing Ghostlight and other consumers to hash and receipt the actual witness instead of treating a semantic chunk as the source. The in-memory MCP smoke proves full-content identity. The recovery Compose allow-list and Yggdrasil release still need the matching update before the live crossing offers this sixth tool.
 - Resident person-shaped rumination is wired behind `VOID_RUMINATION_DAEMON_ENABLED` with its own bounded cadence, Discord evidence window, dedicated doctrine, rules, and voice inputs. The pressure fingerprint is receipted before the model attempt and unchanged pressure cannot retry on the observation clock. Ygg explicitly keeps this false while the MiMo credential is stale; candidate delivery may remain enabled because it performs no inference and the live queue was empty at rollout.
 - Revisions `c0cf875` and `1527b29` extract person-shaped rumination and candidate delivery without scheduling them. Rumination context, operation policy, advocacy accountability, tool-free text actuation, and coordinator are portable; the attempt key hashes meaning-bearing pressure/evidence and ignores elapsed clock, so unchanged state cannot create periodic thought. Candidate delivery is deterministic and parent-owned: one targeted queued candidate, duplicate-receipt refusal, injected Bifrost port, then one `mark_candidate_intervention_spoken` operation with the transport receipt. The next cut is extracting the worker's existing typed Bifrost Discord command path into a shared port rather than duplicating CultMesh mechanics.
 - Deterministic state-packet assembly now lives in `persona-state-packet-projector.ts`, which takes typed state, registry/room evidence, pronoun guidance, curiosity facts, and an explicit observation time. The shell no longer owns section ordering or social-context composition. The downstream regex vocabulary cleaner was deleted: it was a symptom compensator rewriting ordinary words such as `witness` and `canary` after projection. A smoke proves literal source language survives the projector unchanged.

notes/voidbot-current-system-map.md

@@ -110,7 +110,7 @@ VoidBot runs on Yggdrasil. Idunn owns deployment and consumes upstream repositor
 4. `packages/rag/src/qdrant-vector-store.ts` executes the live vector lookups against separate history and source collections.
 5. `scripts/reconcile-source-repos.ts` discovers local Git repos, refreshes push hooks, prunes stale repo shards, and indexes newly discovered repos.
 6. `scripts/index-source-repos.ts` and `scripts/git-post-push-index.mjs` drive explicit or detached per-repo source/lore reindex work.
-7. `apps/worker/src/mcp-server.ts` boots the MCP lane, while `mcp-server-resources.ts` and `mcp-server-tools.ts` expose retrieval to Codex and other sessions through `search_history`, `get_message_context`, `search_sources`, `get_source_context`, and `list_indexed_repos`.
+7. `apps/worker/src/mcp-server.ts` boots the MCP lane, while `mcp-server-resources.ts` and `mcp-server-tools.ts` expose retrieval to Codex and other sessions through `search_history`, `get_message_context`, `search_sources`, `get_source_context`, `get_exact_source_document`, and `list_indexed_repos`. Exact-document reads come from the canonical read-only source archive and return the complete archived content for consumer-owned hashing and receipts; they do not read a checkout or mutate the archive.
 8. The resident worker starts `mcp-http-server.ts` beside its job loop and injects the same long-lived archive repositories, source ingester, and retrieval service used by the daemon. It publishes stateless streamable HTTP at Ygg loopback `127.0.0.1:17875/mcp`, mapped to Starfire by the canonical SSH tunnel. Codex connects to one task-local `serve-voidbot-mcp-bridge.mjs` process over stdio. That bridge owns only outer transport continuity: it creates a lightweight HTTP MCP client per tool request, converts daemon/tunnel unavailability into an ordinary tool error, and remains alive for later calls. It cannot read archives, Qdrant, repositories, or Void state itself and starts no SSH, Docker, or remote process. Qdrant stays loopback-private on Yggdrasil. The separate retained stdio server entrypoint exists only for bounded child-provider capability isolation inside the worker body.
 
 ## Flow 4: Ops And Recovery

package.json

@@ -73,7 +73,8 @@
     "rag:reconcile-sources": "tsx scripts/reconcile-source-repos.ts",
     "rag:migrate-qdrant": "tsx scripts/migrate-local-vectors-to-qdrant.ts",
     "rag:migrate-vector-layout": "tsx scripts/migrate-vector-store-layout.ts",
-    "rag:rebuild": "tsx scripts/rebuild-rag-index.ts"
+    "rag:rebuild": "tsx scripts/rebuild-rag-index.ts",
+    "rag:exact-source-mcp-smoke": "tsx scripts/smoke-exact-source-document-mcp.ts"
   },
   "devDependencies": {
     "@types/node": "^24.0.0",

scripts/smoke-exact-source-document-mcp.ts

@@ -0,0 +1,54 @@
+import { mkdtemp, rm } from "node:fs/promises";
+import { tmpdir } from "node:os";
+import { join } from "node:path";
+
+import { Client } from "@modelcontextprotocol/sdk/client/index.js";
+import { InMemoryTransport } from "@modelcontextprotocol/sdk/inMemory.js";
+import {
+  FileSourceDocumentArchiveRepository,
+  SourceDocumentIngester,
+} from "@voidbot/rag";
+
+import { createVoidbotMcpServer } from "../apps/worker/src/mcp-server-factory";
+import { type VoidbotMcpContext } from "../apps/worker/src/mcp-server-context";
+
+async function main(): Promise<void> {
+  const root = await mkdtemp(join(tmpdir(), "voidbot-exact-source-"));
+  try {
+  const sources = new FileSourceDocumentArchiveRepository(join(root, "sources.json"));
+  await sources.syncRepoDocuments("AetheriaLore", [{
+    id: "AetheriaLore:places/forge.md",
+    repoName: "AetheriaLore",
+    path: "places/forge.md",
+    content: "John keeps the village forge.\nThe eastern road takes six hours.",
+    language: "markdown",
+    title: "The Village Forge",
+  }]);
+  const context = {
+    sourceArchiveRepository: sources,
+    sourceDocumentIngester: new SourceDocumentIngester(),
+  } as VoidbotMcpContext;
+  const server = createVoidbotMcpServer(context);
+  const client = new Client({ name: "exact-source-smoke", version: "1.0.0" });
+  const [clientTransport, serverTransport] = InMemoryTransport.createLinkedPair();
+  await Promise.all([server.connect(serverTransport), client.connect(clientTransport)]);
+  const result = await client.callTool({
+    name: "get_exact_source_document",
+    arguments: { sourceId: "AetheriaLore:places/forge.md" },
+  });
+  const typed = result.structuredContent as Record<string, unknown>;
+  if (typed.found !== true || typed.content !== "John keeps the village forge.\nThe eastern road takes six hours.") {
+    throw new Error(`exact source MCP contract failed: ${JSON.stringify(typed)}`);
+  }
+  await client.close();
+  await server.close();
+  console.log("Exact source MCP smoke passed.");
+  } finally {
+    await rm(root, { recursive: true, force: true });
+  }
+}
+
+void main().catch((error) => {
+  console.error(error);
+  process.exitCode = 1;
+});

state/evidence.jsonl

@@ -461,3 +461,4 @@
 {"ts":"2026-07-19T20:10:00Z","type":"repair","status":"ok","note":"Direct Codex streamable-HTTP registration still closed permanently after a brief Idunn worker deployment or SSH-tunnel gap; stateless server sessions did not imply client transport recovery. Codex now attaches to a task-local stdio continuity bridge with no retrieval authority. Each tool call uses a lightweight HTTP client to the resident daemon, and the bridge lowers endpoint failure to a tool error. A single outer MCP client succeeded before a forced endpoint outage, received isError during the outage, and succeeded again after the same endpoint returned."}
 {"ts":"2026-08-16T17:30:00Z","type":"recovery","status":"ok","note":"Big Yggdrasil was retired after an incomplete evacuation and its recycled 94.130.223.131 address is not trusted. Old Yggdrasil retained no VoidBot state, but Raven held a verified June 3 offsite archive (SHA-256 47572df766e63212012eb75762359113a11c8bbf149b497e120fc4fda7191dfc); the June 4 archive is corrupt. Recovery is intentionally bounded to a loopback read-only MCP over exact archives and VoidBot-owned vectors while lost Discord, Idunn, Odin, Epiphany, Persona, moderation, and delivery authority remains stopped."}
 {"ts":"2026-08-16T17:40:00Z","type":"architecture","status":"ok","note":"Read-only MCP recovery exposed that SerializedFileStore snapshots always created writer locks. Archive repositories now accept an explicit read-only mode: snapshots read without filesystem locks and every mutation method fails immediately. A Node 24 build and a nobody-user probe on a non-writable directory proved snapshot success plus mutation rejection; writable runtimes retain interprocess locking."}
+{"ts":"2026-08-16T19:29:57Z","type":"implementation","status":"ok","note":"Added read-only get_exact_source_document MCP tool backed by the canonical sharded source archive; full content is returned for consumer-owned hashing and receipts. Typecheck and in-memory MCP smoke pass."}

state/map.yaml

@@ -63,6 +63,7 @@ data_flow:
     - keep persistent state lean enough that re-entry sharpens the next turn instead of drowning it
 
 active_subgoals:
+  - Publish the read-only `get_exact_source_document` contract through the Yggdrasil recovery allow-list and immutable release, then verify a full Aetheria document over the restored Starfire loopback tunnel. The source archive remains the sole owner; consumers hash and receipt returned content.
   - Treat `notes/agent-swarm-teardown-plan-2026-05-21.md` as the historical teardown authority and the restart checklist. The repo Face swarm is live again, but only because the scheduler, prompt, model-output logs, parent Interpreter, Bifrost routing, and worker side-effect rails now have explicit observable boundaries. Keep watching for prompt/parser/transport Jenga before adding new swarm features.
   - Make the new project-state surfaces the default continuity path for nontrivial VoidBot work.
   - Keep the rebuilt moderation/self-state foundation on canonical CultCache `.cc` typed operations, never an editable JSON working projection.