From de223f0fff3a54211eb922df4d6019ccec5e97cf Mon Sep 17 00:00:00 2001 From: Meta Vens Date: Sat, 2 May 2026 12:49:23 +0100 Subject: [PATCH] Auto-reconcile source repo indexing --- README.md | 25 ++ notes/fresh-workspace-handoff.md | 2 + notes/voidbot-current-system-map.md | 12 +- package.json | 1 + packages/rag/src/file-store.ts | 109 ++++++- packages/rag/src/source-document-archive.ts | 290 +++++++++++++++--- scripts/check-voidbot-operations.ps1 | 92 ++++++ scripts/index-source-repos.ts | 95 +----- .../install-source-index-post-push-hooks.ts | 165 +--------- scripts/reconcile-source-repos.ts | 267 ++++++++++++++++ scripts/source-repo-discovery.ts | 109 +++++++ scripts/source-repo-hooks.ts | 124 ++++++++ scripts/source-repo-indexing.ts | 111 +++++++ scripts/start-voidbot-stack.ps1 | 10 + state/evidence.jsonl | 1 + state/map.yaml | 2 + 16 files changed, 1127 insertions(+), 288 deletions(-) create mode 100644 scripts/reconcile-source-repos.ts create mode 100644 scripts/source-repo-discovery.ts create mode 100644 scripts/source-repo-hooks.ts create mode 100644 scripts/source-repo-indexing.ts diff --git a/README.md b/README.md index 3899051..e79b513 100644 --- a/README.md +++ b/README.md @@ -323,6 +323,12 @@ Force a clean reindex for specific repos: npm run rag:index-sources -- --repo AetheriaLore --force ``` +Reconcile the discovered local repos against the source archive, install or refresh hooks, prune stale repo shards, and index anything new: + +```bash +npm run rag:reconcile-sources +``` + Recommended posture for a shared project root: - `SOURCE_REPO_PATTERNS=*` @@ -356,6 +362,25 @@ Progress for those detached runs lives here: That gives you a real polling path instead of an attached session silently rotting while you wonder whether it hung. +### Automatic repo discovery and repair + +VoidBot now also reconciles the repo set automatically: + +- `npm run stack:start` runs `npm run rag:reconcile-sources` before the bot and worker come back +- `npm run state:check` compares discovered repos under `SOURCE_REPO_ROOT` to the indexed archive and will launch detached repair work if new repos exist +- the reconcile pass refreshes hooks, indexes newly discovered repos, and prunes stale repo shards for repos that no longer exist under the configured root + +The old “install hooks once and hope forever” story is dead. New repos under the configured root should get picked up by normal startup and watchdog operations without manual babysitting. + +### Source archive layout + +The source archive is no longer one giant `source-documents.json` blob trying to die dramatically at scale. + +- `RAG_SOURCE_ARCHIVE_PATH` now stores a small repo-summary manifest +- per-repo source documents live under the sibling shard directory `source-documents.repos/` + +That keeps incremental repo writes bounded to the repo that actually changed instead of rewriting the whole corpus every time one new project appears. + ## Codex Integration This repo includes a project-scoped `.codex/config.toml`, so Codex sessions opened in `E:\Projects\VoidBot` can use Void's MCP tools automatically. diff --git a/notes/fresh-workspace-handoff.md b/notes/fresh-workspace-handoff.md index 747057a..de79209 100644 --- a/notes/fresh-workspace-handoff.md +++ b/notes/fresh-workspace-handoff.md @@ -27,9 +27,11 @@ Git history and smoke artifacts carry routine proof. `state/evidence.jsonl` shou - The worker MCP surface is no longer one swollen server file; stdio bootstrap stays in `apps/worker/src/mcp-server.ts` while context bootstrap, resource registration, tool registration, Discord notify helpers, and shared formatting/schema helpers live under `apps/worker/src/mcp-server-*.ts`. - Interaction-memory logic is no longer one thousand-line slab; event/tone analysis, shared constants, and profile synthesis are split across `packages/core/src/interaction-memory-*.ts`. - State storage is no longer one persistence omnibus; `packages/core/src/state-storage.ts` is now a thin factory over domain-specific Postgres/file store modules plus bootstrap/migration helpers. +- Source archive storage is no longer one giant JSON monolith; `packages/rag/src/source-document-archive.ts` now keeps a small manifest at `.voidbot/rag/source-documents.json` and per-repo shards under `.voidbot/rag/source-documents.repos/`. - Ops health, backup verification, offsite sync, and dashboard surfaces already exist. - The stack has a dedicated logon startup task path now, so a reboot does not have to mean manual necromancy for bot and worker. - The operations watchdog now supports ignored local extension checks, so adjacent-service fire signals can ride the same owner DM alert path without hardcoding private project gossip into the public repo. +- Source repo discovery now reconciles automatically during `npm run stack:start` and through the watchdog path, so new repos under `SOURCE_REPO_ROOT` stop waiting for manual hook folklore before they get indexed. - The new state/notes surfaces are now the continuity spine for future nontrivial work. ## Likely Next Bounded Move diff --git a/notes/voidbot-current-system-map.md b/notes/voidbot-current-system-map.md index 02307ef..2acc971 100644 --- a/notes/voidbot-current-system-map.md +++ b/notes/voidbot-current-system-map.md @@ -65,7 +65,7 @@ This note is the source-grounded description of how the live VoidBot stack is sh - `packages/rag/src/message-archive.ts` - archived Discord message store under `.voidbot/rag/messages.json`. - `packages/rag/src/source-document-archive.ts` - - archived source/lore document store under `.voidbot/rag/source-documents.json`. + - archived source/lore document manifest under `.voidbot/rag/source-documents.json` plus per-repo shards under `.voidbot/rag/source-documents.repos/`. ## Flow 1: Discord Request To Provider @@ -94,17 +94,18 @@ This note is the source-grounded description of how the live VoidBot stack is sh 1. `packages/rag/src/message-archive.ts` and `packages/rag/src/source-document-archive.ts` keep the raw corpora. 2. `packages/rag/src/retrieval-service.ts` translates history/source queries into vector lookups plus metadata filters. 3. `packages/rag/src/qdrant-vector-store.ts` executes the live vector lookups against separate history and source collections. -4. `scripts/index-source-repos.ts` and `scripts/git-post-push-index.mjs` drive detached source/lore reindex work. -5. `apps/worker/src/mcp-server.ts` boots the MCP lane, while `mcp-server-resources.ts` and `mcp-server-tools.ts` expose retrieval to Codex and other sessions through `search_history`, `get_message_context`, `search_sources`, `get_source_context`, and `list_indexed_repos`. +4. `scripts/reconcile-source-repos.ts` discovers local Git repos, refreshes push hooks, prunes stale repo shards, and indexes newly discovered repos. +5. `scripts/index-source-repos.ts` and `scripts/git-post-push-index.mjs` drive explicit or detached per-repo source/lore reindex work. +6. `apps/worker/src/mcp-server.ts` boots the MCP lane, while `mcp-server-resources.ts` and `mcp-server-tools.ts` expose retrieval to Codex and other sessions through `search_history`, `get_message_context`, `search_sources`, `get_source_context`, and `list_indexed_repos`. ## Flow 4: Ops And Recovery - `scripts/start-voidbot-stack.ps1` - - stack bootstrap, health checks, fresh build, stale-process cleanup, bot/worker restart, runtime status emission. + - stack bootstrap, health checks, fresh build, stale-process cleanup, source-repo reconcile, bot/worker restart, runtime status emission. - `scripts/install-stack-startup-task.ps1` - installs the hidden logon task that runs `start-voidbot-stack.ps1` automatically after reboot or sign-in. - `scripts/check-voidbot-operations.ps1` - - watchdog for process liveness, Qdrant, Postgres, Ollama, Discord auth, backup freshness, offsite sync freshness, and optional ignored local extension checks. + - watchdog for process liveness, Qdrant, Postgres, Ollama, source-repo reconcile drift, Discord auth, backup freshness, offsite sync freshness, and optional ignored local extension checks. - `scripts/backup-voidbot-state.ps1` - local backup of Postgres, Qdrant snapshots, and RAG archives. - `scripts/restore-voidbot-state.ps1` @@ -129,6 +130,7 @@ When present, an ignored local script at `.voidbot/private/check-voidbot-operati - source/lore vectors - `.voidbot/`: - archived raw corpora + - source archive manifest plus per-repo source shards - job artifacts and traces - logs and status files - backups and snapshots diff --git a/package.json b/package.json index 6cf31e4..d46bf16 100644 --- a/package.json +++ b/package.json @@ -33,6 +33,7 @@ "rag:backfill": "tsx scripts/backfill-discord-logs.ts", "rag:index-sources": "tsx scripts/index-source-repos.ts", "rag:install-push-hooks": "tsx scripts/install-source-index-post-push-hooks.ts", + "rag:reconcile-sources": "tsx scripts/reconcile-source-repos.ts", "rag:migrate-qdrant": "tsx scripts/migrate-local-vectors-to-qdrant.ts", "rag:migrate-vector-layout": "tsx scripts/migrate-vector-store-layout.ts", "rag:rebuild": "tsx scripts/rebuild-rag-index.ts" diff --git a/packages/rag/src/file-store.ts b/packages/rag/src/file-store.ts index 0ab322b..984eae7 100644 --- a/packages/rag/src/file-store.ts +++ b/packages/rag/src/file-store.ts @@ -1,4 +1,4 @@ -import { mkdir, readFile, writeFile } from "node:fs/promises"; +import { mkdir, open, readFile, rename, rm, stat, writeFile } from "node:fs/promises"; import { dirname } from "node:path"; export class SerializedFileStore { @@ -10,23 +10,33 @@ export class SerializedFileStore { ) {} public async snapshot(): Promise { - await this.writeChain; - const store = await this.readUnlocked(); + const store = await this.serialize(() => this.withInterprocessLock(() => this.readUnlocked())); return structuredClone(store); } public async overwrite(nextValue: T): Promise { await this.serialize(async () => { - await this.writeUnlocked(nextValue); + await this.withInterprocessLock(() => this.writeUnlocked(nextValue)); }); } public async mutate(mutator: (current: T) => R | Promise): Promise { return this.serialize(async () => { - const current = await this.readUnlocked(); - const result = await mutator(current); - await this.writeUnlocked(current); - return result; + return this.withInterprocessLock(async () => { + const current = await this.readUnlocked(); + const result = await mutator(current); + await this.writeUnlocked(current); + return result; + }); + }); + } + + public async normalize(): Promise { + await this.serialize(async () => { + await this.withInterprocessLock(async () => { + const current = await this.readUnlocked(); + await this.writeUnlocked(current); + }); }); } @@ -42,7 +52,7 @@ export class SerializedFileStore { private async readUnlocked(): Promise { try { const raw = await readFile(this.filePath, "utf8"); - return JSON.parse(stripLeadingBom(raw)) as T; + return parseSerializedValue(stripLeadingBom(raw)) as T; } catch (error) { const code = (error as NodeJS.ErrnoException).code; @@ -56,10 +66,89 @@ export class SerializedFileStore { private async writeUnlocked(value: T): Promise { await mkdir(dirname(this.filePath), { recursive: true }); - await writeFile(this.filePath, `${JSON.stringify(value)}\n`, "utf8"); + const tempPath = `${this.filePath}.${process.pid}.${Date.now()}.tmp`; + const serialized = `${JSON.stringify(value)}\n`; + + try { + await writeFile(tempPath, serialized, "utf8"); + await rm(this.filePath, { force: true }); + await rename(tempPath, this.filePath); + } finally { + await rm(tempPath, { force: true }).catch(() => undefined); + } + } + + private async withInterprocessLock(operation: () => Promise): Promise { + const lockPath = `${this.filePath}.lock`; + await mkdir(dirname(lockPath), { recursive: true }); + const deadline = Date.now() + 30_000; + + for (;;) { + try { + const handle = await open(lockPath, "wx"); + + try { + return await operation(); + } finally { + await handle.close(); + await rm(lockPath, { force: true }).catch(() => undefined); + } + } catch (error) { + const code = (error as NodeJS.ErrnoException).code; + + if (code !== "EEXIST") { + throw error; + } + + const lockIsStale = await isStaleLock(lockPath); + + if (lockIsStale) { + await rm(lockPath, { force: true }).catch(() => undefined); + continue; + } + + if (Date.now() >= deadline) { + throw new Error(`Timed out waiting for file lock: ${lockPath}`); + } + + await delay(100); + } + } } } function stripLeadingBom(input: string): string { return input.charCodeAt(0) === 0xfeff ? input.slice(1) : input; } + +function parseSerializedValue(input: string): unknown { + try { + return JSON.parse(input); + } catch (error) { + const lines = input + .split(/\r?\n/) + .map((line) => line.trim()) + .filter((line) => line.length > 0); + + if (lines.length > 1) { + return JSON.parse(lines[0]); + } + + throw error; + } +} + +async function isStaleLock(lockPath: string): Promise { + try { + const lockStat = await stat(lockPath); + return Date.now() - lockStat.mtimeMs > 10 * 60 * 1000; + } catch { + return false; + } +} + +async function delay(milliseconds: number): Promise { + await new Promise((resolve) => { + setTimeout(resolve, milliseconds); + }); +} diff --git a/packages/rag/src/source-document-archive.ts b/packages/rag/src/source-document-archive.ts index 1f7bb9f..a2a2746 100644 --- a/packages/rag/src/source-document-archive.ts +++ b/packages/rag/src/source-document-archive.ts @@ -1,3 +1,6 @@ +import { mkdir, readFile, rename, rm } from "node:fs/promises"; +import { basename, dirname, extname, join } from "node:path"; + import { normalizeText } from "./history-ingester"; import { SerializedFileStore } from "./file-store"; @@ -22,11 +25,22 @@ export interface IndexedSourceRepoSummary { documentCount: number; } -interface SourceDocumentArchiveStore { +interface LegacySourceDocumentArchiveStore { version: 1; documents: ArchivedSourceDocumentRecord[]; } +interface SourceDocumentArchiveManifest { + version: 2; + repos: IndexedSourceRepoSummary[]; +} + +interface SourceDocumentRepoShardStore { + version: 1; + repoName: string; + documents: ArchivedSourceDocumentRecord[]; +} + export interface SourceDocumentSyncResult { created: number; updated: number; @@ -39,24 +53,28 @@ export interface SourceDocumentSyncResult { } export class FileSourceDocumentArchiveRepository { - private readonly store: SerializedFileStore; + private readonly manifestStore: SerializedFileStore; + private readonly shardRoot: string; + private initialization?: Promise; - public constructor(filePath: string) { - this.store = new SerializedFileStore(filePath, () => ({ - version: 1, - documents: [], + public constructor(private readonly filePath: string) { + this.manifestStore = new SerializedFileStore(filePath, () => ({ + version: 2, + repos: [], })); + this.shardRoot = deriveShardRoot(filePath); } public async syncRepoDocuments( repoName: string, documents: ArchivedSourceDocument[], ): Promise { - return this.store.mutate((store) => { + await this.ensureInitialized(); + + const shardStore = this.getRepoShardStore(repoName); + const result = await shardStore.mutate((store) => { const now = new Date().toISOString(); - const previousRepoSourceIds = store.documents - .filter((document) => document.repoName === repoName) - .map((document) => document.id); + const previousRepoSourceIds = store.documents.map((document) => document.id); const nextDocuments = documents.map((document) => toArchivedSourceDocumentRecord(document, now)); const existingById = new Map(store.documents.map((document, index) => [document.id, index])); const nextIds = new Set(nextDocuments.map((document) => document.id)); @@ -93,10 +111,6 @@ export class FileSourceDocumentArchiveRepository { } const retainedDocuments = store.documents.filter((document) => { - if (document.repoName !== repoName) { - return true; - } - if (nextIds.has(document.id)) { return true; } @@ -116,48 +130,252 @@ export class FileSourceDocumentArchiveRepository { previousRepoSourceIds, changedSourceIds, changedDocuments, - activeDocuments: retainedDocuments - .filter((document) => document.repoName === repoName) - .sort((left, right) => left.path.localeCompare(right.path)), + activeDocuments: retainedDocuments.slice().sort(compareDocuments), }; }); + + await this.syncRepoSummary(repoName, result.activeDocuments.length); + + if (result.activeDocuments.length === 0) { + await this.deleteRepoShard(repoName); + } + + return result; } public async get(sourceId: string): Promise { - const store = await this.store.snapshot(); + await this.ensureInitialized(); + const repoName = parseRepoNameFromSourceId(sourceId); + + if (!repoName) { + return undefined; + } + + const store = await this.getRepoShardStore(repoName).snapshot(); return store.documents.find((document) => document.id === sourceId); } public async listAll(): Promise { - const store = await this.store.snapshot(); - return store.documents.slice().sort(compareDocuments); + await this.ensureInitialized(); + const summaries = await this.listRepoSummaries(); + const repoDocuments = await Promise.all( + summaries.map((summary) => this.listByRepo(summary.repoName)), + ); + + return repoDocuments.flat().sort(compareDocuments); } public async listByRepo(repoName: string): Promise { - const documents = await this.listAll(); - return documents.filter((document) => document.repoName === repoName); + await this.ensureInitialized(); + const store = await this.getRepoShardStore(repoName).snapshot(); + return store.documents.slice().sort(compareDocuments); } public async listRepoSummaries(): Promise { - const store = await this.store.snapshot(); - const counts = new Map(); - - for (const document of store.documents) { - counts.set(document.repoName, (counts.get(document.repoName) ?? 0) + 1); - } - - return [...counts.entries()] - .map(([repoName, documentCount]) => ({ - repoName, - documentCount, - })) + await this.ensureInitialized(); + const store = await this.manifestStore.snapshot(); + return store.repos + .slice() .sort((left, right) => left.repoName.localeCompare(right.repoName)); } public async count(): Promise { - const store = await this.store.snapshot(); - return store.documents.length; + const summaries = await this.listRepoSummaries(); + return summaries.reduce((total, summary) => total + summary.documentCount, 0); } + + public async normalizeStore(): Promise { + await this.ensureInitialized(); + + const summaries = await this.listRepoSummaries(); + await this.manifestStore.normalize(); + + for (const summary of summaries) { + await this.getRepoShardStore(summary.repoName).normalize(); + } + } + + public async removeRepo(repoName: string): Promise { + await this.ensureInitialized(); + + const shardStore = this.getRepoShardStore(repoName); + const store = await shardStore.snapshot(); + const deletedSourceIds = store.documents.map((document) => document.id); + + await this.deleteRepoShard(repoName); + await this.syncRepoSummary(repoName, 0); + + return deletedSourceIds; + } + + private async ensureInitialized(): Promise { + if (!this.initialization) { + this.initialization = this.initialize(); + } + + await this.initialization; + } + + private async initialize(): Promise { + await mkdir(dirname(this.filePath), { recursive: true }); + await mkdir(this.shardRoot, { recursive: true }); + + const raw = await readRawFile(this.filePath); + + if (!raw) { + return; + } + + const parsed = parseArchivePayload(raw); + + if (isManifestStore(parsed)) { + return; + } + + if (isLegacyStore(parsed)) { + await this.migrateLegacyStore(parsed); + } + } + + private async migrateLegacyStore(legacyStore: LegacySourceDocumentArchiveStore): Promise { + const documentsByRepo = new Map(); + + for (const document of legacyStore.documents) { + const repoDocuments = documentsByRepo.get(document.repoName) ?? []; + repoDocuments.push(document); + documentsByRepo.set(document.repoName, repoDocuments); + } + + for (const [repoName, documents] of documentsByRepo.entries()) { + const shardStore = this.getRepoShardStore(repoName); + await shardStore.overwrite({ + version: 1, + repoName, + documents: documents.slice().sort(compareDocuments), + }); + } + + const backupPath = `${this.filePath}.legacy-${Date.now()}.bak`; + await rename(this.filePath, backupPath).catch(async () => { + await rm(backupPath, { force: true }).catch(() => undefined); + await rename(this.filePath, backupPath); + }); + + await this.manifestStore.overwrite({ + version: 2, + repos: [...documentsByRepo.entries()] + .map(([repoName, documents]) => ({ + repoName, + documentCount: documents.length, + })) + .sort((left, right) => left.repoName.localeCompare(right.repoName)), + }); + } + + private getRepoShardStore(repoName: string): SerializedFileStore { + return new SerializedFileStore(resolveRepoShardPath(this.shardRoot, repoName), () => ({ + version: 1, + repoName, + documents: [], + })); + } + + private async syncRepoSummary(repoName: string, documentCount: number): Promise { + await this.manifestStore.mutate((manifest) => { + const retainedRepos = manifest.repos.filter((summary) => summary.repoName !== repoName); + + if (documentCount > 0) { + retainedRepos.push({ + repoName, + documentCount, + }); + } + + manifest.repos = retainedRepos.sort((left, right) => left.repoName.localeCompare(right.repoName)); + }); + } + + private async deleteRepoShard(repoName: string): Promise { + await rm(resolveRepoShardPath(this.shardRoot, repoName), { force: true }).catch(() => undefined); + } +} + +function deriveShardRoot(filePath: string): string { + const extension = extname(filePath); + const baseName = extension.length > 0 ? basename(filePath, extension) : basename(filePath); + return join(dirname(filePath), `${baseName}.repos`); +} + +function resolveRepoShardPath(shardRoot: string, repoName: string): string { + const safeRepoName = encodeURIComponent(repoName); + return join(shardRoot, `${safeRepoName}.json`); +} + +async function readRawFile(filePath: string): Promise { + try { + return stripLeadingBom(await readFile(filePath, "utf8")); + } catch (error) { + const code = (error as NodeJS.ErrnoException).code; + + if (code === "ENOENT") { + return undefined; + } + + throw error; + } +} + +function parseArchivePayload(input: string): unknown { + try { + return JSON.parse(input); + } catch (error) { + const lines = input + .split(/\r?\n/) + .map((line) => line.trim()) + .filter((line) => line.length > 0); + + if (lines.length > 1) { + return JSON.parse(lines[0]); + } + + throw error; + } +} + +function isLegacyStore(value: unknown): value is LegacySourceDocumentArchiveStore { + return Boolean( + value && + typeof value === "object" && + "version" in value && + (value as { version?: unknown }).version === 1 && + "documents" in value && + Array.isArray((value as { documents?: unknown }).documents), + ); +} + +function isManifestStore(value: unknown): value is SourceDocumentArchiveManifest { + return Boolean( + value && + typeof value === "object" && + "version" in value && + (value as { version?: unknown }).version === 2 && + "repos" in value && + Array.isArray((value as { repos?: unknown }).repos), + ); +} + +function parseRepoNameFromSourceId(sourceId: string): string | undefined { + const separatorIndex = sourceId.indexOf(":"); + + if (separatorIndex <= 0) { + return undefined; + } + + return sourceId.slice(0, separatorIndex); +} + +function stripLeadingBom(input: string): string { + return input.charCodeAt(0) === 0xfeff ? input.slice(1) : input; } function toArchivedSourceDocumentRecord( diff --git a/scripts/check-voidbot-operations.ps1 b/scripts/check-voidbot-operations.ps1 index b955187..39f9a08 100644 --- a/scripts/check-voidbot-operations.ps1 +++ b/scripts/check-voidbot-operations.ps1 @@ -235,6 +235,46 @@ function Invoke-CommandWithTimeout { } } +function Invoke-SourceRepoReconcile { + param( + [switch] $CheckOnly, + [switch] $Detached + ) + + $tsxCommand = Join-Path $repoRoot "node_modules\.bin\tsx.cmd" + + if (-not (Test-Path -LiteralPath $tsxCommand)) { + throw "Local tsx launcher is missing at $tsxCommand." + } + + $arguments = @("scripts/reconcile-source-repos.ts", "--json") + + if ($CheckOnly) { + $arguments += "--check" + } + + if ($Detached) { + $arguments += "--detached" + } + + $stdout = (& $tsxCommand @arguments 2>&1 | Out-String).Trim() + $exitCode = $LASTEXITCODE + + if ($exitCode -ne 0) { + if ([string]::IsNullOrWhiteSpace($stdout)) { + $stdout = "Source repo reconcile exited with code $exitCode." + } + + throw $stdout + } + + if ([string]::IsNullOrWhiteSpace($stdout)) { + throw "Source repo reconcile returned no JSON output." + } + + return $stdout | ConvertFrom-Json +} + function Ensure-DockerCliOnPath { $dockerCliDir = "C:\Program Files\Docker\Docker\resources\bin" @@ -652,6 +692,7 @@ try { $vectorStoreKind = if ($config.ContainsKey("VECTOR_STORE_KIND")) { $config["VECTOR_STORE_KIND"] } else { "local_json" } $historyCollection = if ($config.ContainsKey("QDRANT_HISTORY_COLLECTION")) { $config["QDRANT_HISTORY_COLLECTION"] } else { "voidbot_discord_history_chunks" } $sourceCollection = if ($config.ContainsKey("QDRANT_SOURCE_COLLECTION")) { $config["QDRANT_SOURCE_COLLECTION"] } else { "voidbot_repository_source_chunks" } + $qdrantReady = $vectorStoreKind -ne "qdrant" Set-WatchdogStep -CurrentStep "qdrant_checks" -Detail "Verifying Qdrant reachability and required collections." if ($vectorStoreKind -eq "qdrant") { @@ -659,6 +700,7 @@ try { try { $collectionsResponse = Get-Json -Url "$($qdrantUrl.TrimEnd('/'))/collections" $collectionNames = @($collectionsResponse.result.collections | ForEach-Object { [string]$_.name }) + $qdrantReady = $true Add-Check -Name "qdrant.reachable" -Status "passed" -Detail "Qdrant is reachable at $qdrantUrl." -Data @{ url = $qdrantUrl collections = $collectionNames @@ -742,6 +784,7 @@ try { $ragOllamaUrl = if ($config.ContainsKey("RAG_OLLAMA_BASE_URL")) { $config["RAG_OLLAMA_BASE_URL"] } else { "http://127.0.0.1:11434" } $ragOllamaModel = if ($config.ContainsKey("RAG_OLLAMA_MODEL")) { $config["RAG_OLLAMA_MODEL"] } else { "qwen3-embedding:0.6b" } + $ragOllamaReady = $false Set-WatchdogStep -CurrentStep "ollama_checks" -Detail "Checking embedding and local LLM Ollama endpoints." try { @@ -749,6 +792,7 @@ try { $ragModels = @($ragTags.models | ForEach-Object { [string]$_.name }) if ($ragModels -contains $ragOllamaModel) { + $ragOllamaReady = $true Add-Check -Name "rag_ollama.model" -Status "passed" -Detail "Embedding Ollama is reachable and has $ragOllamaModel." -Data @{ url = $ragOllamaUrl model = $ragOllamaModel @@ -805,6 +849,54 @@ try { } } + $sourceRepoRoot = if ($config.ContainsKey("SOURCE_REPO_ROOT")) { $config["SOURCE_REPO_ROOT"] } else { "" } + if ([string]::IsNullOrWhiteSpace($sourceRepoRoot)) { + Add-Check -Name "source.repos" -Status "passed" -Detail "Source repo reconciliation check skipped because SOURCE_REPO_ROOT is blank." + } else { + Set-WatchdogStep -CurrentStep "source_repo_checks" -Detail "Reconciling discovered source repos against the indexed source archive." + + try { + $sourceRepoPreview = Invoke-SourceRepoReconcile -CheckOnly + $missingRepos = @($sourceRepoPreview.missingRepos | ForEach-Object { [string]$_ }) + $staleIndexedRepos = @($sourceRepoPreview.staleIndexedRepos | ForEach-Object { [string]$_ }) + + if ($missingRepos.Count -eq 0) { + Add-Check -Name "source.repos" -Status "passed" -Detail "All discovered source repos are represented in the indexed source archive." -Data @{ + discoveredRepoCount = [int]$sourceRepoPreview.discoveredRepoCount + indexedRepoCount = [int]$sourceRepoPreview.indexedRepoCount + selectedRepoCount = [int]$sourceRepoPreview.selectedRepoCount + } + } elseif ($qdrantReady -and $ragOllamaReady) { + $sourceRepoRepair = Invoke-SourceRepoReconcile -Detached + $launchedRepos = @($sourceRepoRepair.launchedRepos | ForEach-Object { [string]$_ }) + Add-Check -Name "source.repos" -Status "warning" -Detail "Detected $($missingRepos.Count) unindexed source repos and launched detached reconciliation." -Data @{ + missingRepos = $missingRepos + launchedRepos = $launchedRepos + discoveredRepoCount = [int]$sourceRepoPreview.discoveredRepoCount + indexedRepoCount = [int]$sourceRepoPreview.indexedRepoCount + } + } else { + Add-Check -Name "source.repos" -Status "failed" -Detail "Detected $($missingRepos.Count) unindexed source repos, but source indexing dependencies are unhealthy." -Data @{ + missingRepos = $missingRepos + qdrantReady = $qdrantReady + ragOllamaReady = $ragOllamaReady + } + } + + if ($staleIndexedRepos.Count -eq 0) { + Add-Check -Name "source.stale_index" -Status "passed" -Detail "Indexed source archive has no repo summaries for repos that are no longer present." + } else { + Add-Check -Name "source.stale_index" -Status "warning" -Detail "Indexed source archive still carries repo summaries for repos that are no longer present." -Data @{ + staleIndexedRepos = $staleIndexedRepos + } + } + } catch { + Add-Check -Name "source.repos" -Status "failed" -Detail "Source repo reconciliation check failed: $($_.Exception.Message)" -Data @{ + sourceRepoRoot = $sourceRepoRoot + } + } + } + if ($config.ContainsKey("DISCORD_BOT_TOKEN") -and -not [string]::IsNullOrWhiteSpace($config["DISCORD_BOT_TOKEN"])) { Set-WatchdogStep -CurrentStep "discord_checks" -Detail "Validating Discord bot credentials." try { diff --git a/scripts/index-source-repos.ts b/scripts/index-source-repos.ts index bb79a6c..613f502 100644 --- a/scripts/index-source-repos.ts +++ b/scripts/index-source-repos.ts @@ -1,17 +1,9 @@ import "dotenv/config"; -import { readdir } from "node:fs/promises"; -import { join } from "node:path"; - import { loadConfig } from "@voidbot/config"; -import { - crawlRepositoryDocuments, - createTextEmbedder, - createVectorStores, - FileSourceDocumentArchiveRepository, - SourceDocumentIngester, - SourceRagPipeline, -} from "@voidbot/rag"; + +import { discoverSourceRepos, selectSourceRepos } from "./source-repo-discovery"; +import { indexSourceRepos } from "./source-repo-indexing"; interface ScriptOptions { repos?: string[]; @@ -31,67 +23,21 @@ async function main(): Promise { } const repoNames = - options.repos && options.repos.length > 0 - ? options.repos - : await resolveRepoNames(config.sourceRepoRoot, config.sourceRepoPatterns); + await discoverSourceRepos(config.sourceRepoRoot, config.sourceRepoPatterns); + const repos = selectSourceRepos(repoNames, options.repos); - if (repoNames.length === 0) { + if (repos.length === 0) { throw new Error("No source repositories matched SOURCE_REPO_PATTERNS."); } - const embedder = createTextEmbedder({ - backend: config.ragEmbeddingBackend, - hashDimensions: config.ragEmbeddingDimensions, - ollamaBaseUrl: config.ragOllamaBaseUrl, - ollamaModel: config.ragOllamaModel, - ollamaTimeoutMs: config.ragOllamaTimeoutMs, - queryInstruction: config.ragQueryInstruction, + const summary = await indexSourceRepos(config, repos, { + forceReindex: options.forceReindex, + logger: (line) => console.log(line), }); - const vectorStore = createVectorStores({ - kind: config.vectorStore.kind, - historyPath: config.vectorStore.path, - sourceRoot: config.sourceVectorStoreRoot, - qdrant: config.qdrant, - historyEmbedder: embedder, - sourceEmbedder: embedder, - }).source; - const archiveRepository = new FileSourceDocumentArchiveRepository(config.ragSourceArchivePath); - const sourceIngester = new SourceDocumentIngester(); - const pipeline = new SourceRagPipeline(archiveRepository, sourceIngester, vectorStore); - let indexedRepositories = 0; - let totalDocuments = 0; - let totalChunks = 0; - - for (const repoName of repoNames) { - const repoPath = join(config.sourceRepoRoot, repoName); - const scan = await crawlRepositoryDocuments(repoPath, repoName, { - includePathPrefixes: config.sourceRepoIncludePrefixes[repoName], - }); - const result = await pipeline.syncRepoDocuments(repoName, scan.documents, { - forceReindex: options.forceReindex, - }); - indexedRepositories += 1; - totalDocuments += scan.documents.length; - totalChunks += result.indexedChunks; - - console.log( - [ - `Repo ${repoName}:`, - `documents=${scan.documents.length}`, - `created=${result.createdDocuments}`, - `updated=${result.updatedDocuments}`, - `unchanged=${result.unchangedDocuments}`, - `deleted=${result.deletedDocuments}`, - `indexedChunks=${result.indexedChunks}`, - `skippedFiles=${scan.skippedFiles.length}`, - ].join(" "), - ); - } - - console.log(`Indexed repositories: ${indexedRepositories}`); - console.log(`Indexed documents: ${totalDocuments}`); - console.log(`Indexed chunks: ${totalChunks}`); + console.log(`Indexed repositories: ${summary.indexedRepositories}`); + console.log(`Indexed documents: ${summary.totalDocuments}`); + console.log(`Indexed chunks: ${summary.totalChunks}`); } function parseArgs(argv: string[]): ScriptOptions { @@ -117,23 +63,6 @@ function parseArgs(argv: string[]): ScriptOptions { return options; } -async function resolveRepoNames(root: string, patterns: string[]): Promise { - const entries = await readdir(root, { withFileTypes: true }); - const repoNames = entries - .filter((entry) => entry.isDirectory()) - .map((entry) => entry.name) - .filter((name) => patterns.length === 0 || patterns.some((pattern) => matchesPattern(name, pattern))) - .sort((left, right) => left.localeCompare(right)); - - return repoNames; -} - -function matchesPattern(value: string, pattern: string): boolean { - const escaped = pattern.replace(/[.+?^${}()|[\]\\]/g, "\\$&").replace(/\*/g, ".*"); - const regex = new RegExp(`^${escaped}$`, "i"); - return regex.test(value); -} - void main().catch((error) => { console.error(error); process.exitCode = 1; diff --git a/scripts/install-source-index-post-push-hooks.ts b/scripts/install-source-index-post-push-hooks.ts index 5b33ec6..359078e 100644 --- a/scripts/install-source-index-post-push-hooks.ts +++ b/scripts/install-source-index-post-push-hooks.ts @@ -1,11 +1,8 @@ import "dotenv/config"; -import { readdir, chmod, mkdir, readFile, stat, writeFile } from "node:fs/promises"; -import { join, resolve } from "node:path"; - import { loadConfig } from "@voidbot/config"; - -const HOOK_MARKER = "# voidbot-source-index-push-hook"; +import { discoverSourceRepos, selectSourceRepos } from "./source-repo-discovery"; +import { installSourceIndexPushHooks } from "./source-repo-hooks"; interface ScriptOptions { repos?: string[]; @@ -23,77 +20,20 @@ async function main(): Promise { } const options = parseArgs(process.argv.slice(2)); - const repoNames = - options.repos && options.repos.length > 0 - ? options.repos - : await resolveRepoNames(config.sourceRepoRoot, config.sourceRepoPatterns); + const discoveredRepos = await discoverSourceRepos(config.sourceRepoRoot, config.sourceRepoPatterns); + const selectedRepos = selectSourceRepos(discoveredRepos, options.repos); + const summary = await installSourceIndexPushHooks(process.cwd(), selectedRepos); - let installed = 0; - let updated = 0; - let backedUp = 0; - let skipped = 0; - - for (const repoName of repoNames) { - const repoPath = join(config.sourceRepoRoot, repoName); - const gitDir = await resolveGitDir(repoPath); - - if (!gitDir) { - console.log(`Skipping ${repoName}: no .git directory found.`); - skipped += 1; - continue; - } - - const hooksDir = join(gitDir, "hooks"); - const hookPath = join(hooksDir, "pre-push"); - const backupHookPath = join(hooksDir, "pre-push.voidbot.prev"); - - await mkdir(hooksDir, { recursive: true }); - - let existingContent: string | undefined; - - try { - existingContent = await readFile(hookPath, "utf8"); - } catch { - existingContent = undefined; - } - - if ( - existingContent && - !existingContent.includes(HOOK_MARKER) && - !(await pathExists(backupHookPath)) - ) { - await writeFile(backupHookPath, existingContent, "utf8"); - await chmod(backupHookPath, 0o755); - backedUp += 1; - } else if ( - existingContent && - !existingContent.includes(HOOK_MARKER) && - (await pathExists(backupHookPath)) - ) { - console.log(`Skipping ${repoName}: existing pre-push hook already has a Void backup; not clobbering it.`); - skipped += 1; - continue; - } - - const hookScript = renderHookScript(process.cwd(), repoName); - await writeFile(hookPath, hookScript, "utf8"); - await chmod(hookPath, 0o755); - - if (existingContent?.includes(HOOK_MARKER)) { - updated += 1; - console.log(`Updated pre-push hook for ${repoName}.`); - } else { - installed += 1; - console.log(`Installed pre-push hook for ${repoName}.`); - } + for (const message of summary.messages) { + console.log(message); } console.log( [ - `installed=${installed}`, - `updated=${updated}`, - `backedUp=${backedUp}`, - `skipped=${skipped}`, + `installed=${summary.installed}`, + `updated=${summary.updated}`, + `backedUp=${summary.backedUp}`, + `skipped=${summary.skipped}`, ].join(" "), ); } @@ -116,89 +56,6 @@ function parseArgs(argv: string[]): ScriptOptions { return options; } -async function resolveRepoNames(root: string, patterns: string[]): Promise { - const entries = await readdir(root, { withFileTypes: true }); - return entries - .filter((entry) => entry.isDirectory()) - .map((entry) => entry.name) - .filter((name) => patterns.length === 0 || patterns.some((pattern) => matchesPattern(name, pattern))) - .sort((left, right) => left.localeCompare(right)); -} - -async function resolveGitDir(repoPath: string): Promise { - const dotGitPath = join(repoPath, ".git"); - - try { - const dotGitStats = await stat(dotGitPath); - - if (dotGitStats.isDirectory()) { - return dotGitPath; - } - - if (dotGitStats.isFile()) { - const pointer = await readFile(dotGitPath, "utf8"); - const match = pointer.match(/^gitdir:\s*(.+)$/im); - - if (match?.[1]) { - return resolve(repoPath, match[1].trim()); - } - } - } catch { - return undefined; - } - - return undefined; -} - -function renderHookScript(voidbotRoot: string, repoName: string): string { - const normalizedRoot = voidbotRoot.replace(/\\/g, "/"); - const launcherPath = `${normalizedRoot}/scripts/git-post-push-index.mjs`; - const nodePath = process.execPath.replace(/\\/g, "/"); - const quotedNodePath = escapeForDoubleQuotedShell(nodePath); - const quotedLauncherPath = escapeForDoubleQuotedShell(launcherPath); - const quotedRoot = escapeForDoubleQuotedShell(normalizedRoot); - const quotedRepoName = escapeForDoubleQuotedShell(repoName); - - return [ - "#!/bin/sh", - HOOK_MARKER, - "HOOK_DIR=\"$(CDPATH= cd -- \"$(dirname -- \"$0\")\" && pwd)\"", - "PREV_HOOK=\"$HOOK_DIR/pre-push.voidbot.prev\"", - "", - "if [ -x \"$PREV_HOOK\" ]; then", - " \"$PREV_HOOK\" \"$@\"", - " PREV_STATUS=$?", - " if [ \"$PREV_STATUS\" -ne 0 ]; then", - " exit \"$PREV_STATUS\"", - " fi", - "fi", - "", - `\"${quotedNodePath}\" \"${quotedLauncherPath}\" --voidbot-root \"${quotedRoot}\" --repo-name \"${quotedRepoName}\" >/dev/null 2>&1 || true`, - "", - "exit 0", - "", - ].join("\n"); -} - -function escapeForDoubleQuotedShell(value: string): string { - return value.replace(/(["`$\\])/g, "\\$1"); -} - -function matchesPattern(value: string, pattern: string): boolean { - const escaped = pattern.replace(/[.+?^${}()|[\]\\]/g, "\\$&").replace(/\*/g, ".*"); - const regex = new RegExp(`^${escaped}$`, "i"); - return regex.test(value); -} - -async function pathExists(path: string): Promise { - try { - await stat(path); - return true; - } catch { - return false; - } -} - void main().catch((error) => { console.error(error); process.exitCode = 1; diff --git a/scripts/reconcile-source-repos.ts b/scripts/reconcile-source-repos.ts new file mode 100644 index 0000000..d2d9ee8 --- /dev/null +++ b/scripts/reconcile-source-repos.ts @@ -0,0 +1,267 @@ +import "dotenv/config"; + +import { spawn } from "node:child_process"; +import { join } from "node:path"; + +import { loadConfig } from "@voidbot/config"; +import { FileSourceDocumentArchiveRepository } from "@voidbot/rag"; + +import { discoverSourceRepos, selectSourceRepos, type SourceRepoMatch } from "./source-repo-discovery"; +import { installSourceIndexPushHooks } from "./source-repo-hooks"; +import { createSourceVectorStore, indexSourceRepos } from "./source-repo-indexing"; + +interface ScriptOptions { + repos?: string[]; + forceReindex?: boolean; + detached?: boolean; + json?: boolean; + checkOnly?: boolean; +} + +interface ReconcileSummary { + discoveredRepoCount: number; + indexedRepoCount: number; + selectedRepoCount: number; + missingRepos: string[]; + staleIndexedRepos: string[]; + hookInstall: { + installed: number; + updated: number; + backedUp: number; + skipped: number; + }; + prunedRepos: string[]; + normalizedArchive: boolean; + detached: boolean; + forceReindex: boolean; + checkOnly: boolean; + indexedRepos: string[]; + launchedRepos: string[]; +} + +async function main(): Promise { + if (!process.env.DISCORD_OWNER_ID) { + process.env.DISCORD_OWNER_ID = "__source_reconcile__"; + } + + const options = parseArgs(process.argv.slice(2)); + const config = loadConfig(); + + if (!config.sourceRepoRoot) { + throw new Error("SOURCE_REPO_ROOT is not configured."); + } + + const discoveredRepos = await discoverSourceRepos(config.sourceRepoRoot, config.sourceRepoPatterns); + const selectedRepos = selectSourceRepos(discoveredRepos, options.repos); + + if (selectedRepos.length === 0) { + throw new Error("No source repositories matched SOURCE_REPO_PATTERNS."); + } + + const archiveRepository = new FileSourceDocumentArchiveRepository(config.ragSourceArchivePath); + + if (!options.checkOnly) { + await archiveRepository.normalizeStore(); + } + + const indexedRepoSummaries = await archiveRepository.listRepoSummaries(); + const indexedRepoNames = new Set(indexedRepoSummaries.map((summary) => summary.repoName)); + const missingRepos = selectedRepos + .filter((repo) => options.forceReindex || !indexedRepoNames.has(repo.repoName)) + .map((repo) => repo.repoName); + const staleIndexedRepos = indexedRepoSummaries + .map((summary) => summary.repoName) + .filter((repoName) => !discoveredRepos.some((repo) => repo.repoName === repoName)) + .sort((left, right) => left.localeCompare(right)); + + let hookInstall = { + installed: 0, + updated: 0, + backedUp: 0, + skipped: 0, + }; + + if (!options.checkOnly) { + const hookSummary = await installSourceIndexPushHooks(process.cwd(), selectedRepos); + hookInstall = { + installed: hookSummary.installed, + updated: hookSummary.updated, + backedUp: hookSummary.backedUp, + skipped: hookSummary.skipped, + }; + + if (!options.json) { + for (const message of hookSummary.messages) { + console.log(message); + } + } + } + + const reposToIndex = selectedRepos.filter((repo) => missingRepos.includes(repo.repoName)); + const indexedRepos: string[] = []; + const launchedRepos: string[] = []; + const prunedRepos: string[] = []; + + if (!options.checkOnly && reposToIndex.length > 0) { + if (options.detached) { + for (const repo of reposToIndex) { + await launchDetachedRepoIndex(process.cwd(), repo.repoName); + launchedRepos.push(repo.repoName); + } + } else { + const runSummary = await indexSourceRepos(config, reposToIndex, { + forceReindex: options.forceReindex, + logger: options.json ? undefined : (line) => console.log(line), + }); + indexedRepos.push(...runSummary.results.map((result) => result.repoName)); + + if (!options.json) { + console.log(`Indexed repositories: ${runSummary.indexedRepositories}`); + console.log(`Indexed documents: ${runSummary.totalDocuments}`); + console.log(`Indexed chunks: ${runSummary.totalChunks}`); + } + } + } + + if (!options.checkOnly && staleIndexedRepos.length > 0) { + const sourceVectorStore = createSourceVectorStore(config); + + for (const repoName of staleIndexedRepos) { + await sourceVectorStore.deleteByFilters({ + corpusKind: "repository_source", + repoName, + }); + await archiveRepository.removeRepo(repoName); + prunedRepos.push(repoName); + } + } + + const summary: ReconcileSummary = { + discoveredRepoCount: discoveredRepos.length, + indexedRepoCount: indexedRepoSummaries.length, + selectedRepoCount: selectedRepos.length, + missingRepos, + staleIndexedRepos, + hookInstall, + prunedRepos, + normalizedArchive: !options.checkOnly, + detached: options.detached, + forceReindex: options.forceReindex, + checkOnly: options.checkOnly, + indexedRepos, + launchedRepos, + }; + + if (options.json) { + process.stdout.write(`${JSON.stringify(summary)}\n`); + return; + } + + console.log( + [ + `discoveredRepos=${summary.discoveredRepoCount}`, + `indexedRepos=${summary.indexedRepoCount}`, + `selectedRepos=${summary.selectedRepoCount}`, + `missingRepos=${summary.missingRepos.length}`, + `staleIndexedRepos=${summary.staleIndexedRepos.length}`, + `installedHooks=${summary.hookInstall.installed}`, + `updatedHooks=${summary.hookInstall.updated}`, + `backedUpHooks=${summary.hookInstall.backedUp}`, + `skippedHooks=${summary.hookInstall.skipped}`, + `prunedStaleRepos=${summary.prunedRepos.length}`, + `indexedNow=${summary.indexedRepos.length}`, + `launchedDetached=${summary.launchedRepos.length}`, + ].join(" "), + ); +} + +function parseArgs(argv: string[]): ScriptOptions { + const options: ScriptOptions = { + forceReindex: false, + detached: false, + json: false, + checkOnly: false, + }; + + for (let index = 0; index < argv.length; index += 1) { + const argument = argv[index]; + + if ((argument === "--repo" || argument === "--repos") && argv[index + 1]) { + options.repos = argv[index + 1] + .split(",") + .map((entry) => entry.trim()) + .filter((entry) => entry.length > 0); + index += 1; + continue; + } + + if (argument === "--force") { + options.forceReindex = true; + continue; + } + + if (argument === "--detached") { + options.detached = true; + continue; + } + + if (argument === "--json") { + options.json = true; + continue; + } + + if (argument === "--check") { + options.checkOnly = true; + } + } + + return options; +} + +async function launchDetachedRepoIndex(voidbotRoot: string, repoName: string): Promise { + await new Promise((resolvePromise, rejectPromise) => { + const child = spawn( + process.execPath, + [ + join(voidbotRoot, "scripts", "git-post-push-index.mjs"), + "--voidbot-root", + voidbotRoot, + "--repo-name", + repoName, + ], + { + cwd: voidbotRoot, + env: process.env, + stdio: ["ignore", "ignore", "pipe"], + windowsHide: true, + }, + ); + + let stderr = ""; + + child.stderr.on("data", (chunk) => { + stderr += String(chunk); + }); + + child.on("error", rejectPromise); + child.on("close", (code) => { + if ((code ?? 1) !== 0) { + rejectPromise( + new Error( + stderr.trim().length > 0 + ? stderr.trim() + : `Detached source-index launcher failed for ${repoName}.`, + ), + ); + return; + } + + resolvePromise(); + }); + }); +} + +void main().catch((error) => { + console.error(error); + process.exitCode = 1; +}); diff --git a/scripts/source-repo-discovery.ts b/scripts/source-repo-discovery.ts new file mode 100644 index 0000000..c1a2180 --- /dev/null +++ b/scripts/source-repo-discovery.ts @@ -0,0 +1,109 @@ +import { readdir, readFile, stat } from "node:fs/promises"; +import { join, resolve } from "node:path"; + +export interface SourceRepoMatch { + repoName: string; + repoPath: string; + gitDir: string; +} + +export async function discoverSourceRepos( + root: string, + patterns: string[], +): Promise { + const entries = await readdir(root, { withFileTypes: true }); + const repoMatches: SourceRepoMatch[] = []; + + for (const entry of entries) { + if (!entry.isDirectory()) { + continue; + } + + if (patterns.length > 0 && !patterns.some((pattern) => matchesPattern(entry.name, pattern))) { + continue; + } + + const repoPath = join(root, entry.name); + const gitDir = await resolveGitDir(repoPath); + + if (!gitDir) { + continue; + } + + repoMatches.push({ + repoName: entry.name, + repoPath, + gitDir, + }); + } + + repoMatches.sort((left, right) => left.repoName.localeCompare(right.repoName)); + return repoMatches; +} + +export function selectSourceRepos( + availableRepos: SourceRepoMatch[], + requestedRepoNames?: string[], +): SourceRepoMatch[] { + if (!requestedRepoNames || requestedRepoNames.length === 0) { + return availableRepos; + } + + const lookup = new Map( + availableRepos.map((repo) => [repo.repoName.toLowerCase(), repo] as const), + ); + const selectedRepos: SourceRepoMatch[] = []; + const missingRepoNames: string[] = []; + + for (const requestedRepoName of requestedRepoNames) { + const repo = lookup.get(requestedRepoName.toLowerCase()); + + if (!repo) { + missingRepoNames.push(requestedRepoName); + continue; + } + + if (!selectedRepos.some((entry) => entry.repoName === repo.repoName)) { + selectedRepos.push(repo); + } + } + + if (missingRepoNames.length > 0) { + throw new Error( + `Requested source repos were not found under SOURCE_REPO_ROOT: ${missingRepoNames.join(", ")}`, + ); + } + + return selectedRepos; +} + +export async function resolveGitDir(repoPath: string): Promise { + const dotGitPath = join(repoPath, ".git"); + + try { + const dotGitStats = await stat(dotGitPath); + + if (dotGitStats.isDirectory()) { + return dotGitPath; + } + + if (dotGitStats.isFile()) { + const pointer = await readFile(dotGitPath, "utf8"); + const match = pointer.match(/^gitdir:\s*(.+)$/im); + + if (match?.[1]) { + return resolve(repoPath, match[1].trim()); + } + } + } catch { + return undefined; + } + + return undefined; +} + +function matchesPattern(value: string, pattern: string): boolean { + const escaped = pattern.replace(/[.+?^${}()|[\]\\]/g, "\\$&").replace(/\*/g, ".*"); + const regex = new RegExp(`^${escaped}$`, "i"); + return regex.test(value); +} diff --git a/scripts/source-repo-hooks.ts b/scripts/source-repo-hooks.ts new file mode 100644 index 0000000..42a54ef --- /dev/null +++ b/scripts/source-repo-hooks.ts @@ -0,0 +1,124 @@ +import { chmod, mkdir, readFile, stat, writeFile } from "node:fs/promises"; +import { join } from "node:path"; + +import { type SourceRepoMatch } from "./source-repo-discovery"; + +export const HOOK_MARKER = "# voidbot-source-index-push-hook"; + +export interface SourceHookInstallSummary { + installed: number; + updated: number; + backedUp: number; + skipped: number; + messages: string[]; +} + +export async function installSourceIndexPushHooks( + voidbotRoot: string, + repos: SourceRepoMatch[], +): Promise { + let installed = 0; + let updated = 0; + let backedUp = 0; + let skipped = 0; + const messages: string[] = []; + + for (const repo of repos) { + const hooksDir = join(repo.gitDir, "hooks"); + const hookPath = join(hooksDir, "pre-push"); + const backupHookPath = join(hooksDir, "pre-push.voidbot.prev"); + + await mkdir(hooksDir, { recursive: true }); + + let existingContent: string | undefined; + + try { + existingContent = await readFile(hookPath, "utf8"); + } catch { + existingContent = undefined; + } + + if ( + existingContent && + !existingContent.includes(HOOK_MARKER) && + !(await pathExists(backupHookPath)) + ) { + await writeFile(backupHookPath, existingContent, "utf8"); + await chmod(backupHookPath, 0o755); + backedUp += 1; + } else if ( + existingContent && + !existingContent.includes(HOOK_MARKER) && + (await pathExists(backupHookPath)) + ) { + skipped += 1; + messages.push( + `Skipping ${repo.repoName}: existing pre-push hook already has a Void backup; not clobbering it.`, + ); + continue; + } + + const hookScript = renderHookScript(voidbotRoot, repo.repoName); + await writeFile(hookPath, hookScript, "utf8"); + await chmod(hookPath, 0o755); + + if (existingContent?.includes(HOOK_MARKER)) { + updated += 1; + messages.push(`Updated pre-push hook for ${repo.repoName}.`); + } else { + installed += 1; + messages.push(`Installed pre-push hook for ${repo.repoName}.`); + } + } + + return { + installed, + updated, + backedUp, + skipped, + messages, + }; +} + +function renderHookScript(voidbotRoot: string, repoName: string): string { + const normalizedRoot = voidbotRoot.replace(/\\/g, "/"); + const launcherPath = `${normalizedRoot}/scripts/git-post-push-index.mjs`; + const nodePath = process.execPath.replace(/\\/g, "/"); + const quotedNodePath = escapeForDoubleQuotedShell(nodePath); + const quotedLauncherPath = escapeForDoubleQuotedShell(launcherPath); + const quotedRoot = escapeForDoubleQuotedShell(normalizedRoot); + const quotedRepoName = escapeForDoubleQuotedShell(repoName); + + return [ + "#!/bin/sh", + HOOK_MARKER, + "HOOK_DIR=\"$(CDPATH= cd -- \"$(dirname -- \"$0\")\" && pwd)\"", + "PREV_HOOK=\"$HOOK_DIR/pre-push.voidbot.prev\"", + "", + "if [ -x \"$PREV_HOOK\" ]; then", + " \"$PREV_HOOK\" \"$@\"", + " PREV_STATUS=$?", + " if [ \"$PREV_STATUS\" -ne 0 ]; then", + " exit \"$PREV_STATUS\"", + " fi", + "fi", + "", + `\"${quotedNodePath}\" \"${quotedLauncherPath}\" --voidbot-root \"${quotedRoot}\" --repo-name \"${quotedRepoName}\" >/dev/null 2>&1 || true`, + "", + "exit 0", + "", + ].join("\n"); +} + +function escapeForDoubleQuotedShell(value: string): string { + return value.replace(/(["`$\\])/g, "\\$1"); +} + +async function pathExists(path: string): Promise { + try { + await stat(path); + return true; + } catch { + return false; + } +} diff --git a/scripts/source-repo-indexing.ts b/scripts/source-repo-indexing.ts new file mode 100644 index 0000000..1eef333 --- /dev/null +++ b/scripts/source-repo-indexing.ts @@ -0,0 +1,111 @@ +import { type AppConfig } from "@voidbot/config"; +import { + crawlRepositoryDocuments, + createTextEmbedder, + createVectorStores, + FileSourceDocumentArchiveRepository, + SourceDocumentIngester, + SourceRagPipeline, +} from "@voidbot/rag"; + +import { type SourceRepoMatch } from "./source-repo-discovery"; + +export interface IndexedRepoResult { + repoName: string; + documents: number; + createdDocuments: number; + updatedDocuments: number; + unchangedDocuments: number; + deletedDocuments: number; + indexedChunks: number; + skippedFiles: number; +} + +export interface SourceIndexRunSummary { + indexedRepositories: number; + totalDocuments: number; + totalChunks: number; + results: IndexedRepoResult[]; +} + +export function createSourceVectorStore(config: AppConfig) { + const embedder = createTextEmbedder({ + backend: config.ragEmbeddingBackend, + hashDimensions: config.ragEmbeddingDimensions, + ollamaBaseUrl: config.ragOllamaBaseUrl, + ollamaModel: config.ragOllamaModel, + ollamaTimeoutMs: config.ragOllamaTimeoutMs, + queryInstruction: config.ragQueryInstruction, + }); + + return createVectorStores({ + kind: config.vectorStore.kind, + historyPath: config.vectorStore.path, + sourceRoot: config.sourceVectorStoreRoot, + qdrant: config.qdrant, + historyEmbedder: embedder, + sourceEmbedder: embedder, + }).source; +} + +export async function indexSourceRepos( + config: AppConfig, + repos: SourceRepoMatch[], + options: { + forceReindex?: boolean; + logger?: (line: string) => void; + } = {}, +): Promise { + const vectorStore = createSourceVectorStore(config); + const archiveRepository = new FileSourceDocumentArchiveRepository(config.ragSourceArchivePath); + const sourceIngester = new SourceDocumentIngester(); + const pipeline = new SourceRagPipeline(archiveRepository, sourceIngester, vectorStore); + const results: IndexedRepoResult[] = []; + let indexedRepositories = 0; + let totalDocuments = 0; + let totalChunks = 0; + + for (const repo of repos) { + const scan = await crawlRepositoryDocuments(repo.repoPath, repo.repoName, { + includePathPrefixes: config.sourceRepoIncludePrefixes[repo.repoName], + }); + const result = await pipeline.syncRepoDocuments(repo.repoName, scan.documents, { + forceReindex: options.forceReindex, + }); + const repoResult: IndexedRepoResult = { + repoName: repo.repoName, + documents: scan.documents.length, + createdDocuments: result.createdDocuments, + updatedDocuments: result.updatedDocuments, + unchangedDocuments: result.unchangedDocuments, + deletedDocuments: result.deletedDocuments, + indexedChunks: result.indexedChunks, + skippedFiles: scan.skippedFiles.length, + }; + + results.push(repoResult); + indexedRepositories += 1; + totalDocuments += scan.documents.length; + totalChunks += result.indexedChunks; + + options.logger?.( + [ + `Repo ${repo.repoName}:`, + `documents=${repoResult.documents}`, + `created=${repoResult.createdDocuments}`, + `updated=${repoResult.updatedDocuments}`, + `unchanged=${repoResult.unchangedDocuments}`, + `deleted=${repoResult.deletedDocuments}`, + `indexedChunks=${repoResult.indexedChunks}`, + `skippedFiles=${repoResult.skippedFiles}`, + ].join(" "), + ); + } + + return { + indexedRepositories, + totalDocuments, + totalChunks, + results, + }; +} diff --git a/scripts/start-voidbot-stack.ps1 b/scripts/start-voidbot-stack.ps1 index 5614f41..fcd3068 100644 --- a/scripts/start-voidbot-stack.ps1 +++ b/scripts/start-voidbot-stack.ps1 @@ -532,6 +532,16 @@ if ($LASTEXITCODE -ne 0) { Stop-StaleRuntimeProcesses -RepoRoot $repoRoot +$status.stage = "source_reconcile" +Write-StatusFile -Path $statusPath -Status $status + +Write-Host "Reconciling source repos..." +& $npmCommand.Source run rag:reconcile-sources + +if ($LASTEXITCODE -ne 0) { + throw "npm run rag:reconcile-sources failed." +} + $botOut = Join-Path $logDir "bot.log" $botErr = Join-Path $logDir "bot.err.log" $workerOut = Join-Path $logDir "worker.log" diff --git a/state/evidence.jsonl b/state/evidence.jsonl index 8b4708b..ce62e55 100644 --- a/state/evidence.jsonl +++ b/state/evidence.jsonl @@ -15,3 +15,4 @@ {"ts":"2026-05-01T16:47:54Z","type":"research","status":"ok","note":"Expanded the personality roadmap and classifier spec to treat performed selves, contingent-worth defenses, and impression management as first-class structure. Dialogue is not a clean window into the person; concealment and shell-maintenance are themselves part of the signal the system should model."} {"ts":"2026-05-01T16:56:41Z","type":"research","status":"ok","note":"Documented the state-distribution and prompt-projection architecture for Aetheria agents: core variables should live as culture-shaped latent state with history transforms and per-agent perceived overlays, while prompts are temporary projections of perceived state. Volatility, attachment-seeking, and distance-seeking are first-class dimensions in the rebuild, not later garnish."} {"ts":"2026-05-01T17:24:56Z","type":"architecture","status":"ok","note":"Moved deeper personality, culture, prompt-projection, and Aetheria persistent-world design notes into the sibling Ghostlight repo so VoidBot remains a consumer instead of the long-term home for that research."} +{"ts":"2026-05-02T11:48:55Z","type":"implementation","status":"ok","note":"Source repo indexing now auto-reconciles during startup and watchdog runs, and the source archive is sharded by repo behind a manifest because the old monolithic source-documents.json hit real size limits and broke incremental writes."} diff --git a/state/map.yaml b/state/map.yaml index dde5f2b..bb39724 100644 --- a/state/map.yaml +++ b/state/map.yaml @@ -83,6 +83,8 @@ landed_baseline: - The worker MCP surface is now split into a thin stdio entrypoint plus dedicated bootstrap, resource, tool, Discord-notify, and shared-helper modules instead of one swollen server file. - Interaction-memory logic is now split into event/tone analysis and profile-synthesis modules instead of one slab trying to be both the ear and the therapist. - State storage is now split into a thin factory plus per-domain Postgres/file store modules and bootstrap/migration helpers instead of one persistence omnibus. + - Source archive storage is now sharded by repo behind a manifest plus per-repo shard directory instead of one giant `source-documents.json` blob. + - Source repo indexing now reconciles automatically during stack startup and watchdog runs, refreshing hooks, indexing newly discovered repos, and pruning stale repo shards when repos disappear from `SOURCE_REPO_ROOT`. rejected_paths: - name: transcript_only_continuity