Auto-reconcile source repo indexing
This commit is contained in:
Meta Vens
parent
32e4a1f184
commit
de223f0fff
16 changed files with 1125 additions and 286 deletions
25
README.md
25
README.md
|
|
@ -323,6 +323,12 @@ Force a clean reindex for specific repos:
|
|||
npm run rag:index-sources -- --repo AetheriaLore --force
|
||||
```
|
||||
|
||||
Reconcile the discovered local repos against the source archive, install or refresh hooks, prune stale repo shards, and index anything new:
|
||||
|
||||
```bash
|
||||
npm run rag:reconcile-sources
|
||||
```
|
||||
|
||||
Recommended posture for a shared project root:
|
||||
|
||||
- `SOURCE_REPO_PATTERNS=*`
|
||||
|
|
@ -356,6 +362,25 @@ Progress for those detached runs lives here:
|
|||
|
||||
That gives you a real polling path instead of an attached session silently rotting while you wonder whether it hung.
|
||||
|
||||
### Automatic repo discovery and repair
|
||||
|
||||
VoidBot now also reconciles the repo set automatically:
|
||||
|
||||
- `npm run stack:start` runs `npm run rag:reconcile-sources` before the bot and worker come back
|
||||
- `npm run state:check` compares discovered repos under `SOURCE_REPO_ROOT` to the indexed archive and will launch detached repair work if new repos exist
|
||||
- the reconcile pass refreshes hooks, indexes newly discovered repos, and prunes stale repo shards for repos that no longer exist under the configured root
|
||||
|
||||
The old “install hooks once and hope forever” story is dead. New repos under the configured root should get picked up by normal startup and watchdog operations without manual babysitting.
|
||||
|
||||
### Source archive layout
|
||||
|
||||
The source archive is no longer one giant `source-documents.json` blob trying to die dramatically at scale.
|
||||
|
||||
- `RAG_SOURCE_ARCHIVE_PATH` now stores a small repo-summary manifest
|
||||
- per-repo source documents live under the sibling shard directory `source-documents.repos/`
|
||||
|
||||
That keeps incremental repo writes bounded to the repo that actually changed instead of rewriting the whole corpus every time one new project appears.
|
||||
|
||||
## Codex Integration
|
||||
|
||||
This repo includes a project-scoped `.codex/config.toml`, so Codex sessions opened in `E:\Projects\VoidBot` can use Void's MCP tools automatically.
|
||||
|
|
|
|||
|
|
@ -27,9 +27,11 @@ Git history and smoke artifacts carry routine proof. `state/evidence.jsonl` shou
|
|||
- The worker MCP surface is no longer one swollen server file; stdio bootstrap stays in `apps/worker/src/mcp-server.ts` while context bootstrap, resource registration, tool registration, Discord notify helpers, and shared formatting/schema helpers live under `apps/worker/src/mcp-server-*.ts`.
|
||||
- Interaction-memory logic is no longer one thousand-line slab; event/tone analysis, shared constants, and profile synthesis are split across `packages/core/src/interaction-memory-*.ts`.
|
||||
- State storage is no longer one persistence omnibus; `packages/core/src/state-storage.ts` is now a thin factory over domain-specific Postgres/file store modules plus bootstrap/migration helpers.
|
||||
- Source archive storage is no longer one giant JSON monolith; `packages/rag/src/source-document-archive.ts` now keeps a small manifest at `.voidbot/rag/source-documents.json` and per-repo shards under `.voidbot/rag/source-documents.repos/`.
|
||||
- Ops health, backup verification, offsite sync, and dashboard surfaces already exist.
|
||||
- The stack has a dedicated logon startup task path now, so a reboot does not have to mean manual necromancy for bot and worker.
|
||||
- The operations watchdog now supports ignored local extension checks, so adjacent-service fire signals can ride the same owner DM alert path without hardcoding private project gossip into the public repo.
|
||||
- Source repo discovery now reconciles automatically during `npm run stack:start` and through the watchdog path, so new repos under `SOURCE_REPO_ROOT` stop waiting for manual hook folklore before they get indexed.
|
||||
- The new state/notes surfaces are now the continuity spine for future nontrivial work.
|
||||
|
||||
## Likely Next Bounded Move
|
||||
|
|
|
|||
|
|
@ -65,7 +65,7 @@ This note is the source-grounded description of how the live VoidBot stack is sh
|
|||
- `packages/rag/src/message-archive.ts`
|
||||
- archived Discord message store under `.voidbot/rag/messages.json`.
|
||||
- `packages/rag/src/source-document-archive.ts`
|
||||
- archived source/lore document store under `.voidbot/rag/source-documents.json`.
|
||||
- archived source/lore document manifest under `.voidbot/rag/source-documents.json` plus per-repo shards under `.voidbot/rag/source-documents.repos/`.
|
||||
|
||||
## Flow 1: Discord Request To Provider
|
||||
|
||||
|
|
@ -94,17 +94,18 @@ This note is the source-grounded description of how the live VoidBot stack is sh
|
|||
1. `packages/rag/src/message-archive.ts` and `packages/rag/src/source-document-archive.ts` keep the raw corpora.
|
||||
2. `packages/rag/src/retrieval-service.ts` translates history/source queries into vector lookups plus metadata filters.
|
||||
3. `packages/rag/src/qdrant-vector-store.ts` executes the live vector lookups against separate history and source collections.
|
||||
4. `scripts/index-source-repos.ts` and `scripts/git-post-push-index.mjs` drive detached source/lore reindex work.
|
||||
5. `apps/worker/src/mcp-server.ts` boots the MCP lane, while `mcp-server-resources.ts` and `mcp-server-tools.ts` expose retrieval to Codex and other sessions through `search_history`, `get_message_context`, `search_sources`, `get_source_context`, and `list_indexed_repos`.
|
||||
4. `scripts/reconcile-source-repos.ts` discovers local Git repos, refreshes push hooks, prunes stale repo shards, and indexes newly discovered repos.
|
||||
5. `scripts/index-source-repos.ts` and `scripts/git-post-push-index.mjs` drive explicit or detached per-repo source/lore reindex work.
|
||||
6. `apps/worker/src/mcp-server.ts` boots the MCP lane, while `mcp-server-resources.ts` and `mcp-server-tools.ts` expose retrieval to Codex and other sessions through `search_history`, `get_message_context`, `search_sources`, `get_source_context`, and `list_indexed_repos`.
|
||||
|
||||
## Flow 4: Ops And Recovery
|
||||
|
||||
- `scripts/start-voidbot-stack.ps1`
|
||||
- stack bootstrap, health checks, fresh build, stale-process cleanup, bot/worker restart, runtime status emission.
|
||||
- stack bootstrap, health checks, fresh build, stale-process cleanup, source-repo reconcile, bot/worker restart, runtime status emission.
|
||||
- `scripts/install-stack-startup-task.ps1`
|
||||
- installs the hidden logon task that runs `start-voidbot-stack.ps1` automatically after reboot or sign-in.
|
||||
- `scripts/check-voidbot-operations.ps1`
|
||||
- watchdog for process liveness, Qdrant, Postgres, Ollama, Discord auth, backup freshness, offsite sync freshness, and optional ignored local extension checks.
|
||||
- watchdog for process liveness, Qdrant, Postgres, Ollama, source-repo reconcile drift, Discord auth, backup freshness, offsite sync freshness, and optional ignored local extension checks.
|
||||
- `scripts/backup-voidbot-state.ps1`
|
||||
- local backup of Postgres, Qdrant snapshots, and RAG archives.
|
||||
- `scripts/restore-voidbot-state.ps1`
|
||||
|
|
@ -129,6 +130,7 @@ When present, an ignored local script at `.voidbot/private/check-voidbot-operati
|
|||
- source/lore vectors
|
||||
- `.voidbot/`:
|
||||
- archived raw corpora
|
||||
- source archive manifest plus per-repo source shards
|
||||
- job artifacts and traces
|
||||
- logs and status files
|
||||
- backups and snapshots
|
||||
|
|
|
|||
|
|
@ -33,6 +33,7 @@
|
|||
"rag:backfill": "tsx scripts/backfill-discord-logs.ts",
|
||||
"rag:index-sources": "tsx scripts/index-source-repos.ts",
|
||||
"rag:install-push-hooks": "tsx scripts/install-source-index-post-push-hooks.ts",
|
||||
"rag:reconcile-sources": "tsx scripts/reconcile-source-repos.ts",
|
||||
"rag:migrate-qdrant": "tsx scripts/migrate-local-vectors-to-qdrant.ts",
|
||||
"rag:migrate-vector-layout": "tsx scripts/migrate-vector-store-layout.ts",
|
||||
"rag:rebuild": "tsx scripts/rebuild-rag-index.ts"
|
||||
|
|
|
|||
|
|
@ -1,4 +1,4 @@
|
|||
import { mkdir, readFile, writeFile } from "node:fs/promises";
|
||||
import { mkdir, open, readFile, rename, rm, stat, writeFile } from "node:fs/promises";
|
||||
import { dirname } from "node:path";
|
||||
|
||||
export class SerializedFileStore<T> {
|
||||
|
|
@ -10,23 +10,33 @@ export class SerializedFileStore<T> {
|
|||
) {}
|
||||
|
||||
public async snapshot(): Promise<T> {
|
||||
await this.writeChain;
|
||||
const store = await this.readUnlocked();
|
||||
const store = await this.serialize(() => this.withInterprocessLock(() => this.readUnlocked()));
|
||||
return structuredClone(store);
|
||||
}
|
||||
|
||||
public async overwrite(nextValue: T): Promise<void> {
|
||||
await this.serialize(async () => {
|
||||
await this.writeUnlocked(nextValue);
|
||||
await this.withInterprocessLock(() => this.writeUnlocked(nextValue));
|
||||
});
|
||||
}
|
||||
|
||||
public async mutate<R>(mutator: (current: T) => R | Promise<R>): Promise<R> {
|
||||
return this.serialize(async () => {
|
||||
const current = await this.readUnlocked();
|
||||
const result = await mutator(current);
|
||||
await this.writeUnlocked(current);
|
||||
return result;
|
||||
return this.withInterprocessLock(async () => {
|
||||
const current = await this.readUnlocked();
|
||||
const result = await mutator(current);
|
||||
await this.writeUnlocked(current);
|
||||
return result;
|
||||
});
|
||||
});
|
||||
}
|
||||
|
||||
public async normalize(): Promise<void> {
|
||||
await this.serialize(async () => {
|
||||
await this.withInterprocessLock(async () => {
|
||||
const current = await this.readUnlocked();
|
||||
await this.writeUnlocked(current);
|
||||
});
|
||||
});
|
||||
}
|
||||
|
||||
|
|
@ -42,7 +52,7 @@ export class SerializedFileStore<T> {
|
|||
private async readUnlocked(): Promise<T> {
|
||||
try {
|
||||
const raw = await readFile(this.filePath, "utf8");
|
||||
return JSON.parse(stripLeadingBom(raw)) as T;
|
||||
return parseSerializedValue(stripLeadingBom(raw)) as T;
|
||||
} catch (error) {
|
||||
const code = (error as NodeJS.ErrnoException).code;
|
||||
|
||||
|
|
@ -56,10 +66,89 @@ export class SerializedFileStore<T> {
|
|||
|
||||
private async writeUnlocked(value: T): Promise<void> {
|
||||
await mkdir(dirname(this.filePath), { recursive: true });
|
||||
await writeFile(this.filePath, `${JSON.stringify(value)}\n`, "utf8");
|
||||
const tempPath = `${this.filePath}.${process.pid}.${Date.now()}.tmp`;
|
||||
const serialized = `${JSON.stringify(value)}\n`;
|
||||
|
||||
try {
|
||||
await writeFile(tempPath, serialized, "utf8");
|
||||
await rm(this.filePath, { force: true });
|
||||
await rename(tempPath, this.filePath);
|
||||
} finally {
|
||||
await rm(tempPath, { force: true }).catch(() => undefined);
|
||||
}
|
||||
}
|
||||
|
||||
private async withInterprocessLock<R>(operation: () => Promise<R>): Promise<R> {
|
||||
const lockPath = `${this.filePath}.lock`;
|
||||
await mkdir(dirname(lockPath), { recursive: true });
|
||||
const deadline = Date.now() + 30_000;
|
||||
|
||||
for (;;) {
|
||||
try {
|
||||
const handle = await open(lockPath, "wx");
|
||||
|
||||
try {
|
||||
return await operation();
|
||||
} finally {
|
||||
await handle.close();
|
||||
await rm(lockPath, { force: true }).catch(() => undefined);
|
||||
}
|
||||
} catch (error) {
|
||||
const code = (error as NodeJS.ErrnoException).code;
|
||||
|
||||
if (code !== "EEXIST") {
|
||||
throw error;
|
||||
}
|
||||
|
||||
const lockIsStale = await isStaleLock(lockPath);
|
||||
|
||||
if (lockIsStale) {
|
||||
await rm(lockPath, { force: true }).catch(() => undefined);
|
||||
continue;
|
||||
}
|
||||
|
||||
if (Date.now() >= deadline) {
|
||||
throw new Error(`Timed out waiting for file lock: ${lockPath}`);
|
||||
}
|
||||
|
||||
await delay(100);
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
function stripLeadingBom(input: string): string {
|
||||
return input.charCodeAt(0) === 0xfeff ? input.slice(1) : input;
|
||||
}
|
||||
|
||||
function parseSerializedValue(input: string): unknown {
|
||||
try {
|
||||
return JSON.parse(input);
|
||||
} catch (error) {
|
||||
const lines = input
|
||||
.split(/\r?\n/)
|
||||
.map((line) => line.trim())
|
||||
.filter((line) => line.length > 0);
|
||||
|
||||
if (lines.length > 1) {
|
||||
return JSON.parse(lines[0]);
|
||||
}
|
||||
|
||||
throw error;
|
||||
}
|
||||
}
|
||||
|
||||
async function isStaleLock(lockPath: string): Promise<boolean> {
|
||||
try {
|
||||
const lockStat = await stat(lockPath);
|
||||
return Date.now() - lockStat.mtimeMs > 10 * 60 * 1000;
|
||||
} catch {
|
||||
return false;
|
||||
}
|
||||
}
|
||||
|
||||
async function delay(milliseconds: number): Promise<void> {
|
||||
await new Promise((resolve) => {
|
||||
setTimeout(resolve, milliseconds);
|
||||
});
|
||||
}
|
||||
|
|
|
|||
|
|
@ -1,3 +1,6 @@
|
|||
import { mkdir, readFile, rename, rm } from "node:fs/promises";
|
||||
import { basename, dirname, extname, join } from "node:path";
|
||||
|
||||
import { normalizeText } from "./history-ingester";
|
||||
import { SerializedFileStore } from "./file-store";
|
||||
|
||||
|
|
@ -22,11 +25,22 @@ export interface IndexedSourceRepoSummary {
|
|||
documentCount: number;
|
||||
}
|
||||
|
||||
interface SourceDocumentArchiveStore {
|
||||
interface LegacySourceDocumentArchiveStore {
|
||||
version: 1;
|
||||
documents: ArchivedSourceDocumentRecord[];
|
||||
}
|
||||
|
||||
interface SourceDocumentArchiveManifest {
|
||||
version: 2;
|
||||
repos: IndexedSourceRepoSummary[];
|
||||
}
|
||||
|
||||
interface SourceDocumentRepoShardStore {
|
||||
version: 1;
|
||||
repoName: string;
|
||||
documents: ArchivedSourceDocumentRecord[];
|
||||
}
|
||||
|
||||
export interface SourceDocumentSyncResult {
|
||||
created: number;
|
||||
updated: number;
|
||||
|
|
@ -39,24 +53,28 @@ export interface SourceDocumentSyncResult {
|
|||
}
|
||||
|
||||
export class FileSourceDocumentArchiveRepository {
|
||||
private readonly store: SerializedFileStore<SourceDocumentArchiveStore>;
|
||||
private readonly manifestStore: SerializedFileStore<SourceDocumentArchiveManifest>;
|
||||
private readonly shardRoot: string;
|
||||
private initialization?: Promise<void>;
|
||||
|
||||
public constructor(filePath: string) {
|
||||
this.store = new SerializedFileStore(filePath, () => ({
|
||||
version: 1,
|
||||
documents: [],
|
||||
public constructor(private readonly filePath: string) {
|
||||
this.manifestStore = new SerializedFileStore(filePath, () => ({
|
||||
version: 2,
|
||||
repos: [],
|
||||
}));
|
||||
this.shardRoot = deriveShardRoot(filePath);
|
||||
}
|
||||
|
||||
public async syncRepoDocuments(
|
||||
repoName: string,
|
||||
documents: ArchivedSourceDocument[],
|
||||
): Promise<SourceDocumentSyncResult> {
|
||||
return this.store.mutate((store) => {
|
||||
await this.ensureInitialized();
|
||||
|
||||
const shardStore = this.getRepoShardStore(repoName);
|
||||
const result = await shardStore.mutate((store) => {
|
||||
const now = new Date().toISOString();
|
||||
const previousRepoSourceIds = store.documents
|
||||
.filter((document) => document.repoName === repoName)
|
||||
.map((document) => document.id);
|
||||
const previousRepoSourceIds = store.documents.map((document) => document.id);
|
||||
const nextDocuments = documents.map((document) => toArchivedSourceDocumentRecord(document, now));
|
||||
const existingById = new Map(store.documents.map((document, index) => [document.id, index]));
|
||||
const nextIds = new Set(nextDocuments.map((document) => document.id));
|
||||
|
|
@ -93,10 +111,6 @@ export class FileSourceDocumentArchiveRepository {
|
|||
}
|
||||
|
||||
const retainedDocuments = store.documents.filter((document) => {
|
||||
if (document.repoName !== repoName) {
|
||||
return true;
|
||||
}
|
||||
|
||||
if (nextIds.has(document.id)) {
|
||||
return true;
|
||||
}
|
||||
|
|
@ -116,48 +130,252 @@ export class FileSourceDocumentArchiveRepository {
|
|||
previousRepoSourceIds,
|
||||
changedSourceIds,
|
||||
changedDocuments,
|
||||
activeDocuments: retainedDocuments
|
||||
.filter((document) => document.repoName === repoName)
|
||||
.sort((left, right) => left.path.localeCompare(right.path)),
|
||||
activeDocuments: retainedDocuments.slice().sort(compareDocuments),
|
||||
};
|
||||
});
|
||||
|
||||
await this.syncRepoSummary(repoName, result.activeDocuments.length);
|
||||
|
||||
if (result.activeDocuments.length === 0) {
|
||||
await this.deleteRepoShard(repoName);
|
||||
}
|
||||
|
||||
return result;
|
||||
}
|
||||
|
||||
public async get(sourceId: string): Promise<ArchivedSourceDocumentRecord | undefined> {
|
||||
const store = await this.store.snapshot();
|
||||
await this.ensureInitialized();
|
||||
const repoName = parseRepoNameFromSourceId(sourceId);
|
||||
|
||||
if (!repoName) {
|
||||
return undefined;
|
||||
}
|
||||
|
||||
const store = await this.getRepoShardStore(repoName).snapshot();
|
||||
return store.documents.find((document) => document.id === sourceId);
|
||||
}
|
||||
|
||||
public async listAll(): Promise<ArchivedSourceDocumentRecord[]> {
|
||||
const store = await this.store.snapshot();
|
||||
return store.documents.slice().sort(compareDocuments);
|
||||
await this.ensureInitialized();
|
||||
const summaries = await this.listRepoSummaries();
|
||||
const repoDocuments = await Promise.all(
|
||||
summaries.map((summary) => this.listByRepo(summary.repoName)),
|
||||
);
|
||||
|
||||
return repoDocuments.flat().sort(compareDocuments);
|
||||
}
|
||||
|
||||
public async listByRepo(repoName: string): Promise<ArchivedSourceDocumentRecord[]> {
|
||||
const documents = await this.listAll();
|
||||
return documents.filter((document) => document.repoName === repoName);
|
||||
await this.ensureInitialized();
|
||||
const store = await this.getRepoShardStore(repoName).snapshot();
|
||||
return store.documents.slice().sort(compareDocuments);
|
||||
}
|
||||
|
||||
public async listRepoSummaries(): Promise<IndexedSourceRepoSummary[]> {
|
||||
const store = await this.store.snapshot();
|
||||
const counts = new Map<string, number>();
|
||||
|
||||
for (const document of store.documents) {
|
||||
counts.set(document.repoName, (counts.get(document.repoName) ?? 0) + 1);
|
||||
}
|
||||
|
||||
return [...counts.entries()]
|
||||
.map(([repoName, documentCount]) => ({
|
||||
repoName,
|
||||
documentCount,
|
||||
}))
|
||||
await this.ensureInitialized();
|
||||
const store = await this.manifestStore.snapshot();
|
||||
return store.repos
|
||||
.slice()
|
||||
.sort((left, right) => left.repoName.localeCompare(right.repoName));
|
||||
}
|
||||
|
||||
public async count(): Promise<number> {
|
||||
const store = await this.store.snapshot();
|
||||
return store.documents.length;
|
||||
const summaries = await this.listRepoSummaries();
|
||||
return summaries.reduce((total, summary) => total + summary.documentCount, 0);
|
||||
}
|
||||
|
||||
public async normalizeStore(): Promise<void> {
|
||||
await this.ensureInitialized();
|
||||
|
||||
const summaries = await this.listRepoSummaries();
|
||||
await this.manifestStore.normalize();
|
||||
|
||||
for (const summary of summaries) {
|
||||
await this.getRepoShardStore(summary.repoName).normalize();
|
||||
}
|
||||
}
|
||||
|
||||
public async removeRepo(repoName: string): Promise<string[]> {
|
||||
await this.ensureInitialized();
|
||||
|
||||
const shardStore = this.getRepoShardStore(repoName);
|
||||
const store = await shardStore.snapshot();
|
||||
const deletedSourceIds = store.documents.map((document) => document.id);
|
||||
|
||||
await this.deleteRepoShard(repoName);
|
||||
await this.syncRepoSummary(repoName, 0);
|
||||
|
||||
return deletedSourceIds;
|
||||
}
|
||||
|
||||
private async ensureInitialized(): Promise<void> {
|
||||
if (!this.initialization) {
|
||||
this.initialization = this.initialize();
|
||||
}
|
||||
|
||||
await this.initialization;
|
||||
}
|
||||
|
||||
private async initialize(): Promise<void> {
|
||||
await mkdir(dirname(this.filePath), { recursive: true });
|
||||
await mkdir(this.shardRoot, { recursive: true });
|
||||
|
||||
const raw = await readRawFile(this.filePath);
|
||||
|
||||
if (!raw) {
|
||||
return;
|
||||
}
|
||||
|
||||
const parsed = parseArchivePayload(raw);
|
||||
|
||||
if (isManifestStore(parsed)) {
|
||||
return;
|
||||
}
|
||||
|
||||
if (isLegacyStore(parsed)) {
|
||||
await this.migrateLegacyStore(parsed);
|
||||
}
|
||||
}
|
||||
|
||||
private async migrateLegacyStore(legacyStore: LegacySourceDocumentArchiveStore): Promise<void> {
|
||||
const documentsByRepo = new Map<string, ArchivedSourceDocumentRecord[]>();
|
||||
|
||||
for (const document of legacyStore.documents) {
|
||||
const repoDocuments = documentsByRepo.get(document.repoName) ?? [];
|
||||
repoDocuments.push(document);
|
||||
documentsByRepo.set(document.repoName, repoDocuments);
|
||||
}
|
||||
|
||||
for (const [repoName, documents] of documentsByRepo.entries()) {
|
||||
const shardStore = this.getRepoShardStore(repoName);
|
||||
await shardStore.overwrite({
|
||||
version: 1,
|
||||
repoName,
|
||||
documents: documents.slice().sort(compareDocuments),
|
||||
});
|
||||
}
|
||||
|
||||
const backupPath = `${this.filePath}.legacy-${Date.now()}.bak`;
|
||||
await rename(this.filePath, backupPath).catch(async () => {
|
||||
await rm(backupPath, { force: true }).catch(() => undefined);
|
||||
await rename(this.filePath, backupPath);
|
||||
});
|
||||
|
||||
await this.manifestStore.overwrite({
|
||||
version: 2,
|
||||
repos: [...documentsByRepo.entries()]
|
||||
.map(([repoName, documents]) => ({
|
||||
repoName,
|
||||
documentCount: documents.length,
|
||||
}))
|
||||
.sort((left, right) => left.repoName.localeCompare(right.repoName)),
|
||||
});
|
||||
}
|
||||
|
||||
private getRepoShardStore(repoName: string): SerializedFileStore<SourceDocumentRepoShardStore> {
|
||||
return new SerializedFileStore(resolveRepoShardPath(this.shardRoot, repoName), () => ({
|
||||
version: 1,
|
||||
repoName,
|
||||
documents: [],
|
||||
}));
|
||||
}
|
||||
|
||||
private async syncRepoSummary(repoName: string, documentCount: number): Promise<void> {
|
||||
await this.manifestStore.mutate((manifest) => {
|
||||
const retainedRepos = manifest.repos.filter((summary) => summary.repoName !== repoName);
|
||||
|
||||
if (documentCount > 0) {
|
||||
retainedRepos.push({
|
||||
repoName,
|
||||
documentCount,
|
||||
});
|
||||
}
|
||||
|
||||
manifest.repos = retainedRepos.sort((left, right) => left.repoName.localeCompare(right.repoName));
|
||||
});
|
||||
}
|
||||
|
||||
private async deleteRepoShard(repoName: string): Promise<void> {
|
||||
await rm(resolveRepoShardPath(this.shardRoot, repoName), { force: true }).catch(() => undefined);
|
||||
}
|
||||
}
|
||||
|
||||
function deriveShardRoot(filePath: string): string {
|
||||
const extension = extname(filePath);
|
||||
const baseName = extension.length > 0 ? basename(filePath, extension) : basename(filePath);
|
||||
return join(dirname(filePath), `${baseName}.repos`);
|
||||
}
|
||||
|
||||
function resolveRepoShardPath(shardRoot: string, repoName: string): string {
|
||||
const safeRepoName = encodeURIComponent(repoName);
|
||||
return join(shardRoot, `${safeRepoName}.json`);
|
||||
}
|
||||
|
||||
async function readRawFile(filePath: string): Promise<string | undefined> {
|
||||
try {
|
||||
return stripLeadingBom(await readFile(filePath, "utf8"));
|
||||
} catch (error) {
|
||||
const code = (error as NodeJS.ErrnoException).code;
|
||||
|
||||
if (code === "ENOENT") {
|
||||
return undefined;
|
||||
}
|
||||
|
||||
throw error;
|
||||
}
|
||||
}
|
||||
|
||||
function parseArchivePayload(input: string): unknown {
|
||||
try {
|
||||
return JSON.parse(input);
|
||||
} catch (error) {
|
||||
const lines = input
|
||||
.split(/\r?\n/)
|
||||
.map((line) => line.trim())
|
||||
.filter((line) => line.length > 0);
|
||||
|
||||
if (lines.length > 1) {
|
||||
return JSON.parse(lines[0]);
|
||||
}
|
||||
|
||||
throw error;
|
||||
}
|
||||
}
|
||||
|
||||
function isLegacyStore(value: unknown): value is LegacySourceDocumentArchiveStore {
|
||||
return Boolean(
|
||||
value &&
|
||||
typeof value === "object" &&
|
||||
"version" in value &&
|
||||
(value as { version?: unknown }).version === 1 &&
|
||||
"documents" in value &&
|
||||
Array.isArray((value as { documents?: unknown }).documents),
|
||||
);
|
||||
}
|
||||
|
||||
function isManifestStore(value: unknown): value is SourceDocumentArchiveManifest {
|
||||
return Boolean(
|
||||
value &&
|
||||
typeof value === "object" &&
|
||||
"version" in value &&
|
||||
(value as { version?: unknown }).version === 2 &&
|
||||
"repos" in value &&
|
||||
Array.isArray((value as { repos?: unknown }).repos),
|
||||
);
|
||||
}
|
||||
|
||||
function parseRepoNameFromSourceId(sourceId: string): string | undefined {
|
||||
const separatorIndex = sourceId.indexOf(":");
|
||||
|
||||
if (separatorIndex <= 0) {
|
||||
return undefined;
|
||||
}
|
||||
|
||||
return sourceId.slice(0, separatorIndex);
|
||||
}
|
||||
|
||||
function stripLeadingBom(input: string): string {
|
||||
return input.charCodeAt(0) === 0xfeff ? input.slice(1) : input;
|
||||
}
|
||||
|
||||
function toArchivedSourceDocumentRecord(
|
||||
|
|
|
|||
|
|
@ -235,6 +235,46 @@ function Invoke-CommandWithTimeout {
|
|||
}
|
||||
}
|
||||
|
||||
function Invoke-SourceRepoReconcile {
|
||||
param(
|
||||
[switch] $CheckOnly,
|
||||
[switch] $Detached
|
||||
)
|
||||
|
||||
$tsxCommand = Join-Path $repoRoot "node_modules\.bin\tsx.cmd"
|
||||
|
||||
if (-not (Test-Path -LiteralPath $tsxCommand)) {
|
||||
throw "Local tsx launcher is missing at $tsxCommand."
|
||||
}
|
||||
|
||||
$arguments = @("scripts/reconcile-source-repos.ts", "--json")
|
||||
|
||||
if ($CheckOnly) {
|
||||
$arguments += "--check"
|
||||
}
|
||||
|
||||
if ($Detached) {
|
||||
$arguments += "--detached"
|
||||
}
|
||||
|
||||
$stdout = (& $tsxCommand @arguments 2>&1 | Out-String).Trim()
|
||||
$exitCode = $LASTEXITCODE
|
||||
|
||||
if ($exitCode -ne 0) {
|
||||
if ([string]::IsNullOrWhiteSpace($stdout)) {
|
||||
$stdout = "Source repo reconcile exited with code $exitCode."
|
||||
}
|
||||
|
||||
throw $stdout
|
||||
}
|
||||
|
||||
if ([string]::IsNullOrWhiteSpace($stdout)) {
|
||||
throw "Source repo reconcile returned no JSON output."
|
||||
}
|
||||
|
||||
return $stdout | ConvertFrom-Json
|
||||
}
|
||||
|
||||
function Ensure-DockerCliOnPath {
|
||||
$dockerCliDir = "C:\Program Files\Docker\Docker\resources\bin"
|
||||
|
||||
|
|
@ -652,6 +692,7 @@ try {
|
|||
$vectorStoreKind = if ($config.ContainsKey("VECTOR_STORE_KIND")) { $config["VECTOR_STORE_KIND"] } else { "local_json" }
|
||||
$historyCollection = if ($config.ContainsKey("QDRANT_HISTORY_COLLECTION")) { $config["QDRANT_HISTORY_COLLECTION"] } else { "voidbot_discord_history_chunks" }
|
||||
$sourceCollection = if ($config.ContainsKey("QDRANT_SOURCE_COLLECTION")) { $config["QDRANT_SOURCE_COLLECTION"] } else { "voidbot_repository_source_chunks" }
|
||||
$qdrantReady = $vectorStoreKind -ne "qdrant"
|
||||
|
||||
Set-WatchdogStep -CurrentStep "qdrant_checks" -Detail "Verifying Qdrant reachability and required collections."
|
||||
if ($vectorStoreKind -eq "qdrant") {
|
||||
|
|
@ -659,6 +700,7 @@ try {
|
|||
try {
|
||||
$collectionsResponse = Get-Json -Url "$($qdrantUrl.TrimEnd('/'))/collections"
|
||||
$collectionNames = @($collectionsResponse.result.collections | ForEach-Object { [string]$_.name })
|
||||
$qdrantReady = $true
|
||||
Add-Check -Name "qdrant.reachable" -Status "passed" -Detail "Qdrant is reachable at $qdrantUrl." -Data @{
|
||||
url = $qdrantUrl
|
||||
collections = $collectionNames
|
||||
|
|
@ -742,6 +784,7 @@ try {
|
|||
|
||||
$ragOllamaUrl = if ($config.ContainsKey("RAG_OLLAMA_BASE_URL")) { $config["RAG_OLLAMA_BASE_URL"] } else { "http://127.0.0.1:11434" }
|
||||
$ragOllamaModel = if ($config.ContainsKey("RAG_OLLAMA_MODEL")) { $config["RAG_OLLAMA_MODEL"] } else { "qwen3-embedding:0.6b" }
|
||||
$ragOllamaReady = $false
|
||||
|
||||
Set-WatchdogStep -CurrentStep "ollama_checks" -Detail "Checking embedding and local LLM Ollama endpoints."
|
||||
try {
|
||||
|
|
@ -749,6 +792,7 @@ try {
|
|||
$ragModels = @($ragTags.models | ForEach-Object { [string]$_.name })
|
||||
|
||||
if ($ragModels -contains $ragOllamaModel) {
|
||||
$ragOllamaReady = $true
|
||||
Add-Check -Name "rag_ollama.model" -Status "passed" -Detail "Embedding Ollama is reachable and has $ragOllamaModel." -Data @{
|
||||
url = $ragOllamaUrl
|
||||
model = $ragOllamaModel
|
||||
|
|
@ -805,6 +849,54 @@ try {
|
|||
}
|
||||
}
|
||||
|
||||
$sourceRepoRoot = if ($config.ContainsKey("SOURCE_REPO_ROOT")) { $config["SOURCE_REPO_ROOT"] } else { "" }
|
||||
if ([string]::IsNullOrWhiteSpace($sourceRepoRoot)) {
|
||||
Add-Check -Name "source.repos" -Status "passed" -Detail "Source repo reconciliation check skipped because SOURCE_REPO_ROOT is blank."
|
||||
} else {
|
||||
Set-WatchdogStep -CurrentStep "source_repo_checks" -Detail "Reconciling discovered source repos against the indexed source archive."
|
||||
|
||||
try {
|
||||
$sourceRepoPreview = Invoke-SourceRepoReconcile -CheckOnly
|
||||
$missingRepos = @($sourceRepoPreview.missingRepos | ForEach-Object { [string]$_ })
|
||||
$staleIndexedRepos = @($sourceRepoPreview.staleIndexedRepos | ForEach-Object { [string]$_ })
|
||||
|
||||
if ($missingRepos.Count -eq 0) {
|
||||
Add-Check -Name "source.repos" -Status "passed" -Detail "All discovered source repos are represented in the indexed source archive." -Data @{
|
||||
discoveredRepoCount = [int]$sourceRepoPreview.discoveredRepoCount
|
||||
indexedRepoCount = [int]$sourceRepoPreview.indexedRepoCount
|
||||
selectedRepoCount = [int]$sourceRepoPreview.selectedRepoCount
|
||||
}
|
||||
} elseif ($qdrantReady -and $ragOllamaReady) {
|
||||
$sourceRepoRepair = Invoke-SourceRepoReconcile -Detached
|
||||
$launchedRepos = @($sourceRepoRepair.launchedRepos | ForEach-Object { [string]$_ })
|
||||
Add-Check -Name "source.repos" -Status "warning" -Detail "Detected $($missingRepos.Count) unindexed source repos and launched detached reconciliation." -Data @{
|
||||
missingRepos = $missingRepos
|
||||
launchedRepos = $launchedRepos
|
||||
discoveredRepoCount = [int]$sourceRepoPreview.discoveredRepoCount
|
||||
indexedRepoCount = [int]$sourceRepoPreview.indexedRepoCount
|
||||
}
|
||||
} else {
|
||||
Add-Check -Name "source.repos" -Status "failed" -Detail "Detected $($missingRepos.Count) unindexed source repos, but source indexing dependencies are unhealthy." -Data @{
|
||||
missingRepos = $missingRepos
|
||||
qdrantReady = $qdrantReady
|
||||
ragOllamaReady = $ragOllamaReady
|
||||
}
|
||||
}
|
||||
|
||||
if ($staleIndexedRepos.Count -eq 0) {
|
||||
Add-Check -Name "source.stale_index" -Status "passed" -Detail "Indexed source archive has no repo summaries for repos that are no longer present."
|
||||
} else {
|
||||
Add-Check -Name "source.stale_index" -Status "warning" -Detail "Indexed source archive still carries repo summaries for repos that are no longer present." -Data @{
|
||||
staleIndexedRepos = $staleIndexedRepos
|
||||
}
|
||||
}
|
||||
} catch {
|
||||
Add-Check -Name "source.repos" -Status "failed" -Detail "Source repo reconciliation check failed: $($_.Exception.Message)" -Data @{
|
||||
sourceRepoRoot = $sourceRepoRoot
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
if ($config.ContainsKey("DISCORD_BOT_TOKEN") -and -not [string]::IsNullOrWhiteSpace($config["DISCORD_BOT_TOKEN"])) {
|
||||
Set-WatchdogStep -CurrentStep "discord_checks" -Detail "Validating Discord bot credentials."
|
||||
try {
|
||||
|
|
|
|||
|
|
@ -1,17 +1,9 @@
|
|||
import "dotenv/config";
|
||||
|
||||
import { readdir } from "node:fs/promises";
|
||||
import { join } from "node:path";
|
||||
|
||||
import { loadConfig } from "@voidbot/config";
|
||||
import {
|
||||
crawlRepositoryDocuments,
|
||||
createTextEmbedder,
|
||||
createVectorStores,
|
||||
FileSourceDocumentArchiveRepository,
|
||||
SourceDocumentIngester,
|
||||
SourceRagPipeline,
|
||||
} from "@voidbot/rag";
|
||||
|
||||
import { discoverSourceRepos, selectSourceRepos } from "./source-repo-discovery";
|
||||
import { indexSourceRepos } from "./source-repo-indexing";
|
||||
|
||||
interface ScriptOptions {
|
||||
repos?: string[];
|
||||
|
|
@ -31,67 +23,21 @@ async function main(): Promise<void> {
|
|||
}
|
||||
|
||||
const repoNames =
|
||||
options.repos && options.repos.length > 0
|
||||
? options.repos
|
||||
: await resolveRepoNames(config.sourceRepoRoot, config.sourceRepoPatterns);
|
||||
await discoverSourceRepos(config.sourceRepoRoot, config.sourceRepoPatterns);
|
||||
const repos = selectSourceRepos(repoNames, options.repos);
|
||||
|
||||
if (repoNames.length === 0) {
|
||||
if (repos.length === 0) {
|
||||
throw new Error("No source repositories matched SOURCE_REPO_PATTERNS.");
|
||||
}
|
||||
|
||||
const embedder = createTextEmbedder({
|
||||
backend: config.ragEmbeddingBackend,
|
||||
hashDimensions: config.ragEmbeddingDimensions,
|
||||
ollamaBaseUrl: config.ragOllamaBaseUrl,
|
||||
ollamaModel: config.ragOllamaModel,
|
||||
ollamaTimeoutMs: config.ragOllamaTimeoutMs,
|
||||
queryInstruction: config.ragQueryInstruction,
|
||||
const summary = await indexSourceRepos(config, repos, {
|
||||
forceReindex: options.forceReindex,
|
||||
logger: (line) => console.log(line),
|
||||
});
|
||||
const vectorStore = createVectorStores({
|
||||
kind: config.vectorStore.kind,
|
||||
historyPath: config.vectorStore.path,
|
||||
sourceRoot: config.sourceVectorStoreRoot,
|
||||
qdrant: config.qdrant,
|
||||
historyEmbedder: embedder,
|
||||
sourceEmbedder: embedder,
|
||||
}).source;
|
||||
const archiveRepository = new FileSourceDocumentArchiveRepository(config.ragSourceArchivePath);
|
||||
const sourceIngester = new SourceDocumentIngester();
|
||||
const pipeline = new SourceRagPipeline(archiveRepository, sourceIngester, vectorStore);
|
||||
|
||||
let indexedRepositories = 0;
|
||||
let totalDocuments = 0;
|
||||
let totalChunks = 0;
|
||||
|
||||
for (const repoName of repoNames) {
|
||||
const repoPath = join(config.sourceRepoRoot, repoName);
|
||||
const scan = await crawlRepositoryDocuments(repoPath, repoName, {
|
||||
includePathPrefixes: config.sourceRepoIncludePrefixes[repoName],
|
||||
});
|
||||
const result = await pipeline.syncRepoDocuments(repoName, scan.documents, {
|
||||
forceReindex: options.forceReindex,
|
||||
});
|
||||
indexedRepositories += 1;
|
||||
totalDocuments += scan.documents.length;
|
||||
totalChunks += result.indexedChunks;
|
||||
|
||||
console.log(
|
||||
[
|
||||
`Repo ${repoName}:`,
|
||||
`documents=${scan.documents.length}`,
|
||||
`created=${result.createdDocuments}`,
|
||||
`updated=${result.updatedDocuments}`,
|
||||
`unchanged=${result.unchangedDocuments}`,
|
||||
`deleted=${result.deletedDocuments}`,
|
||||
`indexedChunks=${result.indexedChunks}`,
|
||||
`skippedFiles=${scan.skippedFiles.length}`,
|
||||
].join(" "),
|
||||
);
|
||||
}
|
||||
|
||||
console.log(`Indexed repositories: ${indexedRepositories}`);
|
||||
console.log(`Indexed documents: ${totalDocuments}`);
|
||||
console.log(`Indexed chunks: ${totalChunks}`);
|
||||
console.log(`Indexed repositories: ${summary.indexedRepositories}`);
|
||||
console.log(`Indexed documents: ${summary.totalDocuments}`);
|
||||
console.log(`Indexed chunks: ${summary.totalChunks}`);
|
||||
}
|
||||
|
||||
function parseArgs(argv: string[]): ScriptOptions {
|
||||
|
|
@ -117,23 +63,6 @@ function parseArgs(argv: string[]): ScriptOptions {
|
|||
return options;
|
||||
}
|
||||
|
||||
async function resolveRepoNames(root: string, patterns: string[]): Promise<string[]> {
|
||||
const entries = await readdir(root, { withFileTypes: true });
|
||||
const repoNames = entries
|
||||
.filter((entry) => entry.isDirectory())
|
||||
.map((entry) => entry.name)
|
||||
.filter((name) => patterns.length === 0 || patterns.some((pattern) => matchesPattern(name, pattern)))
|
||||
.sort((left, right) => left.localeCompare(right));
|
||||
|
||||
return repoNames;
|
||||
}
|
||||
|
||||
function matchesPattern(value: string, pattern: string): boolean {
|
||||
const escaped = pattern.replace(/[.+?^${}()|[\]\\]/g, "\\$&").replace(/\*/g, ".*");
|
||||
const regex = new RegExp(`^${escaped}$`, "i");
|
||||
return regex.test(value);
|
||||
}
|
||||
|
||||
void main().catch((error) => {
|
||||
console.error(error);
|
||||
process.exitCode = 1;
|
||||
|
|
|
|||
|
|
@ -1,11 +1,8 @@
|
|||
import "dotenv/config";
|
||||
|
||||
import { readdir, chmod, mkdir, readFile, stat, writeFile } from "node:fs/promises";
|
||||
import { join, resolve } from "node:path";
|
||||
|
||||
import { loadConfig } from "@voidbot/config";
|
||||
|
||||
const HOOK_MARKER = "# voidbot-source-index-push-hook";
|
||||
import { discoverSourceRepos, selectSourceRepos } from "./source-repo-discovery";
|
||||
import { installSourceIndexPushHooks } from "./source-repo-hooks";
|
||||
|
||||
interface ScriptOptions {
|
||||
repos?: string[];
|
||||
|
|
@ -23,77 +20,20 @@ async function main(): Promise<void> {
|
|||
}
|
||||
|
||||
const options = parseArgs(process.argv.slice(2));
|
||||
const repoNames =
|
||||
options.repos && options.repos.length > 0
|
||||
? options.repos
|
||||
: await resolveRepoNames(config.sourceRepoRoot, config.sourceRepoPatterns);
|
||||
const discoveredRepos = await discoverSourceRepos(config.sourceRepoRoot, config.sourceRepoPatterns);
|
||||
const selectedRepos = selectSourceRepos(discoveredRepos, options.repos);
|
||||
const summary = await installSourceIndexPushHooks(process.cwd(), selectedRepos);
|
||||
|
||||
let installed = 0;
|
||||
let updated = 0;
|
||||
let backedUp = 0;
|
||||
let skipped = 0;
|
||||
|
||||
for (const repoName of repoNames) {
|
||||
const repoPath = join(config.sourceRepoRoot, repoName);
|
||||
const gitDir = await resolveGitDir(repoPath);
|
||||
|
||||
if (!gitDir) {
|
||||
console.log(`Skipping ${repoName}: no .git directory found.`);
|
||||
skipped += 1;
|
||||
continue;
|
||||
}
|
||||
|
||||
const hooksDir = join(gitDir, "hooks");
|
||||
const hookPath = join(hooksDir, "pre-push");
|
||||
const backupHookPath = join(hooksDir, "pre-push.voidbot.prev");
|
||||
|
||||
await mkdir(hooksDir, { recursive: true });
|
||||
|
||||
let existingContent: string | undefined;
|
||||
|
||||
try {
|
||||
existingContent = await readFile(hookPath, "utf8");
|
||||
} catch {
|
||||
existingContent = undefined;
|
||||
}
|
||||
|
||||
if (
|
||||
existingContent &&
|
||||
!existingContent.includes(HOOK_MARKER) &&
|
||||
!(await pathExists(backupHookPath))
|
||||
) {
|
||||
await writeFile(backupHookPath, existingContent, "utf8");
|
||||
await chmod(backupHookPath, 0o755);
|
||||
backedUp += 1;
|
||||
} else if (
|
||||
existingContent &&
|
||||
!existingContent.includes(HOOK_MARKER) &&
|
||||
(await pathExists(backupHookPath))
|
||||
) {
|
||||
console.log(`Skipping ${repoName}: existing pre-push hook already has a Void backup; not clobbering it.`);
|
||||
skipped += 1;
|
||||
continue;
|
||||
}
|
||||
|
||||
const hookScript = renderHookScript(process.cwd(), repoName);
|
||||
await writeFile(hookPath, hookScript, "utf8");
|
||||
await chmod(hookPath, 0o755);
|
||||
|
||||
if (existingContent?.includes(HOOK_MARKER)) {
|
||||
updated += 1;
|
||||
console.log(`Updated pre-push hook for ${repoName}.`);
|
||||
} else {
|
||||
installed += 1;
|
||||
console.log(`Installed pre-push hook for ${repoName}.`);
|
||||
}
|
||||
for (const message of summary.messages) {
|
||||
console.log(message);
|
||||
}
|
||||
|
||||
console.log(
|
||||
[
|
||||
`installed=${installed}`,
|
||||
`updated=${updated}`,
|
||||
`backedUp=${backedUp}`,
|
||||
`skipped=${skipped}`,
|
||||
`installed=${summary.installed}`,
|
||||
`updated=${summary.updated}`,
|
||||
`backedUp=${summary.backedUp}`,
|
||||
`skipped=${summary.skipped}`,
|
||||
].join(" "),
|
||||
);
|
||||
}
|
||||
|
|
@ -116,89 +56,6 @@ function parseArgs(argv: string[]): ScriptOptions {
|
|||
return options;
|
||||
}
|
||||
|
||||
async function resolveRepoNames(root: string, patterns: string[]): Promise<string[]> {
|
||||
const entries = await readdir(root, { withFileTypes: true });
|
||||
return entries
|
||||
.filter((entry) => entry.isDirectory())
|
||||
.map((entry) => entry.name)
|
||||
.filter((name) => patterns.length === 0 || patterns.some((pattern) => matchesPattern(name, pattern)))
|
||||
.sort((left, right) => left.localeCompare(right));
|
||||
}
|
||||
|
||||
async function resolveGitDir(repoPath: string): Promise<string | undefined> {
|
||||
const dotGitPath = join(repoPath, ".git");
|
||||
|
||||
try {
|
||||
const dotGitStats = await stat(dotGitPath);
|
||||
|
||||
if (dotGitStats.isDirectory()) {
|
||||
return dotGitPath;
|
||||
}
|
||||
|
||||
if (dotGitStats.isFile()) {
|
||||
const pointer = await readFile(dotGitPath, "utf8");
|
||||
const match = pointer.match(/^gitdir:\s*(.+)$/im);
|
||||
|
||||
if (match?.[1]) {
|
||||
return resolve(repoPath, match[1].trim());
|
||||
}
|
||||
}
|
||||
} catch {
|
||||
return undefined;
|
||||
}
|
||||
|
||||
return undefined;
|
||||
}
|
||||
|
||||
function renderHookScript(voidbotRoot: string, repoName: string): string {
|
||||
const normalizedRoot = voidbotRoot.replace(/\\/g, "/");
|
||||
const launcherPath = `${normalizedRoot}/scripts/git-post-push-index.mjs`;
|
||||
const nodePath = process.execPath.replace(/\\/g, "/");
|
||||
const quotedNodePath = escapeForDoubleQuotedShell(nodePath);
|
||||
const quotedLauncherPath = escapeForDoubleQuotedShell(launcherPath);
|
||||
const quotedRoot = escapeForDoubleQuotedShell(normalizedRoot);
|
||||
const quotedRepoName = escapeForDoubleQuotedShell(repoName);
|
||||
|
||||
return [
|
||||
"#!/bin/sh",
|
||||
HOOK_MARKER,
|
||||
"HOOK_DIR=\"$(CDPATH= cd -- \"$(dirname -- \"$0\")\" && pwd)\"",
|
||||
"PREV_HOOK=\"$HOOK_DIR/pre-push.voidbot.prev\"",
|
||||
"",
|
||||
"if [ -x \"$PREV_HOOK\" ]; then",
|
||||
" \"$PREV_HOOK\" \"$@\"",
|
||||
" PREV_STATUS=$?",
|
||||
" if [ \"$PREV_STATUS\" -ne 0 ]; then",
|
||||
" exit \"$PREV_STATUS\"",
|
||||
" fi",
|
||||
"fi",
|
||||
"",
|
||||
`\"${quotedNodePath}\" \"${quotedLauncherPath}\" --voidbot-root \"${quotedRoot}\" --repo-name \"${quotedRepoName}\" >/dev/null 2>&1 || true`,
|
||||
"",
|
||||
"exit 0",
|
||||
"",
|
||||
].join("\n");
|
||||
}
|
||||
|
||||
function escapeForDoubleQuotedShell(value: string): string {
|
||||
return value.replace(/(["`$\\])/g, "\\$1");
|
||||
}
|
||||
|
||||
function matchesPattern(value: string, pattern: string): boolean {
|
||||
const escaped = pattern.replace(/[.+?^${}()|[\]\\]/g, "\\$&").replace(/\*/g, ".*");
|
||||
const regex = new RegExp(`^${escaped}$`, "i");
|
||||
return regex.test(value);
|
||||
}
|
||||
|
||||
async function pathExists(path: string): Promise<boolean> {
|
||||
try {
|
||||
await stat(path);
|
||||
return true;
|
||||
} catch {
|
||||
return false;
|
||||
}
|
||||
}
|
||||
|
||||
void main().catch((error) => {
|
||||
console.error(error);
|
||||
process.exitCode = 1;
|
||||
|
|
|
|||
267
scripts/reconcile-source-repos.ts
Normal file
267
scripts/reconcile-source-repos.ts
Normal file
|
|
@ -0,0 +1,267 @@
|
|||
import "dotenv/config";
|
||||
|
||||
import { spawn } from "node:child_process";
|
||||
import { join } from "node:path";
|
||||
|
||||
import { loadConfig } from "@voidbot/config";
|
||||
import { FileSourceDocumentArchiveRepository } from "@voidbot/rag";
|
||||
|
||||
import { discoverSourceRepos, selectSourceRepos, type SourceRepoMatch } from "./source-repo-discovery";
|
||||
import { installSourceIndexPushHooks } from "./source-repo-hooks";
|
||||
import { createSourceVectorStore, indexSourceRepos } from "./source-repo-indexing";
|
||||
|
||||
interface ScriptOptions {
|
||||
repos?: string[];
|
||||
forceReindex?: boolean;
|
||||
detached?: boolean;
|
||||
json?: boolean;
|
||||
checkOnly?: boolean;
|
||||
}
|
||||
|
||||
interface ReconcileSummary {
|
||||
discoveredRepoCount: number;
|
||||
indexedRepoCount: number;
|
||||
selectedRepoCount: number;
|
||||
missingRepos: string[];
|
||||
staleIndexedRepos: string[];
|
||||
hookInstall: {
|
||||
installed: number;
|
||||
updated: number;
|
||||
backedUp: number;
|
||||
skipped: number;
|
||||
};
|
||||
prunedRepos: string[];
|
||||
normalizedArchive: boolean;
|
||||
detached: boolean;
|
||||
forceReindex: boolean;
|
||||
checkOnly: boolean;
|
||||
indexedRepos: string[];
|
||||
launchedRepos: string[];
|
||||
}
|
||||
|
||||
async function main(): Promise<void> {
|
||||
if (!process.env.DISCORD_OWNER_ID) {
|
||||
process.env.DISCORD_OWNER_ID = "__source_reconcile__";
|
||||
}
|
||||
|
||||
const options = parseArgs(process.argv.slice(2));
|
||||
const config = loadConfig();
|
||||
|
||||
if (!config.sourceRepoRoot) {
|
||||
throw new Error("SOURCE_REPO_ROOT is not configured.");
|
||||
}
|
||||
|
||||
const discoveredRepos = await discoverSourceRepos(config.sourceRepoRoot, config.sourceRepoPatterns);
|
||||
const selectedRepos = selectSourceRepos(discoveredRepos, options.repos);
|
||||
|
||||
if (selectedRepos.length === 0) {
|
||||
throw new Error("No source repositories matched SOURCE_REPO_PATTERNS.");
|
||||
}
|
||||
|
||||
const archiveRepository = new FileSourceDocumentArchiveRepository(config.ragSourceArchivePath);
|
||||
|
||||
if (!options.checkOnly) {
|
||||
await archiveRepository.normalizeStore();
|
||||
}
|
||||
|
||||
const indexedRepoSummaries = await archiveRepository.listRepoSummaries();
|
||||
const indexedRepoNames = new Set(indexedRepoSummaries.map((summary) => summary.repoName));
|
||||
const missingRepos = selectedRepos
|
||||
.filter((repo) => options.forceReindex || !indexedRepoNames.has(repo.repoName))
|
||||
.map((repo) => repo.repoName);
|
||||
const staleIndexedRepos = indexedRepoSummaries
|
||||
.map((summary) => summary.repoName)
|
||||
.filter((repoName) => !discoveredRepos.some((repo) => repo.repoName === repoName))
|
||||
.sort((left, right) => left.localeCompare(right));
|
||||
|
||||
let hookInstall = {
|
||||
installed: 0,
|
||||
updated: 0,
|
||||
backedUp: 0,
|
||||
skipped: 0,
|
||||
};
|
||||
|
||||
if (!options.checkOnly) {
|
||||
const hookSummary = await installSourceIndexPushHooks(process.cwd(), selectedRepos);
|
||||
hookInstall = {
|
||||
installed: hookSummary.installed,
|
||||
updated: hookSummary.updated,
|
||||
backedUp: hookSummary.backedUp,
|
||||
skipped: hookSummary.skipped,
|
||||
};
|
||||
|
||||
if (!options.json) {
|
||||
for (const message of hookSummary.messages) {
|
||||
console.log(message);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
const reposToIndex = selectedRepos.filter((repo) => missingRepos.includes(repo.repoName));
|
||||
const indexedRepos: string[] = [];
|
||||
const launchedRepos: string[] = [];
|
||||
const prunedRepos: string[] = [];
|
||||
|
||||
if (!options.checkOnly && reposToIndex.length > 0) {
|
||||
if (options.detached) {
|
||||
for (const repo of reposToIndex) {
|
||||
await launchDetachedRepoIndex(process.cwd(), repo.repoName);
|
||||
launchedRepos.push(repo.repoName);
|
||||
}
|
||||
} else {
|
||||
const runSummary = await indexSourceRepos(config, reposToIndex, {
|
||||
forceReindex: options.forceReindex,
|
||||
logger: options.json ? undefined : (line) => console.log(line),
|
||||
});
|
||||
indexedRepos.push(...runSummary.results.map((result) => result.repoName));
|
||||
|
||||
if (!options.json) {
|
||||
console.log(`Indexed repositories: ${runSummary.indexedRepositories}`);
|
||||
console.log(`Indexed documents: ${runSummary.totalDocuments}`);
|
||||
console.log(`Indexed chunks: ${runSummary.totalChunks}`);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
if (!options.checkOnly && staleIndexedRepos.length > 0) {
|
||||
const sourceVectorStore = createSourceVectorStore(config);
|
||||
|
||||
for (const repoName of staleIndexedRepos) {
|
||||
await sourceVectorStore.deleteByFilters({
|
||||
corpusKind: "repository_source",
|
||||
repoName,
|
||||
});
|
||||
await archiveRepository.removeRepo(repoName);
|
||||
prunedRepos.push(repoName);
|
||||
}
|
||||
}
|
||||
|
||||
const summary: ReconcileSummary = {
|
||||
discoveredRepoCount: discoveredRepos.length,
|
||||
indexedRepoCount: indexedRepoSummaries.length,
|
||||
selectedRepoCount: selectedRepos.length,
|
||||
missingRepos,
|
||||
staleIndexedRepos,
|
||||
hookInstall,
|
||||
prunedRepos,
|
||||
normalizedArchive: !options.checkOnly,
|
||||
detached: options.detached,
|
||||
forceReindex: options.forceReindex,
|
||||
checkOnly: options.checkOnly,
|
||||
indexedRepos,
|
||||
launchedRepos,
|
||||
};
|
||||
|
||||
if (options.json) {
|
||||
process.stdout.write(`${JSON.stringify(summary)}\n`);
|
||||
return;
|
||||
}
|
||||
|
||||
console.log(
|
||||
[
|
||||
`discoveredRepos=${summary.discoveredRepoCount}`,
|
||||
`indexedRepos=${summary.indexedRepoCount}`,
|
||||
`selectedRepos=${summary.selectedRepoCount}`,
|
||||
`missingRepos=${summary.missingRepos.length}`,
|
||||
`staleIndexedRepos=${summary.staleIndexedRepos.length}`,
|
||||
`installedHooks=${summary.hookInstall.installed}`,
|
||||
`updatedHooks=${summary.hookInstall.updated}`,
|
||||
`backedUpHooks=${summary.hookInstall.backedUp}`,
|
||||
`skippedHooks=${summary.hookInstall.skipped}`,
|
||||
`prunedStaleRepos=${summary.prunedRepos.length}`,
|
||||
`indexedNow=${summary.indexedRepos.length}`,
|
||||
`launchedDetached=${summary.launchedRepos.length}`,
|
||||
].join(" "),
|
||||
);
|
||||
}
|
||||
|
||||
function parseArgs(argv: string[]): ScriptOptions {
|
||||
const options: ScriptOptions = {
|
||||
forceReindex: false,
|
||||
detached: false,
|
||||
json: false,
|
||||
checkOnly: false,
|
||||
};
|
||||
|
||||
for (let index = 0; index < argv.length; index += 1) {
|
||||
const argument = argv[index];
|
||||
|
||||
if ((argument === "--repo" || argument === "--repos") && argv[index + 1]) {
|
||||
options.repos = argv[index + 1]
|
||||
.split(",")
|
||||
.map((entry) => entry.trim())
|
||||
.filter((entry) => entry.length > 0);
|
||||
index += 1;
|
||||
continue;
|
||||
}
|
||||
|
||||
if (argument === "--force") {
|
||||
options.forceReindex = true;
|
||||
continue;
|
||||
}
|
||||
|
||||
if (argument === "--detached") {
|
||||
options.detached = true;
|
||||
continue;
|
||||
}
|
||||
|
||||
if (argument === "--json") {
|
||||
options.json = true;
|
||||
continue;
|
||||
}
|
||||
|
||||
if (argument === "--check") {
|
||||
options.checkOnly = true;
|
||||
}
|
||||
}
|
||||
|
||||
return options;
|
||||
}
|
||||
|
||||
async function launchDetachedRepoIndex(voidbotRoot: string, repoName: string): Promise<void> {
|
||||
await new Promise<void>((resolvePromise, rejectPromise) => {
|
||||
const child = spawn(
|
||||
process.execPath,
|
||||
[
|
||||
join(voidbotRoot, "scripts", "git-post-push-index.mjs"),
|
||||
"--voidbot-root",
|
||||
voidbotRoot,
|
||||
"--repo-name",
|
||||
repoName,
|
||||
],
|
||||
{
|
||||
cwd: voidbotRoot,
|
||||
env: process.env,
|
||||
stdio: ["ignore", "ignore", "pipe"],
|
||||
windowsHide: true,
|
||||
},
|
||||
);
|
||||
|
||||
let stderr = "";
|
||||
|
||||
child.stderr.on("data", (chunk) => {
|
||||
stderr += String(chunk);
|
||||
});
|
||||
|
||||
child.on("error", rejectPromise);
|
||||
child.on("close", (code) => {
|
||||
if ((code ?? 1) !== 0) {
|
||||
rejectPromise(
|
||||
new Error(
|
||||
stderr.trim().length > 0
|
||||
? stderr.trim()
|
||||
: `Detached source-index launcher failed for ${repoName}.`,
|
||||
),
|
||||
);
|
||||
return;
|
||||
}
|
||||
|
||||
resolvePromise();
|
||||
});
|
||||
});
|
||||
}
|
||||
|
||||
void main().catch((error) => {
|
||||
console.error(error);
|
||||
process.exitCode = 1;
|
||||
});
|
||||
109
scripts/source-repo-discovery.ts
Normal file
109
scripts/source-repo-discovery.ts
Normal file
|
|
@ -0,0 +1,109 @@
|
|||
import { readdir, readFile, stat } from "node:fs/promises";
|
||||
import { join, resolve } from "node:path";
|
||||
|
||||
export interface SourceRepoMatch {
|
||||
repoName: string;
|
||||
repoPath: string;
|
||||
gitDir: string;
|
||||
}
|
||||
|
||||
export async function discoverSourceRepos(
|
||||
root: string,
|
||||
patterns: string[],
|
||||
): Promise<SourceRepoMatch[]> {
|
||||
const entries = await readdir(root, { withFileTypes: true });
|
||||
const repoMatches: SourceRepoMatch[] = [];
|
||||
|
||||
for (const entry of entries) {
|
||||
if (!entry.isDirectory()) {
|
||||
continue;
|
||||
}
|
||||
|
||||
if (patterns.length > 0 && !patterns.some((pattern) => matchesPattern(entry.name, pattern))) {
|
||||
continue;
|
||||
}
|
||||
|
||||
const repoPath = join(root, entry.name);
|
||||
const gitDir = await resolveGitDir(repoPath);
|
||||
|
||||
if (!gitDir) {
|
||||
continue;
|
||||
}
|
||||
|
||||
repoMatches.push({
|
||||
repoName: entry.name,
|
||||
repoPath,
|
||||
gitDir,
|
||||
});
|
||||
}
|
||||
|
||||
repoMatches.sort((left, right) => left.repoName.localeCompare(right.repoName));
|
||||
return repoMatches;
|
||||
}
|
||||
|
||||
export function selectSourceRepos(
|
||||
availableRepos: SourceRepoMatch[],
|
||||
requestedRepoNames?: string[],
|
||||
): SourceRepoMatch[] {
|
||||
if (!requestedRepoNames || requestedRepoNames.length === 0) {
|
||||
return availableRepos;
|
||||
}
|
||||
|
||||
const lookup = new Map(
|
||||
availableRepos.map((repo) => [repo.repoName.toLowerCase(), repo] as const),
|
||||
);
|
||||
const selectedRepos: SourceRepoMatch[] = [];
|
||||
const missingRepoNames: string[] = [];
|
||||
|
||||
for (const requestedRepoName of requestedRepoNames) {
|
||||
const repo = lookup.get(requestedRepoName.toLowerCase());
|
||||
|
||||
if (!repo) {
|
||||
missingRepoNames.push(requestedRepoName);
|
||||
continue;
|
||||
}
|
||||
|
||||
if (!selectedRepos.some((entry) => entry.repoName === repo.repoName)) {
|
||||
selectedRepos.push(repo);
|
||||
}
|
||||
}
|
||||
|
||||
if (missingRepoNames.length > 0) {
|
||||
throw new Error(
|
||||
`Requested source repos were not found under SOURCE_REPO_ROOT: ${missingRepoNames.join(", ")}`,
|
||||
);
|
||||
}
|
||||
|
||||
return selectedRepos;
|
||||
}
|
||||
|
||||
export async function resolveGitDir(repoPath: string): Promise<string | undefined> {
|
||||
const dotGitPath = join(repoPath, ".git");
|
||||
|
||||
try {
|
||||
const dotGitStats = await stat(dotGitPath);
|
||||
|
||||
if (dotGitStats.isDirectory()) {
|
||||
return dotGitPath;
|
||||
}
|
||||
|
||||
if (dotGitStats.isFile()) {
|
||||
const pointer = await readFile(dotGitPath, "utf8");
|
||||
const match = pointer.match(/^gitdir:\s*(.+)$/im);
|
||||
|
||||
if (match?.[1]) {
|
||||
return resolve(repoPath, match[1].trim());
|
||||
}
|
||||
}
|
||||
} catch {
|
||||
return undefined;
|
||||
}
|
||||
|
||||
return undefined;
|
||||
}
|
||||
|
||||
function matchesPattern(value: string, pattern: string): boolean {
|
||||
const escaped = pattern.replace(/[.+?^${}()|[\]\\]/g, "\\$&").replace(/\*/g, ".*");
|
||||
const regex = new RegExp(`^${escaped}$`, "i");
|
||||
return regex.test(value);
|
||||
}
|
||||
124
scripts/source-repo-hooks.ts
Normal file
124
scripts/source-repo-hooks.ts
Normal file
|
|
@ -0,0 +1,124 @@
|
|||
import { chmod, mkdir, readFile, stat, writeFile } from "node:fs/promises";
|
||||
import { join } from "node:path";
|
||||
|
||||
import { type SourceRepoMatch } from "./source-repo-discovery";
|
||||
|
||||
export const HOOK_MARKER = "# voidbot-source-index-push-hook";
|
||||
|
||||
export interface SourceHookInstallSummary {
|
||||
installed: number;
|
||||
updated: number;
|
||||
backedUp: number;
|
||||
skipped: number;
|
||||
messages: string[];
|
||||
}
|
||||
|
||||
export async function installSourceIndexPushHooks(
|
||||
voidbotRoot: string,
|
||||
repos: SourceRepoMatch[],
|
||||
): Promise<SourceHookInstallSummary> {
|
||||
let installed = 0;
|
||||
let updated = 0;
|
||||
let backedUp = 0;
|
||||
let skipped = 0;
|
||||
const messages: string[] = [];
|
||||
|
||||
for (const repo of repos) {
|
||||
const hooksDir = join(repo.gitDir, "hooks");
|
||||
const hookPath = join(hooksDir, "pre-push");
|
||||
const backupHookPath = join(hooksDir, "pre-push.voidbot.prev");
|
||||
|
||||
await mkdir(hooksDir, { recursive: true });
|
||||
|
||||
let existingContent: string | undefined;
|
||||
|
||||
try {
|
||||
existingContent = await readFile(hookPath, "utf8");
|
||||
} catch {
|
||||
existingContent = undefined;
|
||||
}
|
||||
|
||||
if (
|
||||
existingContent &&
|
||||
!existingContent.includes(HOOK_MARKER) &&
|
||||
!(await pathExists(backupHookPath))
|
||||
) {
|
||||
await writeFile(backupHookPath, existingContent, "utf8");
|
||||
await chmod(backupHookPath, 0o755);
|
||||
backedUp += 1;
|
||||
} else if (
|
||||
existingContent &&
|
||||
!existingContent.includes(HOOK_MARKER) &&
|
||||
(await pathExists(backupHookPath))
|
||||
) {
|
||||
skipped += 1;
|
||||
messages.push(
|
||||
`Skipping ${repo.repoName}: existing pre-push hook already has a Void backup; not clobbering it.`,
|
||||
);
|
||||
continue;
|
||||
}
|
||||
|
||||
const hookScript = renderHookScript(voidbotRoot, repo.repoName);
|
||||
await writeFile(hookPath, hookScript, "utf8");
|
||||
await chmod(hookPath, 0o755);
|
||||
|
||||
if (existingContent?.includes(HOOK_MARKER)) {
|
||||
updated += 1;
|
||||
messages.push(`Updated pre-push hook for ${repo.repoName}.`);
|
||||
} else {
|
||||
installed += 1;
|
||||
messages.push(`Installed pre-push hook for ${repo.repoName}.`);
|
||||
}
|
||||
}
|
||||
|
||||
return {
|
||||
installed,
|
||||
updated,
|
||||
backedUp,
|
||||
skipped,
|
||||
messages,
|
||||
};
|
||||
}
|
||||
|
||||
function renderHookScript(voidbotRoot: string, repoName: string): string {
|
||||
const normalizedRoot = voidbotRoot.replace(/\\/g, "/");
|
||||
const launcherPath = `${normalizedRoot}/scripts/git-post-push-index.mjs`;
|
||||
const nodePath = process.execPath.replace(/\\/g, "/");
|
||||
const quotedNodePath = escapeForDoubleQuotedShell(nodePath);
|
||||
const quotedLauncherPath = escapeForDoubleQuotedShell(launcherPath);
|
||||
const quotedRoot = escapeForDoubleQuotedShell(normalizedRoot);
|
||||
const quotedRepoName = escapeForDoubleQuotedShell(repoName);
|
||||
|
||||
return [
|
||||
"#!/bin/sh",
|
||||
HOOK_MARKER,
|
||||
"HOOK_DIR=\"$(CDPATH= cd -- \"$(dirname -- \"$0\")\" && pwd)\"",
|
||||
"PREV_HOOK=\"$HOOK_DIR/pre-push.voidbot.prev\"",
|
||||
"",
|
||||
"if [ -x \"$PREV_HOOK\" ]; then",
|
||||
" \"$PREV_HOOK\" \"$@\"",
|
||||
" PREV_STATUS=$?",
|
||||
" if [ \"$PREV_STATUS\" -ne 0 ]; then",
|
||||
" exit \"$PREV_STATUS\"",
|
||||
" fi",
|
||||
"fi",
|
||||
"",
|
||||
`\"${quotedNodePath}\" \"${quotedLauncherPath}\" --voidbot-root \"${quotedRoot}\" --repo-name \"${quotedRepoName}\" >/dev/null 2>&1 || true`,
|
||||
"",
|
||||
"exit 0",
|
||||
"",
|
||||
].join("\n");
|
||||
}
|
||||
|
||||
function escapeForDoubleQuotedShell(value: string): string {
|
||||
return value.replace(/(["`$\\])/g, "\\$1");
|
||||
}
|
||||
|
||||
async function pathExists(path: string): Promise<boolean> {
|
||||
try {
|
||||
await stat(path);
|
||||
return true;
|
||||
} catch {
|
||||
return false;
|
||||
}
|
||||
}
|
||||
111
scripts/source-repo-indexing.ts
Normal file
111
scripts/source-repo-indexing.ts
Normal file
|
|
@ -0,0 +1,111 @@
|
|||
import { type AppConfig } from "@voidbot/config";
|
||||
import {
|
||||
crawlRepositoryDocuments,
|
||||
createTextEmbedder,
|
||||
createVectorStores,
|
||||
FileSourceDocumentArchiveRepository,
|
||||
SourceDocumentIngester,
|
||||
SourceRagPipeline,
|
||||
} from "@voidbot/rag";
|
||||
|
||||
import { type SourceRepoMatch } from "./source-repo-discovery";
|
||||
|
||||
export interface IndexedRepoResult {
|
||||
repoName: string;
|
||||
documents: number;
|
||||
createdDocuments: number;
|
||||
updatedDocuments: number;
|
||||
unchangedDocuments: number;
|
||||
deletedDocuments: number;
|
||||
indexedChunks: number;
|
||||
skippedFiles: number;
|
||||
}
|
||||
|
||||
export interface SourceIndexRunSummary {
|
||||
indexedRepositories: number;
|
||||
totalDocuments: number;
|
||||
totalChunks: number;
|
||||
results: IndexedRepoResult[];
|
||||
}
|
||||
|
||||
export function createSourceVectorStore(config: AppConfig) {
|
||||
const embedder = createTextEmbedder({
|
||||
backend: config.ragEmbeddingBackend,
|
||||
hashDimensions: config.ragEmbeddingDimensions,
|
||||
ollamaBaseUrl: config.ragOllamaBaseUrl,
|
||||
ollamaModel: config.ragOllamaModel,
|
||||
ollamaTimeoutMs: config.ragOllamaTimeoutMs,
|
||||
queryInstruction: config.ragQueryInstruction,
|
||||
});
|
||||
|
||||
return createVectorStores({
|
||||
kind: config.vectorStore.kind,
|
||||
historyPath: config.vectorStore.path,
|
||||
sourceRoot: config.sourceVectorStoreRoot,
|
||||
qdrant: config.qdrant,
|
||||
historyEmbedder: embedder,
|
||||
sourceEmbedder: embedder,
|
||||
}).source;
|
||||
}
|
||||
|
||||
export async function indexSourceRepos(
|
||||
config: AppConfig,
|
||||
repos: SourceRepoMatch[],
|
||||
options: {
|
||||
forceReindex?: boolean;
|
||||
logger?: (line: string) => void;
|
||||
} = {},
|
||||
): Promise<SourceIndexRunSummary> {
|
||||
const vectorStore = createSourceVectorStore(config);
|
||||
const archiveRepository = new FileSourceDocumentArchiveRepository(config.ragSourceArchivePath);
|
||||
const sourceIngester = new SourceDocumentIngester();
|
||||
const pipeline = new SourceRagPipeline(archiveRepository, sourceIngester, vectorStore);
|
||||
const results: IndexedRepoResult[] = [];
|
||||
let indexedRepositories = 0;
|
||||
let totalDocuments = 0;
|
||||
let totalChunks = 0;
|
||||
|
||||
for (const repo of repos) {
|
||||
const scan = await crawlRepositoryDocuments(repo.repoPath, repo.repoName, {
|
||||
includePathPrefixes: config.sourceRepoIncludePrefixes[repo.repoName],
|
||||
});
|
||||
const result = await pipeline.syncRepoDocuments(repo.repoName, scan.documents, {
|
||||
forceReindex: options.forceReindex,
|
||||
});
|
||||
const repoResult: IndexedRepoResult = {
|
||||
repoName: repo.repoName,
|
||||
documents: scan.documents.length,
|
||||
createdDocuments: result.createdDocuments,
|
||||
updatedDocuments: result.updatedDocuments,
|
||||
unchangedDocuments: result.unchangedDocuments,
|
||||
deletedDocuments: result.deletedDocuments,
|
||||
indexedChunks: result.indexedChunks,
|
||||
skippedFiles: scan.skippedFiles.length,
|
||||
};
|
||||
|
||||
results.push(repoResult);
|
||||
indexedRepositories += 1;
|
||||
totalDocuments += scan.documents.length;
|
||||
totalChunks += result.indexedChunks;
|
||||
|
||||
options.logger?.(
|
||||
[
|
||||
`Repo ${repo.repoName}:`,
|
||||
`documents=${repoResult.documents}`,
|
||||
`created=${repoResult.createdDocuments}`,
|
||||
`updated=${repoResult.updatedDocuments}`,
|
||||
`unchanged=${repoResult.unchangedDocuments}`,
|
||||
`deleted=${repoResult.deletedDocuments}`,
|
||||
`indexedChunks=${repoResult.indexedChunks}`,
|
||||
`skippedFiles=${repoResult.skippedFiles}`,
|
||||
].join(" "),
|
||||
);
|
||||
}
|
||||
|
||||
return {
|
||||
indexedRepositories,
|
||||
totalDocuments,
|
||||
totalChunks,
|
||||
results,
|
||||
};
|
||||
}
|
||||
|
|
@ -532,6 +532,16 @@ if ($LASTEXITCODE -ne 0) {
|
|||
|
||||
Stop-StaleRuntimeProcesses -RepoRoot $repoRoot
|
||||
|
||||
$status.stage = "source_reconcile"
|
||||
Write-StatusFile -Path $statusPath -Status $status
|
||||
|
||||
Write-Host "Reconciling source repos..."
|
||||
& $npmCommand.Source run rag:reconcile-sources
|
||||
|
||||
if ($LASTEXITCODE -ne 0) {
|
||||
throw "npm run rag:reconcile-sources failed."
|
||||
}
|
||||
|
||||
$botOut = Join-Path $logDir "bot.log"
|
||||
$botErr = Join-Path $logDir "bot.err.log"
|
||||
$workerOut = Join-Path $logDir "worker.log"
|
||||
|
|
|
|||
|
|
@ -15,3 +15,4 @@
|
|||
{"ts":"2026-05-01T16:47:54Z","type":"research","status":"ok","note":"Expanded the personality roadmap and classifier spec to treat performed selves, contingent-worth defenses, and impression management as first-class structure. Dialogue is not a clean window into the person; concealment and shell-maintenance are themselves part of the signal the system should model."}
|
||||
{"ts":"2026-05-01T16:56:41Z","type":"research","status":"ok","note":"Documented the state-distribution and prompt-projection architecture for Aetheria agents: core variables should live as culture-shaped latent state with history transforms and per-agent perceived overlays, while prompts are temporary projections of perceived state. Volatility, attachment-seeking, and distance-seeking are first-class dimensions in the rebuild, not later garnish."}
|
||||
{"ts":"2026-05-01T17:24:56Z","type":"architecture","status":"ok","note":"Moved deeper personality, culture, prompt-projection, and Aetheria persistent-world design notes into the sibling Ghostlight repo so VoidBot remains a consumer instead of the long-term home for that research."}
|
||||
{"ts":"2026-05-02T11:48:55Z","type":"implementation","status":"ok","note":"Source repo indexing now auto-reconciles during startup and watchdog runs, and the source archive is sharded by repo behind a manifest because the old monolithic source-documents.json hit real size limits and broke incremental writes."}
|
||||
|
|
|
|||
|
|
@ -83,6 +83,8 @@ landed_baseline:
|
|||
- The worker MCP surface is now split into a thin stdio entrypoint plus dedicated bootstrap, resource, tool, Discord-notify, and shared-helper modules instead of one swollen server file.
|
||||
- Interaction-memory logic is now split into event/tone analysis and profile-synthesis modules instead of one slab trying to be both the ear and the therapist.
|
||||
- State storage is now split into a thin factory plus per-domain Postgres/file store modules and bootstrap/migration helpers instead of one persistence omnibus.
|
||||
- Source archive storage is now sharded by repo behind a manifest plus per-repo shard directory instead of one giant `source-documents.json` blob.
|
||||
- Source repo indexing now reconciles automatically during stack startup and watchdog runs, refreshing hooks, indexing newly discovered repos, and pruning stale repo shards when repos disappear from `SOURCE_REPO_ROOT`.
|
||||
|
||||
rejected_paths:
|
||||
- name: transcript_only_continuity
|
||||
|
|
|
|||
Loading…
Reference in a new issue