GameCult/VoidBot
Watch
9
0
Fork
You've already forked VoidBot
 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

Exclude bot prompts from default history retrieval

Browse source
This commit is contained in:
Meta Vens 2026-05-04 09:54:27 +01:00
commit 0ce27b933f
23 changed files with 683 additions and 247 deletions
Download patch file Download diff file Expand all files Collapse all files

1
.env.example
View file

@ -3,6 +3,7 @@ DISCORD_BOT_TOKEN=
DISCORD_APPLICATION_ID=
DISCORD_GUILD_ID=
DISCORD_OWNER_ID=
DISCORD_BOT_TRIGGER_ROLE_IDS=
STATE_STORAGE_BACKEND=postgres
DATABASE_DSN=postgres://voidbot:voidbot@localhost:5432/voidbot
# Leave this as local_json until Qdrant is running and you've migrated the existing vectors.

13
README.md
View file

@ -91,6 +91,12 @@ If you want the remote Ollama generation path too, configure:
If you want a private local persona instead, point `STYLE_PACK_PATH` and `SYSTEM_MESSAGES_PATH` at your own ignored local files.
If you sometimes summon Void through a role mention instead of the bot user mention, set:
- `DISCORD_BOT_TRIGGER_ROLE_IDS`
That lets the history pipeline classify those messages as bot-directed prompts too.
Void usage limits are configurable too:
- `VOID_USAGE_COOLDOWN_SECONDS`
@ -275,6 +281,7 @@ Discord history is live and incremental:
- matching channels are ingested on `MessageCreate`
- edits are re-upserted
- deletes remove indexed chunks
- bot-directed prompts stay in the raw archive but are excluded from default semantic history indexing and lexical fallback, so the bot stops poisoning its own memory with repeated summons
- `/reindex-channel` replays the most recent 100 messages from the current indexed channel
- exported logs can be backfilled later
@ -295,6 +302,12 @@ Useful flags:
- `--no-recursive`
- `--all-channels`
If you need to mark old bot-directed prompts in the archive and prune their history vectors without deleting the raw messages:
```bash
npm run rag:scrub-bot-prompts
```
### Source repos and lore
Source indexing is incremental when you run it. The pipeline crawls the repo, compares the current documents to the archived source snapshot, deletes vectors for changed or removed source IDs, and upserts only the changed chunks.

14
apps/bot/src/discord-bot-handlers.ts
View file

@ -279,6 +279,7 @@ export async function handlePrompt(options: PromptHandlerOptions): Promise<void>
query: options.prompt,
limit: initialRetrievalLimit,
guildId: options.guildContext.guildId,
preserveOverfetch: true,
}),
options.prompt,
).slice(0, initialRetrievalLimit)
@ -492,10 +493,15 @@ export async function handleRejectJob(options: {
export async function handleSearchHistory(
interaction: ChatInputCommandInteraction<CacheType>,
retrievalService: RetrievalService,
archiveRepository: FileMessageArchiveRepository,
systemMessages: SystemMessageCatalog,
): Promise<void> {
const query = interaction.options.getString("query", true);
const results = await retrievalService.searchHistory(query, 5, {
const results = await searchHistoryWithArchiveFallback({
retrievalService,
archiveRepository,
query,
limit: 5,
guildId: interaction.guildId ?? undefined,
});
@ -603,6 +609,8 @@ export async function handleReindexChannel(
channelIndexing: AppConfig["channelIndexing"],
permissionEngine: PermissionEngine,
ragPipeline: RagPipeline,
botUserId: string | undefined,
botRoleIds: string[],
systemMessages: SystemMessageCatalog,
): Promise<void> {
if (!permissionEngine.canReindex(actor)) {
@ -632,7 +640,9 @@ export async function handleReindexChannel(
}
const fetched = await interaction.channel.messages.fetch({ limit: 100 });
const archivedMessages = [...fetched.values()].map(convertDiscordMessageToArchive);
const archivedMessages = [...fetched.values()].map((message) =>
convertDiscordMessageToArchive(message, { botUserId, botRoleIds }),
);
const result = await ragPipeline.upsertMessages(archivedMessages);
await replyEphemeral(

183
apps/bot/src/discord-bot-support.ts
View file

@ -17,11 +17,11 @@ import {
import type { ProviderRegistry } from "@voidbot/providers";
import {
RagPipeline,
FileMessageArchiveRepository,
RetrievalService,
type ArchivedMessageRecord,
} from "@voidbot/rag";
import {
setArchivedMessageKind,
type Actor,
type ArchivedMessage,
type ChannelIndexingPolicy,
@ -35,6 +35,11 @@ import {
shouldIndexChannel,
} from "@voidbot/shared";
export {
filterPromptEchoHistoryResults,
searchHistoryWithArchiveFallback,
} from "@voidbot/rag";
export function buildActorFromMessage(message: Message): Actor {
return {
id: message.author.id,
@ -110,12 +115,16 @@ export async function ingestIfIndexed(
message: Message,
channelIndexing: ChannelIndexingPolicy,
ragPipeline: RagPipeline,
botUserId?: string,
botRoleIds: string[] = [],
): Promise<void> {
if (!shouldIndexChannel(channelIndexing, buildChannelIndexingTarget(message.channel))) {
return;
}
await ragPipeline.upsertMessages([convertDiscordMessageToArchive(message)]);
await ragPipeline.upsertMessages([
convertDiscordMessageToArchive(message, { botUserId, botRoleIds }),
]);
}
export async function rememberAmbientVoidReference(
@ -161,7 +170,13 @@ export function buildChannelIndexingTarget(
return target;
}
export function convertDiscordMessageToArchive(message: Message): ArchivedMessage {
export function convertDiscordMessageToArchive(
message: Message,
options?: {
botUserId?: string;
botRoleIds?: string[];
},
): ArchivedMessage {
const metadata: Record<string, string> = {
jumpUrl: message.url,
};
@ -196,7 +211,16 @@ export function convertDiscordMessageToArchive(message: Message): ArchivedMessag
(attachment) => attachment.url ?? attachment.proxyURL ?? attachment.name,
)
: undefined,
metadata,
metadata: setArchivedMessageKind(
metadata,
isBotDirectedPromptMessage(
message,
options?.botUserId,
options?.botRoleIds ?? [],
)
? "bot_prompt"
: "default",
),
};
}
@ -339,51 +363,6 @@ export function renderRateLimitMessage(
});
}
export async function searchHistoryWithArchiveFallback(options: {
retrievalService: RetrievalService;
archiveRepository: FileMessageArchiveRepository;
query: string;
limit: number;
guildId?: string;
channelId?: string;
authorId?: string;
}): Promise<RetrievalResult[]> {
const overfetchLimit = Math.max(options.limit * 3, options.limit + 4);
const vectorResults = await options.retrievalService.searchHistory(options.query, overfetchLimit, {
guildId: options.guildId,
channelId: options.channelId,
authorId: options.authorId,
});
const lexicalResults = await options.archiveRepository.searchLexical(options.query, overfetchLimit, {
guildId: options.guildId,
channelId: options.channelId,
authorId: options.authorId,
});
if (lexicalResults.length === 0) {
return vectorResults;
}
const merged = [...vectorResults];
const seenSourceIds = new Set(merged.map((result) => result.sourceId));
for (const result of lexicalResults) {
if (seenSourceIds.has(result.sourceId)) {
continue;
}
merged.push(result);
seenSourceIds.add(result.sourceId);
if (merged.length >= overfetchLimit) {
break;
}
}
return merged;
}
export function formatHistoryResults(
results: Awaited<ReturnType<RetrievalService["searchHistory"]>>,
): Array<Record<string, unknown>> {
@ -420,96 +399,6 @@ export function formatSourceResults(
}));
}
export function filterPromptEchoHistoryResults<
T extends {
text: string;
},
>(results: T[], query: string): T[] {
const normalizedQuery = normalizeHistoryEchoText(query);
const queryTokens = buildHistoryEchoTokens(normalizedQuery);
if (normalizedQuery.length === 0) {
return results;
}
const filtered = results.filter((result) => {
const normalizedResult = normalizeHistoryEchoText(result.text);
if (normalizedResult.length === 0) {
return true;
}
if (normalizedResult === normalizedQuery) {
return false;
}
if (queryTokens.length >= 4) {
const resultTokens = buildHistoryEchoTokens(normalizedResult);
const sharedTokenCount = countSharedTokens(queryTokens, resultTokens);
const similarity = sharedTokenCount / Math.max(Math.min(queryTokens.length, resultTokens.length), 1);
if (similarity >= 0.7) {
return false;
}
}
if (
normalizedQuery.length >= 48 &&
(normalizedResult.includes(normalizedQuery) || normalizedQuery.includes(normalizedResult))
) {
return false;
}
return true;
});
return filtered.length > 0 ? filtered : results;
}
function normalizeHistoryEchoText(value: string): string {
return value
.replace(/<@!?\d+>/g, " ")
.replace(/<@&\d+>/g, " ")
.replace(/\s+/g, " ")
.trim()
.toLowerCase();
}
function buildHistoryEchoTokens(value: string): string[] {
return [...new Set(
value
.split(/[^\p{L}\p{N}]+/u)
.map((token) => token.trim())
.filter((token) => token.length >= 4)
.filter((token) => !HISTORY_ECHO_STOP_WORDS.has(token)),
)];
}
function countSharedTokens(left: string[], right: string[]): number {
const rightSet = new Set(right);
return left.reduce((count, token) => count + (rightSet.has(token) ? 1 : 0), 0);
}
const HISTORY_ECHO_STOP_WORDS = new Set([
"about",
"after",
"before",
"could",
"from",
"have",
"please",
"pretty",
"tell",
"that",
"them",
"they",
"this",
"what",
"with",
"would",
"your",
]);
export function formatArchivedMessageContext(
message: ArchivedMessageRecord,
anchorMessageId: string,
@ -972,3 +861,19 @@ function isAmbientVoidReference(
return true;
}
function isBotDirectedPromptMessage(
message: Message,
botUserId: string | undefined,
botRoleIds: string[],
): boolean {
if (!message.inGuild()) {
return true;
}
if (botUserId && message.mentions.has(botUserId)) {
return true;
}
return botRoleIds.some((roleId) => message.mentions.roles.has(roleId));
}

71
apps/bot/src/discord-bot.ts
View file

@ -53,6 +53,7 @@ import {
buildChannelIndexingTarget,
buildGuildContextFromInteraction,
buildGuildContextFromMessage,
filterPromptEchoHistoryResults,
formatArchivedMessageContext,
formatHistoryResults,
formatProviderStatuses,
@ -69,50 +70,6 @@ import {
stripBotMention,
} from "./discord-bot-support";
function filterPromptEchoHistoryResults<
T extends {
text: string;
},
>(results: T[], query: string): T[] {
const normalizedQuery = normalizeHistoryEchoText(query);
if (normalizedQuery.length === 0) {
return results;
}
const filtered = results.filter((result) => {
const normalizedResult = normalizeHistoryEchoText(result.text);
if (normalizedResult.length === 0) {
return true;
}
if (normalizedResult === normalizedQuery) {
return false;
}
if (
normalizedQuery.length >= 48 &&
(normalizedResult.includes(normalizedQuery) || normalizedQuery.includes(normalizedResult))
) {
return false;
}
return true;
});
return filtered.length > 0 ? filtered : results;
}
function normalizeHistoryEchoText(value: string): string {
return value
.replace(/<@!?\d+>/g, " ")
.replace(/<@&\d+>/g, " ")
.replace(/\s+/g, " ")
.trim()
.toLowerCase();
}
export async function startBot(): Promise<void> {
const config = loadConfig();
@ -244,6 +201,7 @@ export async function startBot(): Promise<void> {
guildId: input.guildId,
channelId: input.channelId,
authorId: input.authorId,
preserveOverfetch: true,
});
const results = filterPromptEchoHistoryResults(rawResults, input.query).slice(0, input.limit);
@ -333,7 +291,13 @@ export async function startBot(): Promise<void> {
const isDirectMessage = !message.inGuild();
await ingestIfIndexed(message, config.channelIndexing, ragPipeline);
await ingestIfIndexed(
message,
config.channelIndexing,
ragPipeline,
client.user?.id,
config.botTriggerRoleIds,
);
await rememberAmbientVoidReference(message, client.user?.id, interactionMemory);
if (!client.user || (!isDirectMessage && !message.mentions.has(client.user))) {
@ -403,7 +367,13 @@ export async function startBot(): Promise<void> {
return;
}
await ingestIfIndexed(materializedMessage, config.channelIndexing, ragPipeline);
await ingestIfIndexed(
materializedMessage,
config.channelIndexing,
ragPipeline,
client.user?.id,
config.botTriggerRoleIds,
);
await rememberAmbientVoidReference(
materializedMessage,
client.user?.id,
@ -535,7 +505,12 @@ export async function startBot(): Promise<void> {
});
break;
case "search-history":
await handleSearchHistory(interaction, retrievalService, activeSystemMessages);
await handleSearchHistory(
interaction,
retrievalService,
archiveRepository,
activeSystemMessages,
);
break;
case "summarize-channel":
await handleSummarizeChannel(interaction, activeSystemMessages);
@ -547,6 +522,8 @@ export async function startBot(): Promise<void> {
config.channelIndexing,
permissionEngine,
ragPipeline,
client.user?.id,
config.botTriggerRoleIds,
activeSystemMessages,
);
break;

53
apps/worker/src/index.ts
View file

@ -19,11 +19,13 @@ import {
ProviderRegistry,
} from "@voidbot/providers";
import {
filterPromptEchoHistoryResults,
createTextEmbedder,
createVectorStores,
FileMessageArchiveRepository,
FileSourceDocumentArchiveRepository,
RetrievalService,
searchHistoryWithArchiveFallback,
SourceDocumentIngester,
type ArchivedMessageRecord,
} from "@voidbot/rag";
@ -35,50 +37,6 @@ import {
const config = loadConfig();
function filterPromptEchoHistoryResults<
T extends {
text: string;
},
>(results: T[], query: string): T[] {
const normalizedQuery = normalizeHistoryEchoText(query);
if (normalizedQuery.length === 0) {
return results;
}
const filtered = results.filter((result) => {
const normalizedResult = normalizeHistoryEchoText(result.text);
if (normalizedResult.length === 0) {
return true;
}
if (normalizedResult === normalizedQuery) {
return false;
}
if (
normalizedQuery.length >= 48 &&
(normalizedResult.includes(normalizedQuery) || normalizedQuery.includes(normalizedResult))
) {
return false;
}
return true;
});
return filtered.length > 0 ? filtered : results;
}
function normalizeHistoryEchoText(value: string): string {
return value
.replace(/<@!?\d+>/g, " ")
.replace(/<@&\d+>/g, " ")
.replace(/\s+/g, " ")
.trim()
.toLowerCase();
}
const embedder = createTextEmbedder({
backend: config.ragEmbeddingBackend,
hashDimensions: config.ragEmbeddingDimensions,
@ -214,10 +172,15 @@ async function buildProviderRegistry(systemMessagesCatalog: SystemMessageCatalog
};
},
searchHistory: async (input) => {
const rawResults = await retrievalService.searchHistory(input.query, input.limit, {
const rawResults = await searchHistoryWithArchiveFallback({
retrievalService,
archiveRepository,
query: input.query,
limit: input.limit,
guildId: input.guildId,
channelId: input.channelId,
authorId: input.authorId,
preserveOverfetch: true,
});
const results = filterPromptEchoHistoryResults(rawResults, input.query);

9
apps/worker/src/mcp-server-resources.ts
View file

@ -1,6 +1,7 @@
import { McpServer, ResourceTemplate } from "@modelcontextprotocol/sdk/server/mcp.js";
import { DEFAULT_RETRIEVAL_RESULT_LIMIT } from "@voidbot/shared";
import { searchHistoryWithArchiveFallback } from "@voidbot/rag";
import { type VoidbotMcpContext } from "./mcp-server-context";
import {
@ -87,10 +88,12 @@ export function registerVoidbotResources(
},
async (uri, variables) => {
const query = getRequiredVariable(variables, "query", "history search");
const results = await context.retrievalService.searchHistory(
const results = await searchHistoryWithArchiveFallback({
retrievalService: context.retrievalService,
archiveRepository: context.archiveRepository,
query,
DEFAULT_RETRIEVAL_RESULT_LIMIT,
);
limit: DEFAULT_RETRIEVAL_RESULT_LIMIT,
});
return jsonResource(uri, {
query,

7
apps/worker/src/mcp-server-tools.ts
View file

@ -2,6 +2,7 @@ import { McpServer } from "@modelcontextprotocol/sdk/server/mcp.js";
import { type CallToolResult } from "@modelcontextprotocol/sdk/types.js";
import { DEFAULT_RETRIEVAL_RESULT_LIMIT } from "@voidbot/shared";
import { searchHistoryWithArchiveFallback } from "@voidbot/rag";
import { type VoidbotMcpContext } from "./mcp-server-context";
import { openOwnerDmChannel, postDiscordMessage } from "./mcp-server-discord";
@ -45,7 +46,11 @@ export function registerVoidbotTools(
},
async (input: SearchHistoryArgs): Promise<CallToolResult> => {
const { query, limit, guildId, channelId, authorId } = input;
const results = await context.retrievalService.searchHistory(query, limit ?? DEFAULT_RETRIEVAL_RESULT_LIMIT, {
const results = await searchHistoryWithArchiveFallback({
retrievalService: context.retrievalService,
archiveRepository: context.archiveRepository,
query,
limit: limit ?? DEFAULT_RETRIEVAL_RESULT_LIMIT,
guildId,
channelId,
authorId,

1
notes/fresh-workspace-handoff.md
View file

@ -33,6 +33,7 @@ Git history and smoke artifacts carry routine proof. `state/evidence.jsonl` shou
- The stack has a dedicated logon startup task path now, so a reboot does not have to mean manual necromancy for bot and worker.
- The operations watchdog now supports ignored local extension checks, so adjacent-service fire signals can ride the same owner DM alert path without hardcoding private project gossip into the public repo.
- Source repo discovery now reconciles automatically during `npm run stack:start` and through the watchdog path, so new repos under `SOURCE_REPO_ROOT` stop waiting for manual hook folklore before they get indexed.
- Bot-directed prompts now stay in the raw Discord archive but are excluded from default semantic history indexing and lexical fallback, so repeated summons stop teaching the bot its own call-and-response loops.
- Source-grounding hints are advisory now; the owner and local lanes can decide when repo or lore tool use is actually warranted instead of obeying a lexical front-door veto.
- The new state/notes surfaces are now the continuity spine for future nontrivial work.

13
notes/voidbot-current-system-map.md
View file

@ -65,7 +65,7 @@ This note is the source-grounded description of how the live VoidBot stack is sh
- `packages/rag/src/qdrant-vector-store.ts`
- Qdrant vector persistence for history and source collections.
- `packages/rag/src/message-archive.ts`
- archived Discord message store under `.voidbot/rag/messages.json`.
- archived Discord message store under `.voidbot/rag/messages.json`, including bot-directed prompts that stay preserved for forensics even when they are excluded from default history retrieval.
- `packages/rag/src/source-document-archive.ts`
- archived source/lore document manifest under `.voidbot/rag/source-documents.json` plus per-repo shards under `.voidbot/rag/source-documents.repos/`.
@ -95,11 +95,12 @@ This note is the source-grounded description of how the live VoidBot stack is sh
## Flow 3: Retrieval And Indexing
1. `packages/rag/src/message-archive.ts` and `packages/rag/src/source-document-archive.ts` keep the raw corpora.
2. `packages/rag/src/retrieval-service.ts` translates history/source queries into vector lookups plus metadata filters.
3. `packages/rag/src/qdrant-vector-store.ts` executes the live vector lookups against separate history and source collections.
4. `scripts/reconcile-source-repos.ts` discovers local Git repos, refreshes push hooks, prunes stale repo shards, and indexes newly discovered repos.
5. `scripts/index-source-repos.ts` and `scripts/git-post-push-index.mjs` drive explicit or detached per-repo source/lore reindex work.
6. `apps/worker/src/mcp-server.ts` boots the MCP lane, while `mcp-server-resources.ts` and `mcp-server-tools.ts` expose retrieval to Codex and other sessions through `search_history`, `get_message_context`, `search_sources`, `get_source_context`, and `list_indexed_repos`.
2. Bot-directed prompts are tagged at ingest, kept in the raw Discord archive, and deliberately skipped when the history ingester builds semantic chunks so repeated summons stop poisoning normal retrieval.
3. `packages/rag/src/retrieval-service.ts` translates history/source queries into vector lookups plus metadata filters.
4. `packages/rag/src/qdrant-vector-store.ts` executes the live vector lookups against separate history and source collections.
5. `scripts/reconcile-source-repos.ts` discovers local Git repos, refreshes push hooks, prunes stale repo shards, and indexes newly discovered repos.
6. `scripts/index-source-repos.ts` and `scripts/git-post-push-index.mjs` drive explicit or detached per-repo source/lore reindex work.
7. `apps/worker/src/mcp-server.ts` boots the MCP lane, while `mcp-server-resources.ts` and `mcp-server-tools.ts` expose retrieval to Codex and other sessions through `search_history`, `get_message_context`, `search_sources`, `get_source_context`, and `list_indexed_repos`.
## Flow 4: Ops And Recovery

1
package.json
View file

@ -31,6 +31,7 @@
"qdrant:setup": "powershell -ExecutionPolicy Bypass -File scripts/setup-docker-qdrant.ps1",
"qdrant:up": "powershell -ExecutionPolicy Bypass -File scripts/start-qdrant.ps1",
"rag:backfill": "tsx scripts/backfill-discord-logs.ts",
"rag:scrub-bot-prompts": "tsx scripts/scrub-bot-prompts-from-history.ts",
"rag:index-sources": "tsx scripts/index-source-repos.ts",
"rag:install-push-hooks": "tsx scripts/install-source-index-post-push-hooks.ts",
"rag:reconcile-sources": "tsx scripts/reconcile-source-repos.ts",

3
packages/config/src/index.ts
View file

@ -42,6 +42,7 @@ const envSchema = z.object({
DISCORD_APPLICATION_ID: optionalNonEmptyString,
DISCORD_GUILD_ID: optionalNonEmptyString,
DISCORD_OWNER_ID: z.string().min(1, "DISCORD_OWNER_ID is required"),
DISCORD_BOT_TRIGGER_ROLE_IDS: z.string().default(""),
STATE_STORAGE_BACKEND: z.enum(["file", "postgres"]).default("postgres"),
DATABASE_DSN: z.string().min(1).default("postgres://voidbot:voidbot@localhost:5432/voidbot"),
VECTOR_STORE_KIND: z.enum(["local_json", "qdrant"]).default("local_json"),
@ -112,6 +113,7 @@ export interface AppConfig {
applicationId?: string;
developmentGuildId?: string;
ownerDiscordId: string;
botTriggerRoleIds: string[];
stateStorageBackend: "file" | "postgres";
databaseDsn: string;
enabledProviders: ProviderName[];
@ -283,6 +285,7 @@ export function loadConfig(env: NodeJS.ProcessEnv = process.env): AppConfig {
applicationId: parsed.DISCORD_APPLICATION_ID,
developmentGuildId: parsed.DISCORD_GUILD_ID,
ownerDiscordId: parsed.DISCORD_OWNER_ID,
botTriggerRoleIds: parseList(parsed.DISCORD_BOT_TRIGGER_ROLE_IDS),
stateStorageBackend: parsed.STATE_STORAGE_BACKEND,
databaseDsn: parsed.DATABASE_DSN,
enabledProviders: parseProviders(parsed.ENABLED_PROVIDERS),

17
packages/rag/src/discord-log-importer.ts
View file

@ -250,6 +250,7 @@ function normalizeArchivedMessage(
readString(candidate, ["parentChannel", "name"]) ??
readString(candidate, ["parent", "name"]);
const metadata: Record<string, string> = {
...readStringRecord(candidate.metadata),
sourceFile: basename(sourceFile),
};
@ -338,6 +339,22 @@ function normalizeTimestamp(value: string): string {
return Number.isNaN(parsed.getTime()) ? value : parsed.toISOString();
}
function readStringRecord(value: unknown): Record<string, string> {
if (!isRecord(value)) {
return {};
}
const normalized: Record<string, string> = {};
for (const [key, entry] of Object.entries(value)) {
if (typeof entry === "string") {
normalized[key] = entry;
}
}
return normalized;
}
function readString(source: Record<string, unknown>, path: string[]): string | undefined {
let cursor: unknown = source;

7
packages/rag/src/history-ingester.ts
View file

@ -2,6 +2,8 @@ import {
type ArchivedMessage,
type EmbeddingChunk,
type VectorStore,
readArchivedMessageKind,
isArchivedBotPrompt,
} from "@voidbot/shared";
const DEFAULT_CHUNK_SIZE = 420;
@ -23,6 +25,10 @@ export class HistoryIngester {
}
public chunkMessage(message: ArchivedMessage): EmbeddingChunk[] {
if (isArchivedBotPrompt(message)) {
return [];
}
const segments = splitText(message.content, this.chunkSize);
return segments.map((segment, index) => ({
@ -42,6 +48,7 @@ export class HistoryIngester {
chunkIndex: String(index),
threadId: message.threadId ?? "",
editedAt: message.editedAt ?? "",
messageKind: readArchivedMessageKind(message),
},
}));
}

164
packages/rag/src/history-retrieval.ts Normal file
View file

@ -0,0 +1,164 @@
import type { RetrievalFilters, RetrievalResult } from "@voidbot/shared";
import { FileMessageArchiveRepository } from "./message-archive";
import { RetrievalService } from "./retrieval-service";
const HISTORY_ECHO_STOP_WORDS = new Set([
"about",
"after",
"before",
"could",
"from",
"have",
"please",
"pretty",
"tell",
"that",
"them",
"they",
"this",
"what",
"with",
"would",
"your",
]);
interface HistorySearchWithArchiveFallbackOptions {
retrievalService: RetrievalService;
archiveRepository: FileMessageArchiveRepository;
query: string;
limit: number;
guildId?: string;
channelId?: string;
authorId?: string;
includeBotPrompts?: boolean;
preserveOverfetch?: boolean;
}
export async function searchHistoryWithArchiveFallback(
options: HistorySearchWithArchiveFallbackOptions,
): Promise<RetrievalResult[]> {
const overfetchLimit = Math.max(options.limit * 3, options.limit + 4);
const vectorResults = await options.retrievalService.searchHistory(
options.query,
overfetchLimit,
{
guildId: options.guildId,
channelId: options.channelId,
authorId: options.authorId,
},
);
const lexicalResults = await options.archiveRepository.searchLexical(
options.query,
overfetchLimit,
{
guildId: options.guildId,
channelId: options.channelId,
authorId: options.authorId,
includeBotPrompts: options.includeBotPrompts ?? false,
},
);
if (lexicalResults.length === 0) {
return options.preserveOverfetch
? vectorResults
: vectorResults.slice(0, options.limit);
}
const merged = [...vectorResults];
const seenSourceIds = new Set(merged.map((result) => result.sourceId));
for (const result of lexicalResults) {
if (seenSourceIds.has(result.sourceId)) {
continue;
}
merged.push(result);
seenSourceIds.add(result.sourceId);
if (merged.length >= overfetchLimit) {
break;
}
}
return options.preserveOverfetch ? merged : merged.slice(0, options.limit);
}
export function filterPromptEchoHistoryResults<
T extends {
text: string;
},
>(results: T[], query: string): T[] {
const normalizedQuery = normalizeHistoryEchoText(query);
const queryTokens = buildHistoryEchoTokens(normalizedQuery);
if (normalizedQuery.length === 0) {
return results;
}
const filtered = results.filter((result) => {
const normalizedResult = normalizeHistoryEchoText(result.text);
if (normalizedResult.length === 0) {
return true;
}
if (normalizedResult === normalizedQuery) {
return false;
}
if (queryTokens.length >= 4) {
const resultTokens = buildHistoryEchoTokens(normalizedResult);
const sharedTokenCount = countSharedTokens(queryTokens, resultTokens);
const similarity =
sharedTokenCount /
Math.max(Math.min(queryTokens.length, resultTokens.length), 1);
if (similarity >= 0.7) {
return false;
}
}
if (
normalizedQuery.length >= 48 &&
(normalizedResult.includes(normalizedQuery) ||
normalizedQuery.includes(normalizedResult))
) {
return false;
}
return true;
});
return filtered.length > 0 ? filtered : results;
}
function normalizeHistoryEchoText(value: string): string {
return value
.replace(/<@!?\d+>/g, " ")
.replace(/<@&\d+>/g, " ")
.replace(/\s+/g, " ")
.trim()
.toLowerCase();
}
function buildHistoryEchoTokens(value: string): string[] {
return [
...new Set(
value
.split(/[^\p{L}\p{N}]+/u)
.map((token) => token.trim())
.filter((token) => token.length >= 4)
.filter((token) => !HISTORY_ECHO_STOP_WORDS.has(token)),
),
];
}
function countSharedTokens(left: string[], right: string[]): number {
const rightSet = new Set(right);
return left.reduce(
(count, token) => count + (rightSet.has(token) ? 1 : 0),
0,
);
}

1
packages/rag/src/index.ts
View file

@ -1,5 +1,6 @@
export * from "./embedder-factory";
export * from "./history-ingester";
export * from "./history-retrieval";
export * from "./discord-log-importer";
export * from "./file-store";
export * from "./file-vector-store";

18
packages/rag/src/message-archive.ts
View file

@ -2,6 +2,8 @@ import {
type ArchivedMessage,
type RetrievalFilters,
type RetrievalResult,
isArchivedBotPrompt,
readArchivedMessageKind,
} from "@voidbot/shared";
import { normalizeText } from "./history-ingester";
@ -46,6 +48,11 @@ export interface MessageArchiveMutationResult {
changedMessageIds: string[];
}
interface HistoryArchiveFilters
extends Pick<RetrievalFilters, "guildId" | "channelId" | "authorId"> {
includeBotPrompts?: boolean;
}
export class FileMessageArchiveRepository {
private readonly store: SerializedFileStore<MessageArchiveStore>;
@ -186,7 +193,7 @@ export class FileMessageArchiveRepository {
public async searchLexical(
query: string,
limit = 5,
filters?: Pick<RetrievalFilters, "guildId" | "channelId" | "authorId">,
filters?: HistoryArchiveFilters,
): Promise<RetrievalResult[]> {
const normalizedQuery = normalizeSearchText(query);
const queryTokens = extractSearchTokens(normalizedQuery);
@ -265,10 +272,14 @@ function belongsToSameConversation(
function matchesHistoryFilters(
message: ArchivedMessageRecord,
filters?: Pick<RetrievalFilters, "guildId" | "channelId" | "authorId">,
filters?: HistoryArchiveFilters,
): boolean {
if (!filters) {
return true;
return !isArchivedBotPrompt(message);
}
if (!filters.includeBotPrompts && isArchivedBotPrompt(message)) {
return false;
}
if (filters.guildId && message.guildId !== filters.guildId) {
@ -350,6 +361,7 @@ function toRetrievalResult(message: ArchivedMessageRecord, score: number): Retri
threadId: message.threadId ?? "",
jumpUrl: message.metadata?.jumpUrl ?? "",
corpusKind: "discord_history",
messageKind: readArchivedMessageKind(message),
},
};
}

122
packages/shared/src/history-message-kind.ts Normal file
View file

@ -0,0 +1,122 @@
export const archivedMessageKinds = ["default", "bot_prompt"] as const;
export type ArchivedMessageKind = (typeof archivedMessageKinds)[number];
export const ARCHIVED_MESSAGE_KIND_METADATA_KEY = "messageKind";
interface MetadataCarrier {
metadata?: Record<string, string>;
}
interface BotPromptMessageLike extends MetadataCarrier {
content: string;
}
export function readArchivedMessageKind(
message: MetadataCarrier | undefined,
): ArchivedMessageKind {
return message?.metadata?.[ARCHIVED_MESSAGE_KIND_METADATA_KEY] === "bot_prompt"
? "bot_prompt"
: "default";
}
export function isArchivedBotPrompt(
message: MetadataCarrier | undefined,
): boolean {
return readArchivedMessageKind(message) === "bot_prompt";
}
export function setArchivedMessageKind(
metadata: Record<string, string> | undefined,
kind: ArchivedMessageKind,
): Record<string, string> | undefined {
const nextMetadata = { ...(metadata ?? {}) };
if (kind === "default") {
delete nextMetadata[ARCHIVED_MESSAGE_KIND_METADATA_KEY];
return Object.keys(nextMetadata).length > 0 ? nextMetadata : undefined;
}
nextMetadata[ARCHIVED_MESSAGE_KIND_METADATA_KEY] = kind;
return nextMetadata;
}
export function contentMentionsDiscordUser(
content: string,
userId: string | undefined,
): boolean {
if (!userId || userId.trim().length === 0) {
return false;
}
const escapedId = escapeRegExp(userId.trim());
return new RegExp(`<@!?${escapedId}>`).test(content);
}
export function contentMentionsDiscordRole(
content: string,
roleId: string | undefined,
): boolean {
if (!roleId || roleId.trim().length === 0) {
return false;
}
const escapedId = escapeRegExp(roleId.trim());
return new RegExp(`<@&${escapedId}>`).test(content);
}
export function isDiscordBotPromptContent(
content: string,
options: {
botUserId?: string;
botRoleIds?: string[];
} = {},
): boolean {
if (contentMentionsDiscordUser(content, options.botUserId)) {
return true;
}
return (options.botRoleIds ?? []).some((roleId) =>
contentMentionsDiscordRole(content, roleId),
);
}
export function classifyArchivedMessageKind(
message: BotPromptMessageLike,
options: {
botUserId?: string;
botRoleIds?: string[];
} = {},
): ArchivedMessageKind {
if (isArchivedBotPrompt(message)) {
return "bot_prompt";
}
return isDiscordBotPromptContent(message.content, options)
? "bot_prompt"
: "default";
}
export function applyArchivedMessageKind<T extends BotPromptMessageLike>(
message: T,
options: {
botUserId?: string;
botRoleIds?: string[];
} = {},
): T {
const kind = classifyArchivedMessageKind(message, options);
const nextMetadata = setArchivedMessageKind(message.metadata, kind);
const currentKind = readArchivedMessageKind(message);
if (kind === currentKind && nextMetadata === message.metadata) {
return message;
}
return {
...message,
metadata: nextMetadata,
};
}
function escapeRegExp(value: string): string {
return value.replace(/[.*+?^${}()|[\]\\]/g, "\\$&");
}

1
packages/shared/src/index.ts
View file

@ -1,4 +1,5 @@
export * from "./channel-indexing";
export * from "./history-message-kind";
export const commandNames = [
"ask",

15
scripts/backfill-discord-logs.ts
View file

@ -4,6 +4,8 @@ import { resolve } from "node:path";
import { loadConfig } from "@voidbot/config";
import {
applyArchivedMessageKind,
isArchivedBotPrompt,
createTextEmbedder,
createVectorStores,
FileImportStateRepository,
@ -60,7 +62,13 @@ async function main(): Promise<void> {
channelIndexing: options.respectIndexingRules ? config.channelIndexing : undefined,
previousState,
});
const ingestResult = await ragPipeline.upsertMessages(importResult.messages);
const normalizedMessages = importResult.messages.map((message) =>
applyArchivedMessageKind(message, {
botUserId: config.applicationId,
botRoleIds: config.botTriggerRoleIds,
}),
);
const ingestResult = await ragPipeline.upsertMessages(normalizedMessages);
await importStateRepository.write(importResult.nextState);
@ -69,6 +77,11 @@ async function main(): Promise<void> {
console.log(`Files imported: ${importResult.filesImported}`);
console.log(`Files skipped (unchanged): ${importResult.filesSkipped}`);
console.log(`Messages discovered: ${importResult.messages.length}`);
console.log(
`Bot-directed prompts archived but skipped for semantic history: ${
normalizedMessages.filter((message) => isArchivedBotPrompt(message)).length
}`,
);
console.log(`Invalid records: ${importResult.invalidRecords}`);
console.log(
`Archive changes: created=${ingestResult.createdMessages}, updated=${ingestResult.updatedMessages}, unchanged=${ingestResult.unchangedMessages}`,

214
scripts/scrub-bot-prompts-from-history.ts Normal file
View file

@ -0,0 +1,214 @@
import "dotenv/config";
import { readFile } from "node:fs/promises";
import { Pool } from "pg";
import { loadConfig } from "@voidbot/config";
import {
applyArchivedMessageKind,
isArchivedBotPrompt,
setArchivedMessageKind,
type InteractionMemoryProfile,
} from "@voidbot/shared";
import {
createTextEmbedder,
createVectorStores,
FileMessageArchiveRepository,
HistoryIngester,
RagPipeline,
} from "@voidbot/rag";
interface CliOptions {
dryRun: boolean;
botUserId?: string;
botRoleIds: string[];
}
async function main(): Promise<void> {
if (!process.env.DISCORD_OWNER_ID) {
process.env.DISCORD_OWNER_ID = "__rag_scrub__";
}
const config = loadConfig();
const options = parseArgs(process.argv.slice(2));
const botUserId = options.botUserId ?? config.applicationId;
const botRoleIds = [...new Set([...config.botTriggerRoleIds, ...options.botRoleIds])];
if (!botUserId && botRoleIds.length === 0) {
throw new Error(
"Provide --bot-user-id or configure DISCORD_APPLICATION_ID / DISCORD_BOT_TRIGGER_ROLE_IDS before scrubbing bot prompts.",
);
}
const archiveRepository = new FileMessageArchiveRepository(config.ragArchivePath);
const messages = await archiveRepository.listAllActive();
const recordedPromptMessageIds = await collectRecordedPromptMessageIds(config);
const alreadyTagged = messages.filter((message) => isArchivedBotPrompt(message)).length;
const retaggedMessages = messages.map((message) => {
if (recordedPromptMessageIds.has(message.id)) {
return {
...message,
metadata: setArchivedMessageKind(message.metadata, "bot_prompt"),
};
}
return applyArchivedMessageKind(message, { botUserId, botRoleIds });
});
const newlyTaggedMessages = retaggedMessages.filter(
(message, index) =>
!isArchivedBotPrompt(messages[index]) && isArchivedBotPrompt(message),
);
console.log(`Active archived messages: ${messages.length}`);
console.log(`Recorded direct-prompt message ids: ${recordedPromptMessageIds.size}`);
console.log(`Already tagged bot prompts: ${alreadyTagged}`);
console.log(`Newly detected bot prompts: ${newlyTaggedMessages.length}`);
if (newlyTaggedMessages.length === 0) {
return;
}
if (options.dryRun) {
console.log("Dry run only. No archive rows or vectors were changed.");
return;
}
const embedder = createTextEmbedder({
backend: config.ragEmbeddingBackend,
hashDimensions: config.ragEmbeddingDimensions,
ollamaBaseUrl: config.ragOllamaBaseUrl,
ollamaModel: config.ragOllamaModel,
ollamaTimeoutMs: config.ragOllamaTimeoutMs,
queryInstruction: config.ragQueryInstruction,
});
const vectorStore = createVectorStores({
kind: config.vectorStore.kind,
historyPath: config.vectorStore.path,
sourceRoot: config.sourceVectorStoreRoot,
qdrant: config.qdrant,
historyEmbedder: embedder,
sourceEmbedder: embedder,
}).history;
const ragPipeline = new RagPipeline(
archiveRepository,
new HistoryIngester(),
vectorStore,
);
const result = await ragPipeline.upsertMessages(newlyTaggedMessages);
console.log(
`Archive changes: created=${result.createdMessages}, updated=${result.updatedMessages}, unchanged=${result.unchangedMessages}`,
);
console.log(
`History chunks still indexed for those rows after scrub: ${result.indexedChunks}`,
);
}
function parseArgs(args: string[]): CliOptions {
const options: CliOptions = {
dryRun: false,
botRoleIds: [],
};
for (let index = 0; index < args.length; index += 1) {
const argument = args[index];
if (argument === "--dry-run") {
options.dryRun = true;
continue;
}
if (argument === "--bot-user-id") {
options.botUserId = args[index + 1];
index += 1;
continue;
}
if (argument === "--bot-role-id") {
const roleId = args[index + 1];
if (roleId) {
options.botRoleIds.push(roleId);
}
index += 1;
continue;
}
throw new Error(`Unknown argument: ${argument}`);
}
return options;
}
void main().catch((error) => {
console.error(error);
process.exitCode = 1;
});
async function collectRecordedPromptMessageIds(
config: ReturnType<typeof loadConfig>,
): Promise<Set<string>> {
if (config.stateStorageBackend === "postgres") {
return collectRecordedPromptMessageIdsFromPostgres(config.databaseDsn);
}
return collectRecordedPromptMessageIdsFromFile(config.interactionMemoryFile);
}
async function collectRecordedPromptMessageIdsFromPostgres(
databaseDsn: string,
): Promise<Set<string>> {
const pool = new Pool({ connectionString: databaseDsn });
try {
const result = await pool.query<{ id: string }>(
`select id
from interaction_memory_events
where source_kind = 'direct_prompt'`,
);
return new Set(
result.rows
.map((row) => row.id)
.filter((id): id is string => typeof id === "string" && id.length > 0),
);
} finally {
await pool.end();
}
}
async function collectRecordedPromptMessageIdsFromFile(
interactionMemoryFile: string,
): Promise<Set<string>> {
try {
const raw = await readFile(interactionMemoryFile, "utf8");
const parsed = JSON.parse(stripLeadingBom(raw)) as {
profiles?: InteractionMemoryProfile[];
};
const ids = new Set<string>();
for (const profile of parsed.profiles ?? []) {
for (const event of profile.recentEvents ?? []) {
if (event.sourceKind === "direct_prompt" && event.id.length > 0) {
ids.add(event.id);
}
}
}
return ids;
} catch (error) {
const code = (error as NodeJS.ErrnoException).code;
if (code === "ENOENT") {
return new Set();
}
throw error;
}
}
function stripLeadingBom(input: string): string {
return input.charCodeAt(0) === 0xfeff ? input.slice(1) : input;
}

1
state/evidence.jsonl
View file

@ -35,3 +35,4 @@
{"ts":"2026-05-02T23:40:40Z","type":"implementation","status":"ok","note":"Split room-read sidecar model selection from the main local reply model so Gemma can drive transcript profiling without changing the public/local answer lane."}
{"ts":"2026-05-03T07:40:36Z","type":"implementation","status":"ok","note":"Offsite backup was stopping bot and worker, then leaving the stack dead if the remote local-LLM box was asleep; startup now records local-LLM degradation without aborting, and watchdog treats a failed offsite status file as a warning while remote freshness is checked separately."}
{"ts":"2026-05-03T21:23:51Z","type":"implementation","status":"ok","note":"Local history-heavy prompts can time out when vector retrieval fixates on repeated ask-lines or comes back thin; overfetching history, filtering near-duplicate prompt echoes, and backstopping with raw archive lexical search keeps the local lane from burning its tool budget on empty or circular history reads."}
{"ts":"2026-05-04T08:53:16.7930326Z","type":"implementation","status":"ok","note":"Bot-directed prompts now stay archived for forensics but are tagged and excluded from default semantic history indexing and lexical fallback; a scrubber can retro-tag old prompt rows from interaction-memory event IDs plus mention heuristics and prune their history vectors without deleting the raw messages."}

1
state/map.yaml
View file

@ -64,6 +64,7 @@ accepted_design:
- `owner_codex` and `local_llm` remain distinct lanes with different tool-execution and safety posture.
- Ephemeral room-reading should come from an explicit local-LLM sidecar inference pass plus raw recent-message evidence, not from a rules bucket pretending to understand natural language.
- The VoidBot MCP server is the preferred search plane for archived Discord, indexed GameCult repos, Aetheria lore, and owner notifications.
- Bot-directed prompts should remain in the raw Discord archive for forensics, but default history indexing and retrieval should treat them as toxic self-reference rather than ordinary community memory.
- Deep agent-state, personality, culture, and persistent-world design work now belongs to the sibling Ghostlight repo; VoidBot should consume that work instead of remaining its long-term home.
- Long-running work should expose durable logs or status files instead of relying on one attached session and hope.
- Persistent project memory must stay ruthlessly distilled; git history and artifacts carry routine proof.