diff --git a/notes/fresh-workspace-handoff.md b/notes/fresh-workspace-handoff.md index fdc2ff8..72e6c37 100644 --- a/notes/fresh-workspace-handoff.md +++ b/notes/fresh-workspace-handoff.md @@ -31,6 +31,7 @@ Git history and smoke artifacts carry routine proof. `state/evidence.jsonl` shou - `scripts/run-void-moderator-rumination.ps1` now records reviewed Discord cursors and speech delivery receipts through typed self-state operations. The old `set-cursor` and `record-delivery-receipt` commands were removed from `scripts/moderation-state-store.mjs`. - The unattended moderation child no longer gets to commit whole-state JSON edits. It reads `.voidbot/private/moderation-agent-state.json` as a compatibility projection, writes proposed operation payloads to `.voidbot/status/moderation-rumination-operations.json`, and the parent runner applies them through `scripts/void-self-state.mjs apply-operation`. - `scripts/simulate-void-mood.mjs` now sends sleep-cycle and speaking-pressure results through typed operations after its legacy save, so scheduled-runtime typed documents stay current. The memory organ itself still mutates the legacy monolith and remains the next major surgery. +- `scripts/void-memory-organ.mjs` has started being split by concern. Shared limits/text sets, generic utilities, vector/embedder code, and memory-record intake now live in `scripts/void-memory-organ-constants.mjs`, `scripts/void-memory-organ-utils.mjs`, `scripts/void-memory-organ-vector.mjs`, and `scripts/void-memory-organ-records.mjs`, with `scripts/void-memory-organ-shared.mjs` as a temporary re-export shim. - Most of the deterministic cleanup in `scripts/void-memory-organ.mjs` is recent compensator cruft from failed attempts to manually clean an exploded state file. Do not preserve it as an earned organ. Architect the state boundary so that cleanup pile is unnecessary. - Sleep/distillation must be rewritten with a meaning-preservation contract: concrete subject, claim/question, evidence refs, live tension, and future-action implication survive compression. Bulk can die. Meaning does not get put through the office shredder and called a dream. - The owner Codex lane is no longer one swollen file; orchestration, runtime/parsing, rendering, and shared helpers are split under `packages/providers/src/owner-codex-*.ts`. @@ -89,7 +90,7 @@ Git history and smoke artifacts carry routine proof. `state/evidence.jsonl` shou ## Likely Next Bounded Move - Stop feature work on moderation, mood, agency, and public-lane behavior until the private-state boundary is rebuilt. -- First bounded move after the current commit: move memory-maintenance, incubation, and candidate-intervention outputs behind typed operations instead of direct canonical/working-projection writes. +- First bounded move after the current commit: keep splitting `scripts/void-memory-organ.mjs` by ownership, starting with cleanup/translation versus graph/incubation/agency, then move memory-maintenance, incubation, and candidate-intervention outputs behind typed operations instead of direct canonical/working-projection writes. - Next cuts should delete unstable foundations early after concrete writers move: remove `legacyJsonPath`/working-projection authority, then cut top-level mirror fields and cleanup code that only compensated for those broken boundaries. - Rebuild sleep/distillation before trusting it again. Fixture check: a memory like `AquariumSynthCSharp: Workflow cannot own the body` must survive sleep with its concrete subject, claim, evidence, tension, and implication intact. - Do not build adapters around the JSON projection. That is how the heap learned to stand upright and ask for snacks. diff --git a/scripts/void-memory-organ-constants.mjs b/scripts/void-memory-organ-constants.mjs new file mode 100644 index 0000000..fc52640 --- /dev/null +++ b/scripts/void-memory-organ-constants.mjs @@ -0,0 +1,230 @@ +export const DEFAULT_VECTOR_DIMENSIONS = 96; +export const MAX_EDGE_COUNT = 8; +export const MAX_CLUSTER_COUNT = 2; +export const MAX_INCUBATING_THOUGHTS = 2; +export const MAX_RECENT_EPISODIC_RECORDS = 10; +export const MAX_RECENT_QUIET_EPISODIC_RECORDS = 2; +export const MAX_POST_NAP_EPISODIC_RECORDS = 4; +export const MAX_POST_NAP_QUIET_EPISODIC_RECORDS = 1; +export const MAX_POST_NAP_ARCHIVE_EXCURSIONS = 2; +export const MAX_POST_NAP_REPO_SWEEPS = 2; +export const MAX_POST_NAP_NOVELTY_CHECKS = 3; +export const MAX_POST_NAP_MUSINGS = 6; +export const MAX_POST_NAP_RECENT_MUSINGS = 2; +export const MAX_POST_NAP_CANDIDATE_INTERVENTIONS = 4; +export const MAX_POST_NAP_SEAM_PROMOTIONS = 3; +export const MAX_POST_NAP_RESONANCE_EDGES = 4; +export const MAX_ARCHIVE_EXCURSION_MEMORIES = 4; +export const MAX_REPO_SWEEP_MEMORIES = 4; +export const MAX_SEMANTIC_MEMORIES = 14; +export const MAX_DREAM_MEMORIES = 5; +export const MAX_RUNTIME_REPO_SWEEPS = 1; +export const MAX_RUNTIME_REPO_ACTIVITY_MEMORIES = 1; +export const MAX_RUNTIME_ARCHIVE_EXCURSIONS = 2; +export const MAX_RUNTIME_RUMINATION_SEEDS = 4; +export const MAX_RECENT_SYNTHESIS_COUNT = 1; +export const MAX_SUPPORTING_REFS = 4; +export const MAX_CLUSTER_MEMORY_IDS = 6; +export const MAX_INCUBATION_SOURCE_IDS = 5; +export const MAX_DREAM_SOURCE_IDS = 4; +export const MAX_DISCOMFORT_COUNT = 6; +export const MAX_ACTIVE_TENSION_COUNT = 6; +export const MAX_ADVOCACY_REQUEST_COUNT = 4; +export const MIN_SUPPORT_FOR_IDENTITY_CRYSTALLIZATION = 64; +export const MIN_DEEP_DIVES_FOR_IDENTITY_CRYSTALLIZATION = 96; +export const MAX_RECENT_ANALYTIC_THREADS = 6; +export const MAX_RECENT_QUIET_ANALYTIC_THREADS = 1; +export const EDGE_SIMILARITY_THRESHOLD = 0.56; +export const CLUSTER_SIMILARITY_THRESHOLD = 0.64; +export const TOPIC_MATCH_THRESHOLD = 0.42; +export const REFRACTORY_MATCH_THRESHOLD = 0.48; +export const MAX_TOPIC_SATURATION_COUNT = 6; +export const MAX_REFRACTORY_TOPIC_COUNT = 6; +export const stopwords = new Set([ + "a", + "about", + "after", + "against", + "all", + "also", + "an", + "and", + "another", + "are", + "around", + "as", + "at", + "because", + "been", + "before", + "being", + "between", + "but", + "by", + "can", + "current", + "did", + "discord", + "do", + "does", + "doing", + "for", + "from", + "fresh", + "get", + "got", + "had", + "has", + "have", + "how", + "if", + "in", + "into", + "is", + "it", + "its", + "just", + "keep", + "keeps", + "kind", + "last", + "like", + "live", + "made", + "make", + "means", + "message", + "messages", + "more", + "most", + "new", + "no", + "not", + "now", + "of", + "on", + "one", + "owner", + "or", + "other", + "our", + "out", + "over", + "own", + "post", + "posted", + "quiet", + "recent", + "run", + "same", + "saved", + "seam", + "should", + "small", + "so", + "still", + "that", + "the", + "their", + "them", + "then", + "there", + "they", + "thing", + "this", + "thought", + "traffic", + "through", + "to", + "too", + "toward", + "under", + "up", + "use", + "using", + "very", + "was", + "we", + "were", + "what", + "when", + "where", + "which", + "while", + "with", + "worth", + "would", + "yet", + "you", + "smoke", +]); +export const labelNoiseTokens = new Set([ + "active", + "archive", + "bridge", + "build", + "built", + "candidate", + "cluster", + "cooling", + "deepened", + "dream", + "draft", + "episodic", + "distill", + "excursion", + "excursions", + "distilled", + "holding", + "identity", + "incubating", + "instead", + "intervention", + "interventions", + "latent", + "lane", + "memory", + "memories", + "messagepack", + "moderation", + "musing", + "musings", + "novelty", + "organ", + "organs", + "pass", + "private", + "question", + "questions", + "receipts", + "actually", + "behaving", + "branch", + "compressed", + "recurring", + "repo", + "repos", + "resonance", + "semantic", + "single", + "state", + "status", + "summary", + "surface", + "surfaces", + "sweep", + "sweeps", + "talking", + "than", + "thread", + "threads", + "topic", + "topics", + "trying", + "wants", + "present", + "part", + "rather", + "itself", + "across", +]); diff --git a/scripts/void-memory-organ-records.mjs b/scripts/void-memory-organ-records.mjs new file mode 100644 index 0000000..1fce92b --- /dev/null +++ b/scripts/void-memory-organ-records.mjs @@ -0,0 +1,224 @@ +import { + MAX_RECENT_ANALYTIC_THREADS, + MAX_RECENT_EPISODIC_RECORDS, + MAX_RECENT_QUIET_ANALYTIC_THREADS, + MAX_RECENT_QUIET_EPISODIC_RECORDS, +} from "./void-memory-organ-constants.mjs"; +import { + ensureArray, + ensureMemoryId, + ensureObject, + ensureStringArray, + isLowSignalQuietRoomText, + isObject, + normalizeText, + readString, + selectRecentRecords, +} from "./void-memory-organ-utils.mjs"; + +export function collectMemoryRecords({ memories, runtime }) { + const records = []; + + pushArrayRecords(records, selectRecentRecords(ensureArray(memories.episodic), { + limit: MAX_RECENT_EPISODIC_RECORDS, + quietLimit: MAX_RECENT_QUIET_EPISODIC_RECORDS, + }), "episodic", (entry) => ({ + text: readString(entry, "summary"), + label: readString(entry, "summary") ?? "episodic memory", + timestamp: readString(entry, "timestamp"), + sourceRefs: gatherSourceRefs(entry), + sourceMeta: gatherSourceMeta({ entry, kind: "episodic" }), + })); + pushArrayRecords(records, ensureArray(memories.semantic).slice(-24), "semantic", (entry) => ({ + text: [readString(entry, "subjectLabel"), readString(entry, "kind"), readString(entry, "summary")] + .filter(Boolean) + .join(" | "), + label: readString(entry, "subjectLabel") ?? readString(entry, "kind") ?? "semantic memory", + timestamp: readString(entry, "lastObservedAt"), + sourceRefs: gatherSourceRefs(entry), + sourceMeta: gatherSourceMeta({ entry, kind: "semantic" }), + })); + pushArrayRecords(records, ensureArray(memories.musings).slice(-12), "musing", (entry) => ({ + text: [readString(entry, "topic"), readString(entry, "summary")].filter(Boolean).join(" | "), + label: readString(entry, "topic") ?? "musing", + timestamp: readString(entry, "timestamp"), + sourceRefs: gatherSourceRefs(entry), + sourceMeta: gatherSourceMeta({ entry, kind: "musing" }), + })); + pushArrayRecords(records, ensureArray(memories.dreams).slice(-10), "dream", (entry) => ({ + text: [readString(entry, "theme"), readString(entry, "summary")].filter(Boolean).join(" | "), + label: readString(entry, "theme") ?? "dream", + timestamp: readString(entry, "timestamp"), + sourceRefs: gatherSourceRefs(entry), + sourceMeta: gatherSourceMeta({ entry, kind: "dream" }), + })); + pushArrayRecords(records, ensureArray(runtime.recent_archive_excursions).slice(-8), "archive_excursion", (entry) => ({ + text: [readString(entry, "topicHint"), readString(entry, "whyItWasFresh")].filter(Boolean).join(" | "), + label: readString(entry, "topicHint") ?? "archive excursion", + timestamp: readString(entry, "timestamp"), + sourceRefs: gatherSourceRefs(entry), + sourceMeta: gatherSourceMeta({ entry, kind: "archive_excursion" }), + })); + pushArrayRecords(records, ensureArray(runtime.recent_repo_activity_sweeps).slice(-8), "repo_sweep", (entry) => ({ + text: [readString(entry, "summary"), readString(entry, "whyItMattered")].filter(Boolean).join(" | "), + label: readString(entry, "summary") ?? "repo sweep", + timestamp: readString(entry, "timestamp"), + sourceRefs: gatherSourceRefs(entry), + sourceMeta: gatherSourceMeta({ entry, kind: "repo_sweep" }), + })); + pushArrayRecords( + records, + selectRecentRecords(ensureArray(ensureObject(runtime.thought_lanes).analytic?.active_threads), { + limit: MAX_RECENT_ANALYTIC_THREADS, + quietLimit: MAX_RECENT_QUIET_ANALYTIC_THREADS, + }), + "analytic_thread", + (entry) => ({ + text: [readString(entry, "topic"), readString(entry, "claim"), readString(entry, "counterweight")] + .filter(Boolean) + .join(" | "), + label: readString(entry, "topic") ?? "analytic thread", + timestamp: readString(entry, "lastTouchedAt"), + sourceRefs: gatherSourceRefs(entry), + sourceMeta: gatherSourceMeta({ entry, kind: "analytic_thread" }), + }), + ); + pushArrayRecords( + records, + ensureArray(ensureObject(runtime.thought_lanes).associative?.active_threads).slice(-6), + "associative_thread", + (entry) => ({ + text: [readString(entry, "topic"), readString(entry, "claim"), readString(entry, "counterweight")] + .filter(Boolean) + .join(" | "), + label: readString(entry, "topic") ?? "associative thread", + timestamp: readString(entry, "lastTouchedAt"), + sourceRefs: gatherSourceRefs(entry), + sourceMeta: gatherSourceMeta({ entry, kind: "associative_thread" }), + }), + ); + pushArrayRecords(records, ensureArray(ensureObject(runtime.bridge).recent_syntheses).slice(-8), "bridge_synthesis", (entry) => ({ + text: [readString(entry, "summary"), ...ensureStringArray(entry.dominantTopics)].filter(Boolean).join(" | "), + label: readString(entry, "summary") ?? "bridge synthesis", + timestamp: readString(entry, "timestamp"), + sourceRefs: gatherSourceRefs(entry), + sourceMeta: gatherSourceMeta({ entry, kind: "bridge_synthesis" }), + })); + pushArrayRecords(records, ensureArray(runtime.candidate_interventions).slice(-8), "candidate_intervention", (entry) => ({ + text: [readString(entry, "summary"), readString(entry, "draft")].filter(Boolean).join(" | "), + label: readString(entry, "summary") ?? "candidate intervention", + timestamp: readString(entry, "timestamp"), + sourceRefs: gatherSourceRefs(entry), + sourceMeta: gatherSourceMeta({ entry, kind: "candidate_intervention" }), + })); + + return records; +} + +function pushArrayRecords(target, entries, kind, toRecord) { + for (const entry of entries) { + if (!isObject(entry)) { + continue; + } + + const record = toRecord(entry); + const text = normalizeText(record.text); + + if (!text) { + continue; + } + + entry.memoryId = ensureMemoryId(entry, kind, text); + target.push({ + kind, + entry, + memoryId: entry.memoryId, + text, + label: record.label ?? kind, + timestamp: record.timestamp ?? null, + sourceRefs: ensureStringArray(record.sourceRefs), + sourceMeta: isObject(record.sourceMeta) ? record.sourceMeta : {}, + lowSignalQuietRoom: isLowSignalQuietRoomText(text), + }); + } +} + +function gatherSourceRefs(entry) { + if (!isObject(entry)) { + return []; + } + return [ + ...ensureStringArray(entry.evidenceRefs), + ...ensureStringArray(entry.evidenceMessageIds), + ...ensureStringArray(entry.distilledFrom), + ...ensureStringArray(entry.sourceMemoryIds), + ]; +} + +function gatherSourceMeta({ entry, kind }) { + const sourceMeta = {}; + const repoNames = [ + ...new Set([ + ...ensureStringArray(entry.repoNames), + ...inferRepoNamesFromEntry(entry), + ]), + ]; + if (repoNames.length > 0) { + sourceMeta.repoNames = repoNames; + } + + const channelId = readString(entry, "channelId"); + if (channelId) { + sourceMeta.channelId = channelId; + } + + if (kind === "archive_excursion") { + const archiveTimestamp = + readString(entry, "anchorTimestamp") ?? + readString(entry, "sourceTimestamp") ?? + readString(entry, "timestamp"); + const archiveYear = extractYear(archiveTimestamp); + if (archiveYear) { + sourceMeta.archiveYear = archiveYear; + } + } + + if (kind === "bridge_synthesis") { + const dominantTopics = ensureStringArray(entry.dominantTopics); + if (dominantTopics.length > 0) { + sourceMeta.dominantTopics = dominantTopics; + } + } + + return sourceMeta; +} + +function inferRepoNamesFromEntry(entry) { + const subjectLabel = + readString(entry, "subjectLabel") ?? + readString(entry, "topic") ?? + readString(entry, "label") ?? + readString(entry, "fascinationTarget"); + const repoName = extractRepoNameFromLabel(subjectLabel); + return repoName ? [repoName] : []; +} + +export function extractYear(timestamp) { + if (typeof timestamp !== "string" || timestamp.length < 4) { + return undefined; + } + const match = timestamp.match(/^(\d{4})-/); + return match ? match[1] : undefined; +} + +function extractRepoNameFromLabel(label) { + const normalized = normalizeText(label); + if (!normalized || !normalized.includes(":")) { + return null; + } + const [prefix] = normalized.split(/:\s*/, 1); + if (!prefix || !/[A-Z]/.test(prefix)) { + return null; + } + return prefix; +} diff --git a/scripts/void-memory-organ-shared.mjs b/scripts/void-memory-organ-shared.mjs new file mode 100644 index 0000000..03dc12d --- /dev/null +++ b/scripts/void-memory-organ-shared.mjs @@ -0,0 +1,3 @@ +export * from "./void-memory-organ-constants.mjs"; +export * from "./void-memory-organ-utils.mjs"; +export * from "./void-memory-organ-vector.mjs"; diff --git a/scripts/void-memory-organ-utils.mjs b/scripts/void-memory-organ-utils.mjs new file mode 100644 index 0000000..c7360a7 --- /dev/null +++ b/scripts/void-memory-organ-utils.mjs @@ -0,0 +1,255 @@ +import { createHash } from "node:crypto"; +import { readFileSync } from "node:fs"; + +import { labelNoiseTokens, stopwords } from "./void-memory-organ-constants.mjs"; + +export function topKeywords(text, limit) { + const counts = new Map(); + + for (const token of tokenize(text)) { + if (stopwords.has(token)) { + continue; + } + counts.set(token, (counts.get(token) ?? 0) + 1); + } + + return [...counts.entries()] + .sort((left, right) => right[1] - left[1]) + .slice(0, limit) + .map(([token]) => token); +} + +export function tokenize(input) { + return String(input) + .toLowerCase() + .split(/[^a-z0-9]+/i) + .map((token) => token.trim()) + .filter((token) => token.length > 2); +} + +export function ensureMemoryId(entry, kind, text) { + const existing = readString(entry, "memoryId"); + if (existing) { + return existing; + } + + const timestamp = readString(entry, "timestamp") ?? readString(entry, "lastObservedAt") ?? "undated"; + return `${kind}-${hashString(`${timestamp}|${text}`).slice(0, 12)}`; +} + +export function normalizeText(value) { + if (typeof value !== "string") { + return ""; + } + return value.replace(/\s+/g, " ").trim(); +} + +export function selectRecentRecords(entries, { limit, quietLimit }) { + const recent = ensureArray(entries).slice(-Math.max(limit * 3, limit)); + const quiet = []; + const active = []; + + for (let index = recent.length - 1; index >= 0; index -= 1) { + const entry = recent[index]; + if (!isObject(entry)) { + continue; + } + + const text = normalizeText([ + readString(entry, "summary"), + readString(entry, "claim"), + readString(entry, "topic"), + readString(entry, "counterweight"), + ].filter(Boolean).join(" | ")); + + if (isLowSignalQuietRoomText(text)) { + if (quiet.length < quietLimit) { + quiet.unshift(entry); + } + continue; + } + + if (active.length < limit - quietLimit) { + active.unshift(entry); + } + } + + return [...active, ...quiet].slice(-limit); +} + +export function isLowSignalQuietRoomText(text) { + const normalized = normalizeText(text).toLowerCase(); + if (!normalized) { + return false; + } + + const quietSignals = [ + "no new discord", + "no fresh discord", + "no new messages", + "no fresh live traffic", + "no live moderation", + "no moderation smoke", + "no owner escalation", + "room was empty", + "room is idle", + "there was no live moderation pressure", + "nothing got posted", + "no discord post", + ]; + + let matches = 0; + for (const signal of quietSignals) { + if (normalized.includes(signal)) { + matches += 1; + } + } + + return matches >= 2; +} + +export function overlapRatio(left, right) { + if (left.length === 0 || right.length === 0) { + return 0; + } + + const leftSet = new Set(left); + let overlap = 0; + + for (const value of right) { + if (leftSet.has(value)) { + overlap += 1; + } + } + + return overlap / Math.max(left.length, right.length); +} + +export function mergeStringArrays(left, right) { + return [...new Set([...ensureStringArray(left), ...ensureStringArray(right)])]; +} + +export function newestIsoTimestamp(left, right) { + const leftTime = parseIsoTimestamp(left); + const rightTime = parseIsoTimestamp(right); + if (leftTime === null) { + return rightTime === null ? undefined : right; + } + if (rightTime === null) { + return left; + } + return leftTime >= rightTime ? left : right; +} + +export function cloneJson(value) { + return JSON.parse(JSON.stringify(value)); +} + +export function parseIsoTimestamp(value) { + if (typeof value !== "string" || value.length === 0) { + return null; + } + const time = Date.parse(value); + return Number.isFinite(time) ? time : null; +} + +export function hashString(input) { + return createHash("sha1").update(String(input)).digest("hex"); +} + +export function parseDotEnvSafe(path) { + try { + return parseDotEnv(stripBom(readFileSync(path, "utf8"))); + } catch (error) { + if (error && typeof error === "object" && "code" in error && error.code === "ENOENT") { + return {}; + } + throw error; + } +} + +export function parseDotEnv(raw) { + const result = {}; + + for (const line of raw.split(/\r?\n/)) { + const trimmed = line.trim(); + if (trimmed.length === 0 || trimmed.startsWith("#")) { + continue; + } + + const separatorIndex = trimmed.indexOf("="); + if (separatorIndex === -1) { + continue; + } + + const key = trimmed.slice(0, separatorIndex).trim(); + let value = trimmed.slice(separatorIndex + 1).trim(); + + if ( + value.length >= 2 && + ((value.startsWith("\"") && value.endsWith("\"")) || + (value.startsWith("'") && value.endsWith("'"))) + ) { + value = value.slice(1, -1); + } + + result[key] = value; + } + + return result; +} + +export function normalizeBaseUrl(value) { + return String(value).replace(/\/+$/, ""); +} + +export function readInt(value, fallback) { + const parsed = Number.parseInt(String(value ?? ""), 10); + return Number.isInteger(parsed) && parsed > 0 ? parsed : fallback; +} + +export function ensureArray(value) { + return Array.isArray(value) ? value : []; +} + +export function ensureObject(value) { + return isObject(value) ? value : {}; +} + +export function ensureStringArray(value) { + return ensureArray(value).filter((entry) => typeof entry === "string"); +} + +export function readString(value, key) { + return isObject(value) && typeof value[key] === "string" ? value[key] : undefined; +} + +export function readNumber(value, key) { + return isObject(value) && typeof value[key] === "number" && Number.isFinite(value[key]) + ? value[key] + : undefined; +} + +export function readBoolean(value, key) { + return isObject(value) && typeof value[key] === "boolean" ? value[key] : undefined; +} + +export function round3(value) { + return Math.round(value * 1000) / 1000; +} + +export function round4(value) { + return Math.round(value * 10000) / 10000; +} + +export function clamp(value, min, max) { + return Math.min(max, Math.max(min, value)); +} + +export function stripBom(input) { + return input.charCodeAt(0) === 0xfeff ? input.slice(1) : input; +} + +export function isObject(value) { + return typeof value === "object" && value !== null && !Array.isArray(value); +} diff --git a/scripts/void-memory-organ-vector.mjs b/scripts/void-memory-organ-vector.mjs new file mode 100644 index 0000000..15d1442 --- /dev/null +++ b/scripts/void-memory-organ-vector.mjs @@ -0,0 +1,142 @@ +import { resolve } from "node:path"; + +import { DEFAULT_VECTOR_DIMENSIONS } from "./void-memory-organ-constants.mjs"; +import { normalizeBaseUrl, normalizeText, parseDotEnvSafe, readInt } from "./void-memory-organ-utils.mjs"; + +export function createEmbedder({ repoRootPath }) { + const env = parseDotEnvSafe(resolve(repoRootPath, ".env")); + const backend = env.RAG_EMBEDDING_BACKEND === "hash" ? "hash" : "ollama"; + const dimensions = readInt(env.VOID_MEMORY_VECTOR_DIMENSIONS, DEFAULT_VECTOR_DIMENSIONS); + + if (backend === "ollama") { + return { + backend: "ollama", + model: env.RAG_OLLAMA_MODEL?.trim() || "qwen3-embedding:0.6b", + dimensions, + baseUrl: normalizeBaseUrl(env.RAG_OLLAMA_BASE_URL?.trim() || "http://127.0.0.1:11434"), + async embed(text) { + try { + const response = await fetch(`${this.baseUrl}/api/embed`, { + method: "POST", + headers: { "Content-Type": "application/json" }, + body: JSON.stringify({ + model: this.model, + input: [text], + }), + }); + + if (!response.ok) { + throw new Error(`Ollama responded ${response.status}`); + } + + const payload = await response.json(); + const raw = Array.isArray(payload?.embeddings) ? payload.embeddings[0] : undefined; + if (!Array.isArray(raw) || raw.length === 0) { + throw new Error("No embedding returned."); + } + + return compactVector(raw.map(Number), dimensions); + } catch { + this.backend = "hash"; + this.model = `hash:${dimensions}`; + return hashVector(text, dimensions); + } + }, + }; + } + + return { + backend: "hash", + model: `hash:${dimensions}`, + dimensions, + async embed(text) { + return hashVector(text, dimensions); + }, + }; +} + +export function compactVector(values, targetDimensions) { + if (values.length === targetDimensions) { + return normalizeVector(values); + } + + const compacted = new Array(targetDimensions).fill(0); + + for (let index = 0; index < targetDimensions; index += 1) { + const start = Math.floor((index * values.length) / targetDimensions); + const end = Math.floor(((index + 1) * values.length) / targetDimensions); + const slice = values.slice(start, Math.max(start + 1, end)); + compacted[index] = + slice.reduce((sum, value) => sum + (Number.isFinite(value) ? value : 0), 0) / + Math.max(1, slice.length); + } + + return normalizeVector(compacted); +} + +export function hashVector(text, dimensions) { + const terms = tokenize(text); + + if (terms.length === 0) { + return new Array(dimensions).fill(0); + } + + const vector = new Array(dimensions).fill(0); + + for (const term of terms) { + const hash = createHash("sha1").update(term).digest(); + const index = hash.readUInt32BE(0) % dimensions; + const sign = hash[4] % 2 === 0 ? 1 : -1; + vector[index] += sign; + } + + return normalizeVector(vector); +} + +export function averagePairwiseSimilarity(items) { + if (items.length < 2) { + return 0; + } + + let total = 0; + let count = 0; + + for (let leftIndex = 0; leftIndex < items.length; leftIndex += 1) { + for (let rightIndex = leftIndex + 1; rightIndex < items.length; rightIndex += 1) { + total += cosineSimilarity(items[leftIndex].entry.semanticVector.values, items[rightIndex].entry.semanticVector.values); + count += 1; + } + } + + return count === 0 ? 0 : total / count; +} + +export function cosineSimilarity(left, right) { + if (!Array.isArray(left) || !Array.isArray(right) || left.length !== right.length || left.length === 0) { + return 0; + } + + let dot = 0; + let leftMagnitude = 0; + let rightMagnitude = 0; + + for (let index = 0; index < left.length; index += 1) { + dot += left[index] * right[index]; + leftMagnitude += left[index] ** 2; + rightMagnitude += right[index] ** 2; + } + + if (leftMagnitude === 0 || rightMagnitude === 0) { + return 0; + } + + return dot / Math.sqrt(leftMagnitude * rightMagnitude); +} + +export function normalizeVector(vector) { + const magnitude = Math.sqrt(vector.reduce((sum, value) => sum + value ** 2, 0)); + if (magnitude === 0) { + return vector; + } + return vector.map((value) => value / magnitude); +} diff --git a/scripts/void-memory-organ.mjs b/scripts/void-memory-organ.mjs index a95162a..b92b71b 100644 --- a/scripts/void-memory-organ.mjs +++ b/scripts/void-memory-organ.mjs @@ -1,240 +1,88 @@ -import { createHash } from "node:crypto"; -import { readFileSync } from "node:fs"; import { dirname, resolve } from "node:path"; import { fileURLToPath } from "node:url"; -const repoRoot = resolve(dirname(fileURLToPath(import.meta.url)), ".."); +import { + DEFAULT_VECTOR_DIMENSIONS, + MAX_EDGE_COUNT, + MAX_CLUSTER_COUNT, + MAX_INCUBATING_THOUGHTS, + MAX_RECENT_EPISODIC_RECORDS, + MAX_RECENT_QUIET_EPISODIC_RECORDS, + MAX_POST_NAP_EPISODIC_RECORDS, + MAX_POST_NAP_QUIET_EPISODIC_RECORDS, + MAX_POST_NAP_ARCHIVE_EXCURSIONS, + MAX_POST_NAP_REPO_SWEEPS, + MAX_POST_NAP_NOVELTY_CHECKS, + MAX_POST_NAP_MUSINGS, + MAX_POST_NAP_RECENT_MUSINGS, + MAX_POST_NAP_CANDIDATE_INTERVENTIONS, + MAX_POST_NAP_SEAM_PROMOTIONS, + MAX_POST_NAP_RESONANCE_EDGES, + MAX_ARCHIVE_EXCURSION_MEMORIES, + MAX_REPO_SWEEP_MEMORIES, + MAX_SEMANTIC_MEMORIES, + MAX_DREAM_MEMORIES, + MAX_RUNTIME_REPO_SWEEPS, + MAX_RUNTIME_REPO_ACTIVITY_MEMORIES, + MAX_RUNTIME_ARCHIVE_EXCURSIONS, + MAX_RUNTIME_RUMINATION_SEEDS, + MAX_RECENT_SYNTHESIS_COUNT, + MAX_SUPPORTING_REFS, + MAX_CLUSTER_MEMORY_IDS, + MAX_INCUBATION_SOURCE_IDS, + MAX_DREAM_SOURCE_IDS, + MAX_DISCOMFORT_COUNT, + MAX_ACTIVE_TENSION_COUNT, + MAX_ADVOCACY_REQUEST_COUNT, + MIN_SUPPORT_FOR_IDENTITY_CRYSTALLIZATION, + MIN_DEEP_DIVES_FOR_IDENTITY_CRYSTALLIZATION, + MAX_RECENT_ANALYTIC_THREADS, + MAX_RECENT_QUIET_ANALYTIC_THREADS, + EDGE_SIMILARITY_THRESHOLD, + CLUSTER_SIMILARITY_THRESHOLD, + TOPIC_MATCH_THRESHOLD, + REFRACTORY_MATCH_THRESHOLD, + MAX_TOPIC_SATURATION_COUNT, + MAX_REFRACTORY_TOPIC_COUNT, + stopwords, + labelNoiseTokens, + createEmbedder, + compactVector, + hashVector, + averagePairwiseSimilarity, + cosineSimilarity, + normalizeVector, + topKeywords, + tokenize, + ensureMemoryId, + normalizeText, + selectRecentRecords, + isLowSignalQuietRoomText, + overlapRatio, + mergeStringArrays, + newestIsoTimestamp, + cloneJson, + parseIsoTimestamp, + hashString, + parseDotEnvSafe, + parseDotEnv, + normalizeBaseUrl, + readInt, + ensureArray, + ensureObject, + ensureStringArray, + readString, + readNumber, + readBoolean, + round3, + round4, + clamp, + stripBom, + isObject, +} from "./void-memory-organ-shared.mjs"; +import { collectMemoryRecords, extractYear } from "./void-memory-organ-records.mjs"; -const DEFAULT_VECTOR_DIMENSIONS = 96; -const MAX_EDGE_COUNT = 8; -const MAX_CLUSTER_COUNT = 2; -const MAX_INCUBATING_THOUGHTS = 2; -const MAX_RECENT_EPISODIC_RECORDS = 10; -const MAX_RECENT_QUIET_EPISODIC_RECORDS = 2; -const MAX_POST_NAP_EPISODIC_RECORDS = 4; -const MAX_POST_NAP_QUIET_EPISODIC_RECORDS = 1; -const MAX_POST_NAP_ARCHIVE_EXCURSIONS = 2; -const MAX_POST_NAP_REPO_SWEEPS = 2; -const MAX_POST_NAP_NOVELTY_CHECKS = 3; -const MAX_POST_NAP_MUSINGS = 6; -const MAX_POST_NAP_RECENT_MUSINGS = 2; -const MAX_POST_NAP_CANDIDATE_INTERVENTIONS = 4; -const MAX_POST_NAP_SEAM_PROMOTIONS = 3; -const MAX_POST_NAP_RESONANCE_EDGES = 4; -const MAX_ARCHIVE_EXCURSION_MEMORIES = 4; -const MAX_REPO_SWEEP_MEMORIES = 4; -const MAX_SEMANTIC_MEMORIES = 14; -const MAX_DREAM_MEMORIES = 5; -const MAX_RUNTIME_REPO_SWEEPS = 1; -const MAX_RUNTIME_REPO_ACTIVITY_MEMORIES = 1; -const MAX_RUNTIME_ARCHIVE_EXCURSIONS = 2; -const MAX_RUNTIME_RUMINATION_SEEDS = 4; -const MAX_RECENT_SYNTHESIS_COUNT = 1; -const MAX_SUPPORTING_REFS = 4; -const MAX_CLUSTER_MEMORY_IDS = 6; -const MAX_INCUBATION_SOURCE_IDS = 5; -const MAX_DREAM_SOURCE_IDS = 4; -const MAX_DISCOMFORT_COUNT = 6; -const MAX_ACTIVE_TENSION_COUNT = 6; -const MAX_ADVOCACY_REQUEST_COUNT = 4; -const MIN_SUPPORT_FOR_IDENTITY_CRYSTALLIZATION = 64; -const MIN_DEEP_DIVES_FOR_IDENTITY_CRYSTALLIZATION = 96; -const MAX_RECENT_ANALYTIC_THREADS = 6; -const MAX_RECENT_QUIET_ANALYTIC_THREADS = 1; -const EDGE_SIMILARITY_THRESHOLD = 0.56; -const CLUSTER_SIMILARITY_THRESHOLD = 0.64; -const TOPIC_MATCH_THRESHOLD = 0.42; -const REFRACTORY_MATCH_THRESHOLD = 0.48; -const MAX_TOPIC_SATURATION_COUNT = 6; -const MAX_REFRACTORY_TOPIC_COUNT = 6; -const stopwords = new Set([ - "a", - "about", - "after", - "against", - "all", - "also", - "an", - "and", - "another", - "are", - "around", - "as", - "at", - "because", - "been", - "before", - "being", - "between", - "but", - "by", - "can", - "current", - "did", - "discord", - "do", - "does", - "doing", - "for", - "from", - "fresh", - "get", - "got", - "had", - "has", - "have", - "how", - "if", - "in", - "into", - "is", - "it", - "its", - "just", - "keep", - "keeps", - "kind", - "last", - "like", - "live", - "made", - "make", - "means", - "message", - "messages", - "more", - "most", - "new", - "no", - "not", - "now", - "of", - "on", - "one", - "owner", - "or", - "other", - "our", - "out", - "over", - "own", - "post", - "posted", - "quiet", - "recent", - "run", - "same", - "saved", - "seam", - "should", - "small", - "so", - "still", - "that", - "the", - "their", - "them", - "then", - "there", - "they", - "thing", - "this", - "thought", - "traffic", - "through", - "to", - "too", - "toward", - "under", - "up", - "use", - "using", - "very", - "was", - "we", - "were", - "what", - "when", - "where", - "which", - "while", - "with", - "worth", - "would", - "yet", - "you", - "smoke", -]); -const labelNoiseTokens = new Set([ - "active", - "archive", - "bridge", - "build", - "built", - "candidate", - "cluster", - "cooling", - "deepened", - "dream", - "draft", - "episodic", - "distill", - "excursion", - "excursions", - "distilled", - "holding", - "identity", - "incubating", - "instead", - "intervention", - "interventions", - "latent", - "lane", - "memory", - "memories", - "messagepack", - "moderation", - "musing", - "musings", - "novelty", - "organ", - "organs", - "pass", - "private", - "question", - "questions", - "receipts", - "actually", - "behaving", - "branch", - "compressed", - "recurring", - "repo", - "repos", - "resonance", - "semantic", - "single", - "state", - "status", - "summary", - "surface", - "surfaces", - "sweep", - "sweeps", - "talking", - "than", - "thread", - "threads", - "topic", - "topics", - "trying", - "wants", - "present", - "part", - "rather", - "itself", - "across", -]); +const repoRoot = resolve(dirname(fileURLToPath(import.meta.url)), ".."); export async function reconcileSemanticMemoryState({ state, @@ -1405,133 +1253,6 @@ function summarizeThoughtLaneSource(sourceValue, fallback) { return null; } -function collectMemoryRecords({ memories, runtime }) { - const records = []; - - pushArrayRecords(records, selectRecentRecords(ensureArray(memories.episodic), { - limit: MAX_RECENT_EPISODIC_RECORDS, - quietLimit: MAX_RECENT_QUIET_EPISODIC_RECORDS, - }), "episodic", (entry) => ({ - text: readString(entry, "summary"), - label: readString(entry, "summary") ?? "episodic memory", - timestamp: readString(entry, "timestamp"), - sourceRefs: gatherSourceRefs(entry), - sourceMeta: gatherSourceMeta({ entry, kind: "episodic" }), - })); - pushArrayRecords(records, ensureArray(memories.semantic).slice(-24), "semantic", (entry) => ({ - text: [readString(entry, "subjectLabel"), readString(entry, "kind"), readString(entry, "summary")] - .filter(Boolean) - .join(" | "), - label: readString(entry, "subjectLabel") ?? readString(entry, "kind") ?? "semantic memory", - timestamp: readString(entry, "lastObservedAt"), - sourceRefs: gatherSourceRefs(entry), - sourceMeta: gatherSourceMeta({ entry, kind: "semantic" }), - })); - pushArrayRecords(records, ensureArray(memories.musings).slice(-12), "musing", (entry) => ({ - text: [readString(entry, "topic"), readString(entry, "summary")].filter(Boolean).join(" | "), - label: readString(entry, "topic") ?? "musing", - timestamp: readString(entry, "timestamp"), - sourceRefs: gatherSourceRefs(entry), - sourceMeta: gatherSourceMeta({ entry, kind: "musing" }), - })); - pushArrayRecords(records, ensureArray(memories.dreams).slice(-10), "dream", (entry) => ({ - text: [readString(entry, "theme"), readString(entry, "summary")].filter(Boolean).join(" | "), - label: readString(entry, "theme") ?? "dream", - timestamp: readString(entry, "timestamp"), - sourceRefs: gatherSourceRefs(entry), - sourceMeta: gatherSourceMeta({ entry, kind: "dream" }), - })); - pushArrayRecords(records, ensureArray(runtime.recent_archive_excursions).slice(-8), "archive_excursion", (entry) => ({ - text: [readString(entry, "topicHint"), readString(entry, "whyItWasFresh")].filter(Boolean).join(" | "), - label: readString(entry, "topicHint") ?? "archive excursion", - timestamp: readString(entry, "timestamp"), - sourceRefs: gatherSourceRefs(entry), - sourceMeta: gatherSourceMeta({ entry, kind: "archive_excursion" }), - })); - pushArrayRecords(records, ensureArray(runtime.recent_repo_activity_sweeps).slice(-8), "repo_sweep", (entry) => ({ - text: [readString(entry, "summary"), readString(entry, "whyItMattered")].filter(Boolean).join(" | "), - label: readString(entry, "summary") ?? "repo sweep", - timestamp: readString(entry, "timestamp"), - sourceRefs: gatherSourceRefs(entry), - sourceMeta: gatherSourceMeta({ entry, kind: "repo_sweep" }), - })); - pushArrayRecords( - records, - selectRecentRecords(ensureArray(ensureObject(runtime.thought_lanes).analytic?.active_threads), { - limit: MAX_RECENT_ANALYTIC_THREADS, - quietLimit: MAX_RECENT_QUIET_ANALYTIC_THREADS, - }), - "analytic_thread", - (entry) => ({ - text: [readString(entry, "topic"), readString(entry, "claim"), readString(entry, "counterweight")] - .filter(Boolean) - .join(" | "), - label: readString(entry, "topic") ?? "analytic thread", - timestamp: readString(entry, "lastTouchedAt"), - sourceRefs: gatherSourceRefs(entry), - sourceMeta: gatherSourceMeta({ entry, kind: "analytic_thread" }), - }), - ); - pushArrayRecords( - records, - ensureArray(ensureObject(runtime.thought_lanes).associative?.active_threads).slice(-6), - "associative_thread", - (entry) => ({ - text: [readString(entry, "topic"), readString(entry, "claim"), readString(entry, "counterweight")] - .filter(Boolean) - .join(" | "), - label: readString(entry, "topic") ?? "associative thread", - timestamp: readString(entry, "lastTouchedAt"), - sourceRefs: gatherSourceRefs(entry), - sourceMeta: gatherSourceMeta({ entry, kind: "associative_thread" }), - }), - ); - pushArrayRecords(records, ensureArray(ensureObject(runtime.bridge).recent_syntheses).slice(-8), "bridge_synthesis", (entry) => ({ - text: [readString(entry, "summary"), ...ensureStringArray(entry.dominantTopics)].filter(Boolean).join(" | "), - label: readString(entry, "summary") ?? "bridge synthesis", - timestamp: readString(entry, "timestamp"), - sourceRefs: gatherSourceRefs(entry), - sourceMeta: gatherSourceMeta({ entry, kind: "bridge_synthesis" }), - })); - pushArrayRecords(records, ensureArray(runtime.candidate_interventions).slice(-8), "candidate_intervention", (entry) => ({ - text: [readString(entry, "summary"), readString(entry, "draft")].filter(Boolean).join(" | "), - label: readString(entry, "summary") ?? "candidate intervention", - timestamp: readString(entry, "timestamp"), - sourceRefs: gatherSourceRefs(entry), - sourceMeta: gatherSourceMeta({ entry, kind: "candidate_intervention" }), - })); - - return records; -} - -function pushArrayRecords(target, entries, kind, toRecord) { - for (const entry of entries) { - if (!isObject(entry)) { - continue; - } - - const record = toRecord(entry); - const text = normalizeText(record.text); - - if (!text) { - continue; - } - - entry.memoryId = ensureMemoryId(entry, kind, text); - target.push({ - kind, - entry, - memoryId: entry.memoryId, - text, - label: record.label ?? kind, - timestamp: record.timestamp ?? null, - sourceRefs: ensureStringArray(record.sourceRefs), - sourceMeta: isObject(record.sourceMeta) ? record.sourceMeta : {}, - lowSignalQuietRoom: isLowSignalQuietRoomText(text), - }); - } -} - async function ensureSemanticVector({ record, embedder, now }) { const prior = isObject(record.entry.semanticVector) ? record.entry.semanticVector : {}; const contentHash = hashString(record.text); @@ -3866,455 +3587,3 @@ function mapCountEntries(map, keyName) { .sort((left, right) => right.count - left.count); } -function gatherSourceRefs(entry) { - if (!isObject(entry)) { - return []; - } - return [ - ...ensureStringArray(entry.evidenceRefs), - ...ensureStringArray(entry.evidenceMessageIds), - ...ensureStringArray(entry.distilledFrom), - ...ensureStringArray(entry.sourceMemoryIds), - ]; -} - -function gatherSourceMeta({ entry, kind }) { - const sourceMeta = {}; - const repoNames = [ - ...new Set([ - ...ensureStringArray(entry.repoNames), - ...inferRepoNamesFromEntry(entry), - ]), - ]; - if (repoNames.length > 0) { - sourceMeta.repoNames = repoNames; - } - - const channelId = readString(entry, "channelId"); - if (channelId) { - sourceMeta.channelId = channelId; - } - - if (kind === "archive_excursion") { - const archiveTimestamp = - readString(entry, "anchorTimestamp") ?? - readString(entry, "sourceTimestamp") ?? - readString(entry, "timestamp"); - const archiveYear = extractYear(archiveTimestamp); - if (archiveYear) { - sourceMeta.archiveYear = archiveYear; - } - } - - if (kind === "bridge_synthesis") { - const dominantTopics = ensureStringArray(entry.dominantTopics); - if (dominantTopics.length > 0) { - sourceMeta.dominantTopics = dominantTopics; - } - } - - return sourceMeta; -} - -function inferRepoNamesFromEntry(entry) { - const subjectLabel = - readString(entry, "subjectLabel") ?? - readString(entry, "topic") ?? - readString(entry, "label") ?? - readString(entry, "fascinationTarget"); - const repoName = extractRepoNameFromLabel(subjectLabel); - return repoName ? [repoName] : []; -} - -function extractYear(timestamp) { - if (typeof timestamp !== "string" || timestamp.length < 4) { - return undefined; - } - const match = timestamp.match(/^(\d{4})-/); - return match ? match[1] : undefined; -} - -function createEmbedder({ repoRootPath }) { - const env = parseDotEnvSafe(resolve(repoRootPath, ".env")); - const backend = env.RAG_EMBEDDING_BACKEND === "hash" ? "hash" : "ollama"; - const dimensions = readInt(env.VOID_MEMORY_VECTOR_DIMENSIONS, DEFAULT_VECTOR_DIMENSIONS); - - if (backend === "ollama") { - return { - backend: "ollama", - model: env.RAG_OLLAMA_MODEL?.trim() || "qwen3-embedding:0.6b", - dimensions, - baseUrl: normalizeBaseUrl(env.RAG_OLLAMA_BASE_URL?.trim() || "http://127.0.0.1:11434"), - async embed(text) { - try { - const response = await fetch(`${this.baseUrl}/api/embed`, { - method: "POST", - headers: { "Content-Type": "application/json" }, - body: JSON.stringify({ - model: this.model, - input: [text], - }), - }); - - if (!response.ok) { - throw new Error(`Ollama responded ${response.status}`); - } - - const payload = await response.json(); - const raw = Array.isArray(payload?.embeddings) ? payload.embeddings[0] : undefined; - if (!Array.isArray(raw) || raw.length === 0) { - throw new Error("No embedding returned."); - } - - return compactVector(raw.map(Number), dimensions); - } catch { - this.backend = "hash"; - this.model = `hash:${dimensions}`; - return hashVector(text, dimensions); - } - }, - }; - } - - return { - backend: "hash", - model: `hash:${dimensions}`, - dimensions, - async embed(text) { - return hashVector(text, dimensions); - }, - }; -} - -function compactVector(values, targetDimensions) { - if (values.length === targetDimensions) { - return normalizeVector(values); - } - - const compacted = new Array(targetDimensions).fill(0); - - for (let index = 0; index < targetDimensions; index += 1) { - const start = Math.floor((index * values.length) / targetDimensions); - const end = Math.floor(((index + 1) * values.length) / targetDimensions); - const slice = values.slice(start, Math.max(start + 1, end)); - compacted[index] = - slice.reduce((sum, value) => sum + (Number.isFinite(value) ? value : 0), 0) / - Math.max(1, slice.length); - } - - return normalizeVector(compacted); -} - -function hashVector(text, dimensions) { - const terms = tokenize(text); - - if (terms.length === 0) { - return new Array(dimensions).fill(0); - } - - const vector = new Array(dimensions).fill(0); - - for (const term of terms) { - const hash = createHash("sha1").update(term).digest(); - const index = hash.readUInt32BE(0) % dimensions; - const sign = hash[4] % 2 === 0 ? 1 : -1; - vector[index] += sign; - } - - return normalizeVector(vector); -} - -function averagePairwiseSimilarity(items) { - if (items.length < 2) { - return 0; - } - - let total = 0; - let count = 0; - - for (let leftIndex = 0; leftIndex < items.length; leftIndex += 1) { - for (let rightIndex = leftIndex + 1; rightIndex < items.length; rightIndex += 1) { - total += cosineSimilarity(items[leftIndex].entry.semanticVector.values, items[rightIndex].entry.semanticVector.values); - count += 1; - } - } - - return count === 0 ? 0 : total / count; -} - -function cosineSimilarity(left, right) { - if (!Array.isArray(left) || !Array.isArray(right) || left.length !== right.length || left.length === 0) { - return 0; - } - - let dot = 0; - let leftMagnitude = 0; - let rightMagnitude = 0; - - for (let index = 0; index < left.length; index += 1) { - dot += left[index] * right[index]; - leftMagnitude += left[index] ** 2; - rightMagnitude += right[index] ** 2; - } - - if (leftMagnitude === 0 || rightMagnitude === 0) { - return 0; - } - - return dot / Math.sqrt(leftMagnitude * rightMagnitude); -} - -function normalizeVector(vector) { - const magnitude = Math.sqrt(vector.reduce((sum, value) => sum + value ** 2, 0)); - if (magnitude === 0) { - return vector; - } - return vector.map((value) => value / magnitude); -} - -function topKeywords(text, limit) { - const counts = new Map(); - - for (const token of tokenize(text)) { - if (stopwords.has(token)) { - continue; - } - counts.set(token, (counts.get(token) ?? 0) + 1); - } - - return [...counts.entries()] - .sort((left, right) => right[1] - left[1]) - .slice(0, limit) - .map(([token]) => token); -} - -function tokenize(input) { - return String(input) - .toLowerCase() - .split(/[^a-z0-9]+/i) - .map((token) => token.trim()) - .filter((token) => token.length > 2); -} - -function ensureMemoryId(entry, kind, text) { - const existing = readString(entry, "memoryId"); - if (existing) { - return existing; - } - - const timestamp = readString(entry, "timestamp") ?? readString(entry, "lastObservedAt") ?? "undated"; - return `${kind}-${hashString(`${timestamp}|${text}`).slice(0, 12)}`; -} - -function normalizeText(value) { - if (typeof value !== "string") { - return ""; - } - return value.replace(/\s+/g, " ").trim(); -} - -function selectRecentRecords(entries, { limit, quietLimit }) { - const recent = ensureArray(entries).slice(-Math.max(limit * 3, limit)); - const quiet = []; - const active = []; - - for (let index = recent.length - 1; index >= 0; index -= 1) { - const entry = recent[index]; - if (!isObject(entry)) { - continue; - } - - const text = normalizeText([ - readString(entry, "summary"), - readString(entry, "claim"), - readString(entry, "topic"), - readString(entry, "counterweight"), - ].filter(Boolean).join(" | ")); - - if (isLowSignalQuietRoomText(text)) { - if (quiet.length < quietLimit) { - quiet.unshift(entry); - } - continue; - } - - if (active.length < limit - quietLimit) { - active.unshift(entry); - } - } - - return [...active, ...quiet].slice(-limit); -} - -function isLowSignalQuietRoomText(text) { - const normalized = normalizeText(text).toLowerCase(); - if (!normalized) { - return false; - } - - const quietSignals = [ - "no new discord", - "no fresh discord", - "no new messages", - "no fresh live traffic", - "no live moderation", - "no moderation smoke", - "no owner escalation", - "room was empty", - "room is idle", - "there was no live moderation pressure", - "nothing got posted", - "no discord post", - ]; - - let matches = 0; - for (const signal of quietSignals) { - if (normalized.includes(signal)) { - matches += 1; - } - } - - return matches >= 2; -} - -function overlapRatio(left, right) { - if (left.length === 0 || right.length === 0) { - return 0; - } - - const leftSet = new Set(left); - let overlap = 0; - - for (const value of right) { - if (leftSet.has(value)) { - overlap += 1; - } - } - - return overlap / Math.max(left.length, right.length); -} - -function mergeStringArrays(left, right) { - return [...new Set([...ensureStringArray(left), ...ensureStringArray(right)])]; -} - -function newestIsoTimestamp(left, right) { - const leftTime = parseIsoTimestamp(left); - const rightTime = parseIsoTimestamp(right); - if (leftTime === null) { - return rightTime === null ? undefined : right; - } - if (rightTime === null) { - return left; - } - return leftTime >= rightTime ? left : right; -} - -function parseIsoTimestamp(value) { - if (typeof value !== "string" || value.length === 0) { - return null; - } - const time = Date.parse(value); - return Number.isFinite(time) ? time : null; -} - -function hashString(input) { - return createHash("sha1").update(String(input)).digest("hex"); -} - -function parseDotEnvSafe(path) { - try { - return parseDotEnv(stripBom(readFileSync(path, "utf8"))); - } catch (error) { - if (error && typeof error === "object" && "code" in error && error.code === "ENOENT") { - return {}; - } - throw error; - } -} - -function parseDotEnv(raw) { - const result = {}; - - for (const line of raw.split(/\r?\n/)) { - const trimmed = line.trim(); - if (trimmed.length === 0 || trimmed.startsWith("#")) { - continue; - } - - const separatorIndex = trimmed.indexOf("="); - if (separatorIndex === -1) { - continue; - } - - const key = trimmed.slice(0, separatorIndex).trim(); - let value = trimmed.slice(separatorIndex + 1).trim(); - - if ( - value.length >= 2 && - ((value.startsWith("\"") && value.endsWith("\"")) || - (value.startsWith("'") && value.endsWith("'"))) - ) { - value = value.slice(1, -1); - } - - result[key] = value; - } - - return result; -} - -function normalizeBaseUrl(value) { - return String(value).replace(/\/+$/, ""); -} - -function readInt(value, fallback) { - const parsed = Number.parseInt(String(value ?? ""), 10); - return Number.isInteger(parsed) && parsed > 0 ? parsed : fallback; -} - -function ensureArray(value) { - return Array.isArray(value) ? value : []; -} - -function ensureObject(value) { - return isObject(value) ? value : {}; -} - -function ensureStringArray(value) { - return ensureArray(value).filter((entry) => typeof entry === "string"); -} - -function readString(value, key) { - return isObject(value) && typeof value[key] === "string" ? value[key] : undefined; -} - -function readNumber(value, key) { - return isObject(value) && typeof value[key] === "number" && Number.isFinite(value[key]) - ? value[key] - : undefined; -} - -function readBoolean(value, key) { - return isObject(value) && typeof value[key] === "boolean" ? value[key] : undefined; -} - -function round3(value) { - return Math.round(value * 1000) / 1000; -} - -function round4(value) { - return Math.round(value * 10000) / 10000; -} - -function clamp(value, min, max) { - return Math.min(max, Math.max(min, value)); -} - -function stripBom(input) { - return input.charCodeAt(0) === 0xfeff ? input.slice(1) : input; -} - -function isObject(value) { - return typeof value === "object" && value !== null && !Array.isArray(value); -} diff --git a/state/evidence.jsonl b/state/evidence.jsonl index abc7bfd..7f7616d 100644 --- a/state/evidence.jsonl +++ b/state/evidence.jsonl @@ -96,3 +96,4 @@ {"ts":"2026-05-16T11:14:12Z","type":"implementation","status":"ok","note":"Moderation runner now records reviewed cursor and delivered speech receipts through typed self-state operations instead of moderation-state-store mutation commands. The old set-cursor and record-delivery-receipt commands/functions were removed from the legacy moderation-state-store surface. Receipt operation smoke verifies it records one receipt and closes the matching open case."} {"ts":"2026-05-16T11:17:09Z","type":"implementation","status":"ok","note":"The unattended moderation runner no longer commits child edits from moderation-agent-state.json. The child is instructed to treat the JSON as read-only and write a JSON array of typed operation payloads to .voidbot/status/moderation-rumination-operations.json; the parent applies each operation through scripts/void-self-state.mjs before recording cursor/receipt operations. Cheap parser/typecheck verification passed; full scheduled dry run was not run to avoid mutating live moderation state during code verification."} {"ts":"2026-05-16T11:20:42Z","type":"implementation","status":"ok","note":"Mood drift now sends sleep-cycle and speaking-pressure results through typed self-state operations after saving the legacy monolith, so scheduled-runtime typed documents stay current. The larger memory organ still mutates legacy state directly; next surgery is to move memory/incubation/candidate outputs behind operations and then remove working-projection materialization."} +{"ts":"2026-05-16T11:44:36Z","type":"implementation","status":"ok","note":"Started breaking up scripts/void-memory-organ.mjs by ownership. Extracted shared constants/text sets, generic utilities, vector/embedder substrate, and memory-record intake into separate modules while keeping the public reconcileSemanticMemoryState entrypoint stable. Hash/template smoke exercised the extracted modules without touching live state. Remaining monolith work is policy organs: cleanup/translation, graph/cluster thought synthesis, incubation/bridge, sleep/identity/speech, and agency."} diff --git a/state/map.yaml b/state/map.yaml index 465a537..4d0b012 100644 --- a/state/map.yaml +++ b/state/map.yaml @@ -149,4 +149,4 @@ rejected_paths: current_status: summary: VoidBot's ordinary bot/worker/provider/RAG/Postgres/Qdrant structure is still conceptually sound, and the moderation/mood/private-self-state rebuild has cut the worst mutation path: routine runner writes now cross a typed operation service instead of direct JSON projection commits. Remaining risk is that the legacy monolith and read-only projection still exist for compatibility, mood drift still refreshes that projection, and memory/sleep cleanup remains swollen compensator code. - next_action: JSON projection authority is no longer a routine moderation-runner write path, and mood drift now updates typed scheduled-runtime documents for sleep/speaking pressure. Next move memory-maintenance/incubation/candidate outputs behind typed operations, then delete `legacyJsonPath`/working-projection materialization and cut legacy mirror fields. Redesign sleep/distillation around a meaning-preservation contract before trusting it to compress memory. + next_action: JSON projection authority is no longer a routine moderation-runner write path, and mood drift now updates typed scheduled-runtime documents for sleep/speaking pressure. The memory organ split has begun: constants, utilities, vector/embedder substrate, and record intake are separate files. Next split the remaining policy organs by ownership, starting with cleanup/translation versus graph/incubation/agency, then move memory-maintenance/incubation/candidate outputs behind typed operations and delete `legacyJsonPath`/working-projection materialization.