From f3eb2548b1c584bd2f76b256dd92a8626c573da0 Mon Sep 17 00:00:00 2001 From: Meta Vens Date: Mon, 18 May 2026 01:19:57 +0100 Subject: [PATCH] Unify repo and agent memory graph plans --- notes/agent-memory-fractal-cache-plan.md | 839 ++++++++++++++++++++ notes/epiphany-memory-graph-unified-plan.md | 423 ++++++++++ notes/fresh-workspace-handoff.md | 3 +- notes/repo-fractal-dataflow-cache-plan.md | 130 ++- state/ledgers.msgpack | Bin 129055 -> 129434 bytes state/map.yaml | 20 +- 6 files changed, 1366 insertions(+), 49 deletions(-) create mode 100644 notes/agent-memory-fractal-cache-plan.md create mode 100644 notes/epiphany-memory-graph-unified-plan.md diff --git a/notes/agent-memory-fractal-cache-plan.md b/notes/agent-memory-fractal-cache-plan.md new file mode 100644 index 00000000..6af327f5 --- /dev/null +++ b/notes/agent-memory-fractal-cache-plan.md @@ -0,0 +1,839 @@ +# Agent Memory Fractal Cache Plan + +## Correction: One Memory Graph, Not Two Organs + +This note was drafted as a sibling to the repo-model plan. That split is +conceptually wrong if preserved as architecture. + +The repo fractal dataflow graph and agent memory are the same creature wearing +two wigs: + +- both store typed claims about a domain +- both have nodes, edges, anchors, summaries, freshness, confidence, and + lifecycle +- both need conservative summaries so context can collapse without lying +- both need semantic embeddings for fast recall +- both need review gates before provisional claims become doctrine +- both must survive Qdrant deletion with slower but correct typed traversal + +The correct foundation is a unified `EpiphanyMemoryGraph` organ with domain +profiles: + +- `repo_architecture` +- `repo_dataflow` +- `role_self` +- `conversation` +- `incubation` +- `agency_pressure` +- `candidate_intervention` +- `identity` +- `evidence` + +Repo-model and agent-memory are views, policies, and lifecycle profiles over +that shared substrate. They should not grow separate graph/cache/storage +engines. Separate engines would become duplicate authority with better naming. + +Keep this note as the role/agent-memory profile plan, but implement shared +graph/cache primitives first. + +This plan applies the same cache discipline as +`notes/repo-fractal-dataflow-cache-plan.md` to Epiphany's agent memories, using +VoidBot as source-grounded lineage. + +VoidBot's useful lesson is not "store more memories." It is that a mind needs +memory phases, ownership, sleep, revision, retirement, and semantic recall +without letting the vector store become the soul. + +```text +experience / role output / rumination / sleep pressure +-> typed memory operation +-> reviewed memory lifecycle law +-> durable CultCache memory documents +-> conservative memory summaries and clusters +-> Qdrant semantic resonance cache +-> memory context packet +-> role / Face / coordinator / sleep pass +``` + +The cache is allowed to help the machine remember. It is not allowed to decide +what is true. + +## Source-Grounded VoidBot Lessons + +VoidBot currently juggles several memory surfaces: + +- typed self-state in a CultCache `.cc` store through + `packages/core/src/void-self-state-service.ts` +- short-term rumination memory, clustered at the typed service boundary +- sleep-owned memory maintenance through `prompts/void-memory-maintenance.md` + and `scripts/run-void-memory-maintenance.ps1` +- durable memory lifecycle operations: + - `record_short_term_memory` + - `apply_memory_distillation` + - `revise_durable_memory` + - `retire_durable_memory` + - `crystallize_memory_into_identity` + - `merge_incubation_support` + - `queue_candidate_intervention` + - `upsert_agency_pressure` +- Qdrant collections split between Discord history and repository/source + vectors +- sleep enforcement: if short-term memory pressure exists, a real sleep pass + returning no operation fails; if sleep leaves short-term residue behind, that + also fails + +The useful invariants: + +- rumination writes provisional memory only +- sleep owns durable promotion, pruning, merging, retirement, and + crystallization +- durable memory is plastic, not immutable +- every memory needs a concrete target, claim/question, tension, action + implication, and anchors or an explicit `anchor:missing` +- model output crosses a typed operation boundary before state changes +- parent code applies or rejects operations; the model does not edit state + directly +- Qdrant accelerates recall; typed state owns memory truth + +The scars to avoid: + +- orchestration spread across scripts, prompts, status files, and Node helpers +- memory lifecycle law living partly in prose +- Qdrant payloads carrying too much canonical content +- short-term/durable/incubation/agency pressure sharing one ambiguous "memory" + noun + +Epiphany should keep the invariants and rebuild the mechanism as native Rust +typed documents plus CultNet contracts. + +## Current Epiphany Mechanism + +Epiphany already has partial organs: + +- `epiphany-core/src/agent_memory.rs` + - stores each role's Ghostlight-shaped dossier in `state/agents.msgpack` + - exposes typed `AgentSelfPatch` + - reviews/applies bounded lane memory, goal, value, and private-note patches + - rejects project truth, authority, graphs, checkpoints, jobs, planning, + scratch, code edits, and raw transcripts + +- `epiphany-core/src/bin/epiphany-agent-memory-store.rs` + - CLI for status, validation, review-patch, apply-patch, migration, and smoke + +- `epiphany-core/src/heartbeat_state.rs` + - derives memory resonance, incubation, thought lanes, bridge judgments, + candidate interventions, appraisals, reactions, and sleep/dream state + - currently stores many cognition surfaces as `serde_json::Value` in the + heartbeat cognition document + +- `schemas/canonical-agent-state-schema.md` + - states the role dossier boundary: project truth belongs in thread state; + dossiers hold role-local judgment + +This is a real foundation, but not yet the best version. The current machinery +has durable role dossiers and reviewed `selfPatch`, but it does not yet have a +typed memory lifecycle organ equivalent to VoidBot's short-term -> sleep -> +durable/revised/retired/identity flow. Resonance/incubation are also still +receipt-shaped JSON rather than first-class typed memory documents. + +## Target Architecture + +Add the agent-memory profile to the shared `EpiphanyMemoryGraph` organ: + +```text +role dossier memories / selfPatch requests / heartbeat rumination / sleep pass +-> typed memory operation proposals +-> shared memory graph lifecycle validator +-> typed CultCache memory graph documents +-> conservative summaries +-> Qdrant memory graph embeddings +-> resonance/incubation/context packet +-> role prompt / Face surface / coordinator / sleep maintenance +``` + +Durable truth stays in typed CultCache documents. Qdrant stores vectors keyed to +typed memory document ids and source hashes. Heartbeat cognition consumes memory +context packets instead of inventing its own free-form memory weather. + +## Core Inputs + +- Role dossiers from `state/agents.msgpack` +- Reviewed `AgentSelfPatch` requests +- Heartbeat rumination receipts +- Sleep-cycle pressure +- Role worker findings and accepted/rejected result receipts +- Face speech receipts and candidate interventions +- User-visible conversation summaries when explicitly reviewed +- Repo-model context packets when a memory is grounded in project architecture +- Evidence ids and anchor refs + +## Durable State Stores + +Use typed CultCache documents: + +- `epiphany.agent_memory.dossier` + - existing Ghostlight-shaped role dossier document + +- `epiphany.agent_memory.short_term` + - provisional rumination residue + - source role, target, claim/question, tension, action implication, anchors, + tags, salience, confidence, saturation, created/updated time + +- `epiphany.agent_memory.durable` + - accepted role memory with lifecycle status + - supports semantic, episodic, relationship, value, identity, doctrine, habit, + and dream categories + +- `epiphany.agent_memory.identity` + - crystallized self/profile memory that may project into dossier values, + private notes, or stable role doctrine only through explicit review + +- `epiphany.agent_memory.incubation` + - thought threads that are alive but not ready for durable memory or speech + +- `epiphany.agent_memory.agency_pressure` + - discomforts, self-advocacy pressure, world-advocacy pressure, and wiring + requests; not speech text + +- `epiphany.agent_memory.candidate_intervention` + - possible Face speech, coordinator note, selfPatch request, or future + review action + +- `epiphany.agent_memory.summary` + - conservative summaries over memory clusters, role-local themes, and + cross-role resonance + +- `epiphany.agent_memory.embedding_manifest` + - collection names, embedding model, vector dimensions, indexed memory ids, + source hashes, and stale records + +- `epiphany.agent_memory.context_packet` + - the memory neighborhood selected for a role turn, Face bubble, sleep pass, + or coordinator decision + +## Core Transformations + +### 1. Memory Operation Boundary + +All mutation enters as typed operations: + +- `RecordShortTermMemory` +- `ClusterShortTermMemory` +- `PruneShortTermMemory` +- `ApplyMemoryDistillation` +- `MergeIncubationSupport` +- `ReviseDurableMemory` +- `RetireDurableMemory` +- `CrystallizeMemoryIntoIdentity` +- `QueueCandidateIntervention` +- `RetireCandidateIntervention` +- `MarkCandidateInterventionSpoken` +- `UpsertAgencyPressure` +- `RetireAgencyPressure` +- `RefreshMemoryEmbeddingManifest` + +The model may propose operations. Parent/core code validates and applies them. + +### 2. Phase Ownership + +- Rumination may create or deepen short-term memory. +- Sleep must account for every short-term memory under pressure. +- Durable memory changes happen only through sleep maintenance or explicit + reviewed selfPatch acceptance. +- Identity crystallization is review-gated and rare. +- Agency pressure may influence readiness or speaking pressure, but may not + manufacture speech text by itself. + +### 3. Memory Graph + +Build a typed memory graph: + +- nodes: short-term, durable, identity, incubation, agency pressure, candidate + intervention, speech receipt, evidence anchor, repo-model anchor +- edges: distills, revises, retires, supports, contradicts, grounds, triggered, + spoken-as, cools, clusters-with, resonates-with + +The graph is the anatomy. Qdrant is the nerve signal. + +### 4. Conservative Summaries + +Every memory cluster/domain gets a summary: + +- target +- strongest claim/question +- live tension +- action implication +- anchors +- support count +- contradiction count +- salience/confidence range +- saturation/refractory state +- freshness + +If the summary cannot preserve meaning, it cannot replace child memories in a +context packet. + +### 5. Semantic Resonance Cache + +Embed typed memory documents and summaries into Qdrant: + +- point id: stable typed document id +- payload: document id, document type, role id, target, kind, lifecycle status, + source hash, schema version, salience/confidence, updated time, compact anchor + refs +- vector: summary/claim/tension/action text + +Do not store full canonical memory payloads only in Qdrant. The full payload +lives in CultCache. + +### 6. Context Packet Selection + +Given role id, task intent, recent stimulus, sleep mode, and context budget, +select: + +- directly relevant durable memories +- live short-term residue +- active incubation threads +- agency pressure obligations +- candidate interventions ready for review/speech +- contradictory or retired memories that prevent stale doctrine +- exact anchors/evidence refs required to preserve meaning + +## Core Outputs + +- `EpiphanyAgentMemoryContextPacket` + - role/task memory neighborhood + - selected memories and summaries + - active agency pressure + - incubation threads + - candidate interventions + - warnings for stale/missing embeddings + - required review gates + +- `EpiphanyAgentMemoryMaintenanceReceipt` + - sleep/rumination maintenance run receipt + - operations proposed, accepted, rejected, and applied + - short-term residue accountability + +- `EpiphanyAgentMemoryResonance` + - typed cross-memory/cross-role resonance edges and clusters + +- `EpiphanyAgentMemoryFreshness` + - stale vector entries, stale source anchors, old short-term residue, and + over-saturated themes + +## Module Ownership + +- `epiphany-state-model` + - owns public typed document structs if these documents need CultNet/schema + advertisement outside `epiphany-core` + +- `epiphany-core::agent_memory` + - keeps Ghostlight dossier storage, `AgentSelfPatch` review/application, and + role-local patch policy + +- `epiphany-core::memory_graph` + - owns shared memory lifecycle documents, operation validation/application, + graph law, summaries, resonance, freshness, context packet planning, and + Qdrant-backed cache integration + +- `epiphany-core::memory_graph::profiles::agent` + - owns role-self, short-term, incubation, agency-pressure, + candidate-intervention, identity, and sleep-accountability policy over the + shared graph + +- `epiphany-core::heartbeat_state` + - consumes memory context/resonance packets for pacing, appraisal, rumination, + and sleep + - should stop owning untyped cognition memory blobs as stable machinery + +- `epiphany-openai-runtime` + - may run sleep/maintenance prompts, but writes only typed operation proposal + documents and receipts + +- `epiphany-runtime-spine` + - advertises memory maintenance and context query contracts over CultNet + +- Aquarium + - displays memory state, sleep receipts, incubation, agency pressure, and + candidate interventions; it is not the memory source of truth + +## Invariants + +- Role dossiers are self truth, not project truth. +- Short-term memory is provisional and must not survive sleep unchanged when + sleep maintenance is forced. +- Durable memory is plastic: it may be revised, retired, or crystallized through + typed operations. +- Every memory must preserve target, claim/question, tension, action + implication, and anchors or explicit `anchor:missing`. +- Model-generated memory changes are proposals until core validation applies + them. +- Qdrant is semantic recall/cache, not canonical memory. +- Resonance is evidence of similarity/pressure, not proof. +- Agency pressure is not speech text and not authority. +- Sleep can reduce bulk, never meaning. +- No memory operation may edit code, project graph truth, planning authority, + runtime lifecycle, or checkpoints. + +## Architectural Smells To Avoid + +### One Big Memory Blob + +Current temptation: extend role dossiers with more arrays until everything fits. + +Real need: compact role-local identity and useful durable memory. + +What breaks if deleted: convenience for reading one file. + +Verdict: not essential. It hides lifecycle ownership. + +Simpler architecture: split dossier, short-term, durable, incubation, agency +pressure, candidate, summary, and embedding manifest documents. + +### Sleep As A Vibe + +Current temptation: let sleep be a prompt that "reflects" and maybe writes +something. + +Real need: forced accounting for residue. + +What breaks if deleted: fake dream flavor. + +Verdict: sleep is essential only if it enforces lifecycle law. + +Simpler architecture: sleep maintenance receives a typed memory context packet +and must account for every short-term item under pressure. + +### Qdrant As The Brain + +Current temptation: put full memories in Qdrant payloads and retrieve them as +truth. + +Real need: fast resonance and recall. + +What breaks if deleted: latency and semantic ranking. + +Verdict: if Qdrant deletion loses memory, ownership is wrong. + +Simpler architecture: CultCache typed documents own memory; Qdrant points carry +ids, hashes, compact metadata, and vectors. + +### SelfPatch As Universal Mutation + +Current temptation: keep expanding `AgentSelfPatch`. + +Real need: reviewed lane-local memory/goal/value/private-note mutation. + +What breaks if deleted: role result self-persistence path. + +Verdict: keep it bounded. Do not turn it into memory lifecycle, sleep, agency, +and incubation all at once. + +Simpler architecture: selfPatch remains lane-local reviewed mutation; memory +maintenance uses explicit lifecycle operations. + +### Heartbeat Owning Memory Weather + +Current mechanism: heartbeat routine currently builds resonance/incubation as +JSON-shaped cognition surfaces. + +Real need: pacing/appraisal input and routine receipts. + +What breaks if deleted: current routine status projection. + +Verdict: useful but in the wrong shape. + +Simpler architecture: heartbeat consumes typed memory context/resonance +documents and writes typed routine receipts, not free-form memory authority. + +## Ranked Teardown Plan + +### Keep + +- Ghostlight-shaped role dossiers as the compact role body. +- Existing reviewed `AgentSelfPatch` boundary for lane-local memory/goal/value + mutations. +- Completion-gated heartbeat cooldown and sleep/rumination physiology. +- VoidBot's phase law: rumination provisional, sleep durable. +- Qdrant/Ollama as semantic cache backend. + +### Cut + +- Any future memory design where Qdrant payload is the only memory payload. +- Any sleep pass that can ignore short-term pressure and still report success. +- Any model path that edits memory store directly. +- Any memory operation that stores project graph/planning/runtime truth inside a + role dossier. + +### Collapse + +- Memory freshness, resonance freshness, and embedding manifest freshness should + collapse into one agent-memory freshness surface. +- Current heartbeat cognition memory resonance/incubation JSON should collapse + into typed agent-memory model documents plus routine receipts. + +### Split + +- Split `agent_memory.rs`: + - dossier documents and store + - selfPatch review/application + - memory lifecycle model + - context packet projection + - vector cache adapter +- Split heartbeat stable scheduler state from memory cognition receipts. + +### Rebuild + +- Build `epiphany-core::memory_graph` as the shared lifecycle organ before + adding vector memory or profile-specific producers. +- Build the agent profile and native sleep maintenance command on top of that + shared graph, consuming typed memory context and applying typed operations + with fixtures. + +## Implementation Roadmap + +### Phase 0: Contract Inventory + +Objective: map live memory state and avoid designing over fantasy. + +Read: + +- `epiphany-core/src/agent_memory.rs` +- `epiphany-core/src/bin/epiphany-agent-memory-store.rs` +- `epiphany-core/src/heartbeat_state.rs` +- `schemas/canonical-agent-state-schema.md` +- `schemas/heartbeat-state-schema.md` +- VoidBot: + - `packages/core/src/void-self-state-service.ts` + - `prompts/void-memory-maintenance.md` + - `notes/voidbot-current-system-map.md` + - `packages/rag/src/qdrant-vector-store.ts` + +Exit criteria: + +- This note is kept current with any source-grounded correction. + +### Phase 1: Shared Memory Graph Profile Types + +Files: + +- `epiphany-state-model/src/lib.rs` +- `epiphany-core/src/memory_graph.rs` +- `epiphany-core/src/memory_graph/documents.rs` +- `epiphany-core/src/memory_graph/profiles/agent.rs` + +Add structs: + +- `EpiphanyAgentShortTermMemory` +- `EpiphanyAgentDurableMemory` +- `EpiphanyAgentIdentityMemory` +- `EpiphanyAgentIncubationThread` +- `EpiphanyAgentAgencyPressure` +- `EpiphanyAgentCandidateIntervention` +- `EpiphanyAgentMemorySummary` +- `EpiphanyAgentMemoryEmbeddingManifest` +- `EpiphanyAgentMemoryContextPacket` +- `EpiphanyAgentMemoryMaintenanceReceipt` + +Tests: + +- serde round trip +- required target/claim/tension/action/anchor validation +- lifecycle status enum coverage +- empty/default serialization stays lean + +Exit criteria: + +- Shared memory graph documents compile and validate agent-memory profile + records without Qdrant or model calls. + +### Phase 2: Operation Law + +Files: + +- `epiphany-core/src/memory_graph/operations.rs` +- `epiphany-core/src/memory_graph/validation.rs` +- `epiphany-core/src/memory_graph/profiles/agent.rs` + +Add operations: + +- record/cluster/prune short-term +- apply distillation +- merge incubation support +- revise/retire durable memory +- crystallize identity memory +- queue/retire/mark candidate intervention +- upsert/retire agency pressure + +Tests: + +- rumination operation cannot write durable memory +- sleep operation must account for all short-term records when forced +- revision retires superseded durable memories with a reason +- crystallization writes identity memory without directly rewriting trait means +- agency pressure requires target, claim/question, implication, intensity, and + anchors or `anchor:missing` +- operation touching project truth is rejected + +Exit criteria: + +- Parent/core memory law exists as pure Rust and can reject bad model output. + +### Phase 3: CultCache Store And CLI + +Files: + +- `epiphany-core/src/memory_graph/store.rs` +- `epiphany-core/src/bin/epiphany-agent-memory-store.rs` + +Commands: + +```powershell +cargo run --manifest-path .\epiphany-core\Cargo.toml --bin epiphany-agent-memory-store -- memory-status --store .\state\agent-memory.msgpack +cargo run --manifest-path .\epiphany-core\Cargo.toml --bin epiphany-agent-memory-store -- memory-apply-operation --store .\state\agent-memory.msgpack --operation +cargo run --manifest-path .\epiphany-core\Cargo.toml --bin epiphany-agent-memory-store -- memory-context --store .\state\agent-memory.msgpack --role-id modeling --query "graph ownership" +``` + +Tests: + +- MessagePack round trip for all memory lifecycle documents +- apply-operation writes expected typed documents +- context over fixture store returns selected memory packet + +Exit criteria: + +- Agent memory lifecycle state can be persisted and queried without heartbeat. + +### Phase 4: Short-Term Clustering And Sleep Accountability + +Files: + +- `epiphany-core/src/memory_graph/profiles/agent/clustering.rs` +- `epiphany-core/src/memory_graph/profiles/agent/sleep.rs` + +Behavior: + +- repeated target/topic pressure updates one provisional memory +- anchors/tags accumulate +- unchanged paraphrases do not stack +- forced sleep fails if short-term residue remains unaccounted +- sleep may distill, merge into incubation, or prune with reason + +Tests: + +- repeated short-term records cluster by role/target/topic +- fresh anchors deepen a cluster +- sleep with forced pressure and no operations fails +- sleep leaving source short-term ids alive fails + +Exit criteria: + +- Epiphany has VoidBot's most important memory invariant natively. + +### Phase 5: Conservative Summaries And Memory Graph + +Files: + +- `epiphany-core/src/memory_graph/summary.rs` +- `epiphany-core/src/memory_graph/context_cut.rs` +- `epiphany-core/src/memory_graph/profiles/agent.rs` + +Behavior: + +- create memory graph nodes and edges +- summarize clusters without losing target/claim/tension/action/anchors +- context packet can use summaries when child detail is unnecessary +- retired/contradictory memories can be included to prevent stale doctrine + +Tests: + +- summary cannot omit action implication +- retired memories are excluded by default but included when contradiction guard + is requested +- context packet descends when summary confidence is low or agency pressure is + high + +Exit criteria: + +- Memory context packets work without Qdrant. + +### Phase 6: Qdrant Memory Cache + +Files: + +- `epiphany-core/src/memory_graph/embedding_cache.rs` +- `epiphany-core/src/retrieval/embedding.rs` after retrieval split +- `epiphany-core/src/retrieval/qdrant.rs` after retrieval split + +Behavior: + +- embed durable, short-term, incubation, agency, candidate, and summary docs +- store vectors keyed to typed doc ids and source hashes +- payload contains only compact metadata +- query planner uses Qdrant for candidate ranking when ready +- missing/stale Qdrant falls back to typed memory graph traversal + +Tests: + +- Qdrant hit for missing typed doc id is ignored +- deleting Qdrant preserves correctness +- stale source hash schedules refresh +- vector ranking cannot invent memory packet contents + +Exit criteria: + +- Semantic resonance is fast but rebuildable. + +### Phase 7: Sleep Maintenance Runner + +Files: + +- `epiphany-core/src/bin/epiphany-agent-memory-maintenance.rs` +- `epiphany-core/src/prompts/agent_memory_maintenance.md` +- `epiphany-openai-runtime` integration only through typed model request/result + documents + +Behavior: + +- build typed maintenance context packet +- render prompt with allowed operations +- model returns typed operation proposals +- parent/core validates and applies +- real sleep pass under pressure cannot return no-op +- receipt records proposed/applied/rejected operations + +Tests: + +- fake model fixture promotes short-term to durable memory +- fake model fixture merges into incubation and prunes source +- invalid operation is rejected and recorded +- no-op under forced sleep fails + +Exit criteria: + +- Sleep maintenance is native and auditable. + +### Phase 8: Heartbeat Integration + +Files: + +- `epiphany-core/src/heartbeat_state.rs` +- `epiphany-core/src/heartbeat_state/heartbeat_cognition.rs` once split +- `epiphany-core/src/heartbeat_state/heartbeat_store.rs` + +Behavior: + +- heartbeat routine consumes typed memory context/resonance packets +- appraisals/reactions use memory pressure and agency pressure +- heartbeat no longer owns durable memory cognition as generic JSON +- sleep cycle triggers memory maintenance through typed intent/receipt, not + direct store mutation + +Tests: + +- heartbeat sleep turn opens memory maintenance intent under pressure +- heartbeat routine can run with missing Qdrant +- appraisals respond to agency pressure without creating speech text + +Exit criteria: + +- Heartbeat is physiology; agent-memory model owns memory lifecycle. + +### Phase 9: CultNet And Aquarium Surface + +Files: + +- `epiphany-core/src/runtime_spine.rs` +- `epiphany-core/src/bin/epiphany-runtime-spine.rs` +- `schemas/cultnet/` + +Advertise contracts: + +- `epiphany.agent_memory.status` +- `epiphany.agent_memory.context_query` +- `epiphany.agent_memory.context_packet` +- `epiphany.agent_memory.operation_intent` +- `epiphany.agent_memory.operation_receipt` +- `epiphany.agent_memory.maintenance_intent` +- `epiphany.agent_memory.maintenance_receipt` + +Aquarium displays: + +- active short-term memory +- durable memory clusters +- retired/revised lineage +- incubation threads +- agency pressure +- candidate interventions +- sleep maintenance receipts +- vector freshness + +Exit criteria: + +- Operator can inspect the brain without opening raw model transcripts. + +### Phase 10: Role Prompt Integration + +Behavior: + +- role worker launches receive memory context packets +- Face receives candidate/agency/incubation context without direct transcript + leakage +- coordinator sees pressure and receipts, not raw thought streams +- `AgentSelfPatch` remains a narrow lane-local mutation surface + +Tests: + +- prompt rendering includes bounded memory packet +- stale memory packet warns +- role cannot receive another role's private memories unless explicitly shared +- project truth does not enter memory context as doctrine + +Exit criteria: + +- Agent memory is a living typed system, not a transcript paste bucket. + +## First Implementable Ticket + +Do not start here directly anymore. Start with +`notes/epiphany-memory-graph-unified-plan.md`. + +Title: `Add shared EpiphanyMemoryGraph typed documents and pure validation` + +Scope: + +- Add shared memory graph documents in `epiphany-state-model/src/lib.rs`. +- Add `epiphany-core/src/memory_graph.rs` plus document, id, validation, + freshness, lifecycle, and context-cut submodules. +- Include `role_self`, `short_term`, `incubation`, `agency_pressure`, + `candidate_intervention`, and `identity` profile enums/policy hooks, but do + not implement sleep maintenance yet. +- Export only pure APIs from `epiphany-core/src/lib.rs` +- Add tests for required memory anatomy, profile legality, summary + conservativeness, and operation rejection + +Out of scope: + +- no Qdrant +- no scanner/repo-model integration +- no prompt/model runner +- no heartbeat integration +- no Aquarium surface +- no migration of existing `state/agents.msgpack` + +Verification: + +```powershell +cargo fmt --manifest-path .\epiphany-core\Cargo.toml +cargo test --manifest-path .\epiphany-core\Cargo.toml --lib memory_graph +cargo check --manifest-path .\epiphany-core\Cargo.toml +``` + +Definition of done: + +- Epiphany has shared typed memory-graph bones. +- A bad operation cannot smuggle project truth into memory. +- A sleep-forced no-op can be rejected before any model-runner exists. +- The next ticket can add CultCache persistence without redesigning the memory + law. diff --git a/notes/epiphany-memory-graph-unified-plan.md b/notes/epiphany-memory-graph-unified-plan.md new file mode 100644 index 00000000..bc97a208 --- /dev/null +++ b/notes/epiphany-memory-graph-unified-plan.md @@ -0,0 +1,423 @@ +# Epiphany Memory Graph Unified Plan + +The repo fractal dataflow graph and agent memory are one architecture. + +They differ in policy, not substrate. A source module, a durable role memory, an +incubating thought, an agency pressure, and an evidence record are all typed +claims in a domain. They need anchors, edges, lifecycle, summaries, freshness, +and semantic recall. Building separate engines for them would duplicate +authority and then charge us rent forever. + +The shared organ is `EpiphanyMemoryGraph`. + +```text +typed domain claim +-> memory graph node / edge / anchor +-> lifecycle state +-> conservative summary +-> embedding manifest +-> semantic cache +-> context cut +-> role/repo/Face/coordinator packet +``` + +## Core Principle + +Memory is not a bag of text. Memory is a graph of typed claims with lifecycle. + +Qdrant is not memory. Qdrant is the fast semantic nerve wrapped around memory. +It may rank candidate nodes and summaries. It may not be the only place a claim +exists. + +## Domain Profiles + +The substrate is shared. Profiles define policy. + +### `repo_architecture` + +Owns: + +- modules +- crates +- binaries +- schemas +- runtime contracts +- adapters +- test seams +- ownership/invariant claims + +Lifecycle: + +- observed +- proposed +- accepted +- stale +- retired + +### `repo_dataflow` + +Owns: + +- inputs +- durable stores +- transformations +- outputs +- read/write/derive/adapt/persist edges +- runtime and wire contracts + +Lifecycle: + +- observed +- proposed +- accepted +- stale +- retired + +### `role_self` + +Owns: + +- lane-local semantic memory +- episodic memory +- relationship memory +- goals +- values +- habits +- private notes + +Lifecycle: + +- proposed +- accepted +- revised +- retired +- crystallized + +### `short_term` + +Owns: + +- rumination residue +- provisional thoughts +- fresh anchors +- unresolved local tensions + +Lifecycle: + +- active +- clustered +- distilled +- incubated +- pruned + +### `incubation` + +Owns: + +- thoughts not ready for durable memory, speech, or action +- support memories +- open questions +- saturation/refractory state + +Lifecycle: + +- active +- deepening +- cooling +- promoted +- retired + +### `agency_pressure` + +Owns: + +- discomforts +- self-advocacy requests +- world-advocacy requests +- wiring requests + +Lifecycle: + +- active +- obligated +- cooled +- answered +- retired + +### `candidate_intervention` + +Owns: + +- possible Face speech +- possible coordinator note +- possible selfPatch +- possible future review action + +Lifecycle: + +- queued +- deferred +- spoken +- applied +- retired + +### `evidence` + +Owns: + +- verifier receipts +- source anchors +- artifact refs +- accepted/rejected scars + +Lifecycle: + +- observed +- reviewed +- accepted +- contradicted +- superseded + +## Shared Typed Documents + +First-class documents: + +- `EpiphanyMemoryGraphSnapshot` +- `EpiphanyMemoryDomain` +- `EpiphanyMemoryNode` +- `EpiphanyMemoryEdge` +- `EpiphanyMemoryAnchor` +- `EpiphanyMemorySummary` +- `EpiphanyMemoryLifecycleReceipt` +- `EpiphanyMemoryEmbeddingManifest` +- `EpiphanyMemoryFreshness` +- `EpiphanyMemoryContextQuery` +- `EpiphanyMemoryContextPacket` +- `EpiphanyMemoryPatchCandidate` + +Node minimum: + +- `id` +- `domain_id` +- `profile` +- `kind` +- `title` +- `claim` +- `question` +- `tension` +- `action_implication` +- `anchors` +- `source_hashes` +- `lifecycle` +- `salience` +- `confidence` +- `created_at` +- `updated_at` + +Edge minimum: + +- `id` +- `source_id` +- `target_id` +- `kind` +- `profile` +- `claim` +- `anchors` +- `lifecycle` +- `confidence` + +Summary minimum: + +- `id` +- `domain_id` +- `covers_node_ids` +- `covers_edge_ids` +- `target` +- `claim` +- `tension` +- `action_implication` +- `anchor_count` +- `source_hashes` +- `freshness` +- `confidence` +- `known_omissions` + +## Shared Operations + +Generic graph operations: + +- `upsert_node` +- `upsert_edge` +- `retire_node` +- `revise_node` +- `merge_nodes` +- `attach_anchor` +- `summarize_cluster` +- `mark_stale` +- `refresh_embedding` +- `select_context_cut` + +Profile-specific operations: + +- repo profiles may propose architecture/dataflow graph patches +- role profiles may apply reviewed selfPatch-derived memory +- short-term profile may cluster/prune/distill +- incubation profile may deepen/cool/promote/retire +- agency profile may queue obligation or cool pressure +- candidate profile may mark spoken/applied/retired + +The validator enforces profile law before generic graph mutation. + +## Context Packets + +A context packet is a cut through the memory graph. + +It may contain: + +- exact nodes +- exact edges +- conservative summaries +- anchors/evidence refs +- stale/missing warnings +- contradiction guards +- verification seams +- lifecycle receipts +- profile-specific obligations + +Repo work gets architecture/dataflow-heavy packets. + +Role work gets role_self/short_term/incubation/agency-heavy packets. + +Face gets candidate/agency/incubation packets with public-surface filters. + +Coordinator gets pressure, stale state, obligations, and receipts. + +Same graph. Different cuts. + +## Invariants + +- One memory graph substrate owns typed claims. +- Profiles own policy and lifecycle restrictions. +- Qdrant stores vectors for graph documents, not canonical graph documents. +- Deleting Qdrant leaves a slower but correct typed graph. +- Model output proposes operations; core validation applies or rejects. +- Short-term memory cannot survive forced sleep unaccounted. +- Repo graph candidates cannot become accepted architecture truth without + review. +- Role memory cannot store project truth. +- Evidence anchors must remain distinguishable from remembered claims. +- Conservative summaries must preserve target, claim/question, tension, + implication, anchors, freshness, and known omissions. + +## Implementation Roadmap + +### Phase 1: Shared Memory Graph Types + +Files: + +- `epiphany-state-model/src/lib.rs` +- `epiphany-core/src/memory_graph.rs` +- `epiphany-core/src/memory_graph/documents.rs` + +Add shared document structs and enums. + +Tests: + +- serde round trip +- stable id helpers +- edge references existing node ids +- summaries cannot cover missing ids +- profile enum covers repo and agent memory domains + +### Phase 2: Shared Validation And Context Cut + +Files: + +- `epiphany-core/src/memory_graph/validation.rs` +- `epiphany-core/src/memory_graph/context_cut.rs` +- `epiphany-core/src/memory_graph/freshness.rs` + +Tests: + +- parent summary used when fresh/high-confidence +- descent required when stale/low-confidence/high-relevance +- Qdrant absence is irrelevant to correctness +- stale source hash propagates through summaries + +### Phase 3: Repo Profile + +Files: + +- `epiphany-core/src/memory_graph/profiles/repo.rs` +- `notes/repo-fractal-dataflow-cache-plan.md` + +Implements repo domain/node/edge policy on shared graph. + +### Phase 4: Agent Memory Profile + +Files: + +- `epiphany-core/src/memory_graph/profiles/agent.rs` +- `epiphany-core/src/agent_memory_model.rs` +- `notes/agent-memory-fractal-cache-plan.md` + +Implements short-term/durable/incubation/agency/candidate policy on shared +graph. + +### Phase 5: CultCache Store + +Store shared graph docs as typed CultCache documents. + +Commands should query memory graph status and context packets without Qdrant. + +### Phase 6: Qdrant Cache + +Embed shared graph nodes, edges, and summaries. + +Qdrant candidate ranking feeds context-cut planning but never emits canonical +memory payloads. + +### Phase 7: Sleep And Repo Refresh + +Sleep maintenance and repo scanning become profile-specific producers of typed +memory graph operations. + +### Phase 8: CultNet And Prompt Integration + +Advertise memory graph read/mutation/context contracts through runtime-spine. +Render context packets into role prompts and operator surfaces. + +## First Implementable Ticket + +Title: `Add shared EpiphanyMemoryGraph typed documents and pure validation` + +Scope: + +- Add shared memory graph document types. +- Add stable id helpers. +- Add pure validation for nodes, edges, summaries, anchors, and profile/lifecycle + legality. +- Add pure context-cut fixture tests. +- Update repo/agent memory plans to point at this substrate. + +Out of scope: + +- no Qdrant +- no scanner +- no sleep runner +- no app-server route +- no prompt integration + +Verification: + +```powershell +cargo fmt --manifest-path .\epiphany-core\Cargo.toml +cargo test --manifest-path .\epiphany-core\Cargo.toml --lib memory_graph +cargo check --manifest-path .\epiphany-core\Cargo.toml +``` + +Definition of done: + +- The shared creature has a skeleton. +- Repo modeling and agent memory can add profiles without duplicating graph, + cache, freshness, or context-packet machinery. diff --git a/notes/fresh-workspace-handoff.md b/notes/fresh-workspace-handoff.md index 4fbb81fe..ed0097d0 100644 --- a/notes/fresh-workspace-handoff.md +++ b/notes/fresh-workspace-handoff.md @@ -39,7 +39,7 @@ remembers doctrine; the branch remembers the blade. - `epiphany-openai-runtime` worker result ingress now parses assistant output directly into typed role/reorient ingress structs selected from the durable launch document, stores nested `statePatch` and `selfPatch` as typed MessagePack in `EpiphanyRuntimeRoleWorkerResult`, and no longer passes worker result data through generic `serde_json::Value`. JSON survives there only as the OpenAI text-output edge before typed deserialization. - Agent self-memory review/application now has typed `AgentSelfPatch` core APIs. JSON-value selfPatch functions are compatibility wrappers for CLI/artifact ingress; native callers should parse once at the edge and then call `review_agent_self_patch_document` / `apply_agent_self_patch_document`. Role-result self-persistence no longer exports a JSON-value reviewer; the live API is `review_role_self_patch_document`. - `epiphany-openai-codex-spine` no longer uses Codex's Responses client, model-provider layer, `codex-api`, `codex-model-provider`, `codex-model-provider-info`, or `codex-protocol` for outbound model requests. It reaches `codex-login` only through `epiphany-openai-auth-spine` so credential loading/refresh stays Codex-compatible while request/status/event/runtime surfaces stay typed Epiphany documents. -- `notes/repo-fractal-dataflow-cache-plan.md` is the current repo-model direction drawn from `Fractal Domains And The Cache That Bites`: the repo's semantic architecture/dataflow graph should be a typed CultCache document organ, with Qdrant storing rebuildable embeddings for nodes, edges, conservative summaries, and context cuts. Qdrant is a contribution cache, not truth. The roadmap's first implementable ticket is typed RepoModel documents plus pure `epiphany-core::repo_model` id/validation/freshness/context-cut tests; scanner, CultCache persistence, Qdrant, CultNet advertisement, and coding-lane integration come after the typed bones exist. +- `notes/epiphany-memory-graph-unified-plan.md` is the current memory architecture direction drawn from `Fractal Domains And The Cache That Bites` plus the VoidBot memory scar tissue. The repo semantic architecture/dataflow graph and agent memory are one `EpiphanyMemoryGraph` substrate with repo, role, short-term, incubation, agency, candidate-intervention, and evidence profiles. Qdrant is a contribution cache for typed graph documents, not truth. The first implementable ticket is shared memory graph documents plus pure `epiphany-core::memory_graph` id/validation/freshness/context-cut tests; profile-specific scanners, sleep consolidation, CultCache persistence, Qdrant, CultNet advertisement, and prompt integration come after the shared skeleton exists. `notes/repo-fractal-dataflow-cache-plan.md` and `notes/agent-memory-fractal-cache-plan.md` are profile plans, not rival organs. - Durable Epiphany state types now live in root crate `epiphany-state-model`. `codex-protocol` re-exports them only so old rollout/API callers keep compiling; do not add new durable `Epiphany*` state structs under `vendor/codex`. `epiphany-core` imports the native model directly and no longer depends on `codex-protocol`. Pressure derives from native `EpiphanyTokenUsageSnapshot`; Codex token telemetry is mapped at the bridge. Codex rollout/event reconstruction now lives in `codex-core::epiphany_rollout`, where the Codex `RolloutItem` / `EventMsg` types belong. Exact retrieval path matching is native to `epiphany-core`, so `epiphany-core` has no direct vendored Codex crate dependency. - State-update contract proof and job launch/interrupt DTO ownership now live in native `epiphany-core`. The old giant `codex_thread.rs` Epiphany state-update test fixture was cut, and `epiphany-codex-bridge` no longer imports Epiphany job launch request types from `codex_core`; vendored `CodexThread` may call the native mutation/launch law for compatibility, but it must not regain ownership of state-update semantics, heartbeat launch-plan policy, or Epiphany DTOs. - MCP remains JSON because MCP is a JSON protocol. The latest processor cut only moved MCP refresh/OAuth/status/resource/tool-call route handling into `codex_message_processor/mcp_routes.rs`, reducing the root processor to about 8,977 lines; it did not claim MCP payloads are CultNet or make MCP an Epiphany-owned runtime contract. @@ -96,6 +96,7 @@ remembers doctrine; the branch remembers the blade. - The heartbeat initiative pass is landed as a bounded tool seam. `state/agent-heartbeats.msgpack` tracks Self, Face, Imagination, Eyes, Body, Hands, Soul, and Life as Ghostlight-style initiative participants with arena, participant kind, speed, next-ready time, reaction bias, interrupt threshold, load, status, constraints, personality cooldown, mood cooldown, effective cooldown, adaptive pacing, history, and pending turns through `epiphany-core::EpiphanyHeartbeatStateEntry` and the native `epiphany-heartbeat-store` binary. Role dossiers set baseline timing; appraisal mood/anxiety bends it through urgency, arousal, thought pressure, guardedness, and reaction intensity, so Hands-like work pressure and high-need anxious lanes recover sooner without being heartbeaten while a prior turn is running. `epiphany-heartbeat-store pump` computes pressure from external urgency, mood/anxiety, reaction intensity, thought pressure, and current pending load, then chooses tempo plus target concurrency: calm can launch zero and sleep slow; alarm can fill most active lanes. Idle turns are for rumination: light thought shuffling, role-quality attention, and candidate selfPatch pressure. Sleep/dream cycle passes are the intended distillation window for durable self-memory and doctrine. JSON heartbeat state and Python wrapper state are gone; general CultCache schema sync, polyglot domain loading, and debug display tools belong in CultLib. This is a callable scheduler seam, not yet an always-on daemon; a heart valve, not a whole circulatory god. - The first Ghostlight-derived timing slice is landed in Epiphany, but Ghostlight is reference lineage rather than a sibling runtime to preserve. `epiphany-heartbeat-store init --profile ghostlight-scene --scene-id --scene-participant ` creates a typed CultCache MessagePack scene heartbeat store whose participants are `arena=scene`, `participantKind=character`; `tick` emits `ghostlight.initiative_schedule.v0` receipts with `scene_turn` actions and local-affordance basis. The generic Epiphany maintenance lanes are not auto-patched into scene stores. - The first Void-derived routine slice is landed in Epiphany, but VoidBot is reference lineage rather than a runtime dependency. `epiphany-heartbeat-store routine --store .\state\agent-heartbeats.msgpack --artifact-dir .\.epiphany-heartbeats --agent-store .\state\agents.msgpack` reads typed role dossiers, computes bounded memory resonance, maintains incubation themes, runs analytic and associative cognition lanes, writes bridge syntheses/saturation/tension, projects the active thought cluster through each role's Ghostlight-shaped personality vectors, derives participant-local reactions, applies personality/mood timing, advances the sleep/dream cycle, updates the typed heartbeat store, and emits an auditable `epiphany.void_routine.v0` receipt. The routine now also carries anti-loop physiology imported from Void's calmer brain: `noveltyToSelf`, `noveltyToRoom`, source coverage, saturation pressure, refractory cooling, and explicit permission to let a live unsaturated thought deepen without forcing novelty theater every pass. This mutates only heartbeat physiology fields: project truth and role memory mutation remain on their reviewed surfaces. +- The next memory foundation must not build separate engines for repo graph and agent mind. Heartbeat cognition, role self-memory, repo architecture/dataflow, evidence scars, incubation, agency pressure, and context packets should converge through `EpiphanyMemoryGraph` profiles so one typed graph owns anchors, lifecycle, summaries, freshness, context cuts, and Qdrant embedding manifests. - Manual Codex-run rumination still has an explicit aftercare rule until Epiphany owns the full sleep consolidator. In the intended Epiphany cycle, sub-agents ruminate when idle and distill when they sleep; in this supervising Codex thread, a heartbeat/routine vigil is only physiology until a separate closing pass reviews the receipts and decides whether map, handoff, evidence, or role self-memory should change. - The Face public-surface pass is landed as a bounded lane. Face's role dossier lives in `state/agents.msgpack`; `epiphany_specialists.toml` gives it a VoidBot-heartbeat-derived prompt stripped of moderation authority; `state/face-discord.toml` and the native `epiphany-face-discord` binary enforce that Face may interact only through #aquarium. Missing channel id or token writes candidate chat artifacts instead of posting elsewhere. If a Face has a persona name/avatar, `post` uses the shared Discord webhook pipe so each Epiphany can speak with its own nickname and avatar without a separate bot identity. - The Void memory bridge is now native enough to keep the useful organs attached. `state/void-memory.toml` and `epiphany-void-memory` can check Void's Docker Postgres state spine, query Qdrant Discord-history and source/lore collections with the configured Ollama embedding model, and fetch raw archive context windows from Void's file-backed Discord archive. Native `epiphany-mvp-status` includes a `voidMemory` block so Aquarium/backend clients can inspect whether the mouth and memory are actually wired. diff --git a/notes/repo-fractal-dataflow-cache-plan.md b/notes/repo-fractal-dataflow-cache-plan.md index 481d79e9..56367aed 100644 --- a/notes/repo-fractal-dataflow-cache-plan.md +++ b/notes/repo-fractal-dataflow-cache-plan.md @@ -1,5 +1,33 @@ # Repo Fractal Dataflow Cache Plan +## Correction: This Is A Memory Graph Profile + +The repo fractal dataflow graph is not separate from agent memory. It is a +domain profile of the same underlying machine. + +The shared creature is an `EpiphanyMemoryGraph`: + +```text +typed domain claim +-> node / edge / anchor / summary +-> lifecycle state +-> conservative context cut +-> semantic embedding cache +-> role/repo/Face/coordinator context packet +``` + +The repo graph uses this substrate for architecture/dataflow/source domains. +Agent memory uses it for role-local self memories, short-term residue, +incubation, identity, agency pressure, and candidate interventions. Evidence +and planning can also attach as graph domains. + +Do not implement a standalone repo-model graph engine and then another +agent-memory graph engine. That is duplicated authority. Build shared typed +memory graph primitives, then add domain-specific policies. + +This note remains the `repo_architecture` / `repo_dataflow` profile plan. The +agent-memory profile plan lives in `notes/agent-memory-fractal-cache-plan.md`. + This note applies the lesson from `E:\Projects\gamecult-site\GameCult\Blog\fractal-domains-cache-that-bites.md` to Epiphany's repo-modeling architecture. @@ -42,14 +70,15 @@ better coat. ## Desired Mechanism -Introduce an Epiphany-owned `RepoModel` organ: +Add repo architecture/dataflow profiles to the shared +`EpiphanyMemoryGraph` organ: ```text repo files / docs / manifests / build metadata / git history -> typed repo scanner -> typed code/document domains --> fractal architecture/dataflow graph documents --> conservative node and edge summaries +-> shared memory graph architecture/dataflow nodes and edges +-> conservative memory graph summaries -> Qdrant embeddings for nodes, edges, summaries, and candidate neighborhoods -> query planner chooses a stable context cut -> coding agent receives exact typed context packet @@ -71,7 +100,8 @@ It makes selection fast. It does not become the source of truth. ## Durable State Stores -Keep truth in typed CultCache documents: +Keep truth in typed CultCache documents. These are repo-profile projections of +the shared memory graph documents, not a separate graph store: - `epiphany.repo_model.snapshot`: repo identity, scan revision, source hashes, root domains, and model status. @@ -156,12 +186,16 @@ must not be the only home of graph truth. Target module network: - `epiphany-state-model` - - owns typed document structs for repo model snapshots, nodes, edges, - summaries, embedding manifests, context packets, freshness, and receipts. + - owns shared memory graph document structs plus repo-profile public contract + projections when they need CultNet/schema advertisement. -- `epiphany-core::repo_model` - - owns domain mapping, graph grammar, summary construction, freshness, - context-cut planning, validation, and patch proposal policy. +- `epiphany-core::memory_graph` + - owns shared graph grammar, lifecycle law, summary construction, freshness, + context-cut planning, validation, and Qdrant embedding manifests. + +- `epiphany-core::memory_graph::profiles::repo` + - owns repo architecture/dataflow domain mapping, scanner policy, source-hash + freshness, and reviewable patch proposal policy over the shared graph. - `epiphany-core::retrieval` - remains text/chunk retrieval and low-level Qdrant/Ollama backend adapter @@ -316,16 +350,16 @@ staleness, open questions, and required verification hooks. ### Rebuild -- Build `epiphany-core::repo_model` as the new architecture organ before adding - more retrieval features. -- Build typed repo-model CultNet contracts so Aquarium and coding lanes can ask - for context packets without Codex JSON-RPC. +- Build `epiphany-core::memory_graph` as the shared architecture/memory organ + before adding more retrieval features. +- Build typed memory-graph CultNet contracts so Aquarium and coding lanes can + ask for repo-profile context packets without Codex JSON-RPC. ## First Slice Do not start by embedding everything. First prove the typed shape. -1. Add typed repo-model document structs in `epiphany-state-model`: +1. Add shared typed memory graph document structs in `epiphany-state-model`: - snapshot - domain - node @@ -334,8 +368,9 @@ Do not start by embedding everything. First prove the typed shape. - embedding manifest - context packet - freshness -2. Add `epiphany-core::repo_model` with pure in-memory validation and context - cut planning over mocked documents. + - repo profile kind/lifecycle hooks +2. Add `epiphany-core::memory_graph` with repo-profile policy hooks plus pure + in-memory validation and context cut planning over mocked documents. 3. Add unit tests: - node ids are stable from domain/path/symbol - edges cannot reference missing nodes @@ -389,7 +424,7 @@ The plan above is architecturally concrete, but not yet implementation-concrete. This roadmap is the cut order. Do not jump to Qdrant first. A fast lie is still a lie; it just gets to the wrong answer with better posture. -### Phase 0: Name The Contract +### Phase 0: Name The Shared Contract Objective: make the target type surface explicit before any scanner, model, or vector store can smuggle authority into the system. @@ -402,6 +437,16 @@ Files: Add typed state-model structs: +- `EpiphanyMemoryGraphSnapshot` +- `EpiphanyMemoryDomain` +- `EpiphanyMemoryNode` +- `EpiphanyMemoryEdge` +- `EpiphanyMemorySummary` +- `EpiphanyMemoryEmbeddingManifest` +- `EpiphanyMemoryFreshness` +- `EpiphanyMemoryContextQuery` +- `EpiphanyMemoryContextPacket` +- `EpiphanyMemoryPatchCandidate` - `EpiphanyRepoModelSnapshot` - `EpiphanyRepoDomain` - `EpiphanyRepoNode` @@ -436,31 +481,34 @@ Tests: Exit criteria: -- The repo-model document vocabulary compiles without Qdrant. +- The shared memory graph document vocabulary compiles without Qdrant. +- Repo-profile aliases/projections can be represented without a second graph + engine. - The contracts can be advertised without any scanner implementation. -### Phase 1: Pure Repo Model Core +### Phase 1: Pure Memory Graph Core With Repo Profile Objective: build the in-memory law engine that can validate and select graph context without touching disk, Qdrant, Ollama, Codex, or app-server routing. Files: -- `epiphany-core/src/repo_model.rs` -- `epiphany-core/src/repo_model/ids.rs` -- `epiphany-core/src/repo_model/validation.rs` -- `epiphany-core/src/repo_model/freshness.rs` -- `epiphany-core/src/repo_model/context_cut.rs` +- `epiphany-core/src/memory_graph.rs` +- `epiphany-core/src/memory_graph/ids.rs` +- `epiphany-core/src/memory_graph/validation.rs` +- `epiphany-core/src/memory_graph/freshness.rs` +- `epiphany-core/src/memory_graph/context_cut.rs` +- `epiphany-core/src/memory_graph/profiles/repo.rs` - `epiphany-core/src/lib.rs` Functions: -- `repo_model_domain_id(kind, path_or_name) -> String` -- `repo_model_node_id(domain_id, kind, path, symbol) -> String` -- `repo_model_edge_id(source_id, target_id, kind, code_refs) -> String` -- `validate_repo_model_snapshot(snapshot) -> Vec` -- `derive_repo_model_freshness(snapshot, source_hashes, dirty_paths) -> EpiphanyRepoFreshness` -- `plan_repo_context_cut(snapshot, query, budget) -> EpiphanyRepoContextPacket` +- `memory_graph_domain_id(profile, kind, path_or_name) -> String` +- `memory_graph_node_id(domain_id, kind, path, symbol) -> String` +- `memory_graph_edge_id(source_id, target_id, kind, code_refs) -> String` +- `validate_memory_graph_snapshot(snapshot) -> Vec` +- `derive_memory_graph_freshness(snapshot, source_hashes, dirty_paths) -> EpiphanyMemoryFreshness` +- `plan_memory_graph_context_cut(snapshot, query, budget) -> EpiphanyMemoryContextPacket` Tests: @@ -477,9 +525,10 @@ Tests: Exit criteria: -- `cargo test --manifest-path .\epiphany-core\Cargo.toml --lib repo_model` +- `cargo test --manifest-path .\epiphany-core\Cargo.toml --lib memory_graph` passes. -- Repo context packets can be produced from fixture documents without Qdrant. +- Repo-profile context packets can be produced from fixture documents without + Qdrant. ### Phase 2: Deterministic Scanner @@ -816,15 +865,18 @@ objective ## First Implementable Ticket -Start here if the user says "build it." +Do not start here directly anymore. Start with +`notes/epiphany-memory-graph-unified-plan.md`. -Title: `Add typed RepoModel documents and pure context-cut validation` +Title: `Add shared EpiphanyMemoryGraph typed documents and pure validation` Scope: -- Add repo-model document structs to `epiphany-state-model/src/lib.rs`. -- Add `epiphany-core/src/repo_model.rs` plus `ids`, `validation`, - `freshness`, and `context_cut` submodules. +- Add shared memory graph document structs to `epiphany-state-model/src/lib.rs`. +- Add `epiphany-core/src/memory_graph.rs` plus document, id, validation, + freshness, and context-cut submodules. +- Include `repo_architecture` and `repo_dataflow` profile enums/policy hooks, + but do not implement the scanner yet. - Export the pure APIs from `epiphany-core/src/lib.rs`. - Add fixture-only unit tests for stable ids, missing-edge validation, stale hash propagation, and parent-summary context cuts. @@ -842,12 +894,12 @@ Verification: ```powershell cargo fmt --manifest-path .\epiphany-core\Cargo.toml -cargo test --manifest-path .\epiphany-core\Cargo.toml --lib repo_model +cargo test --manifest-path .\epiphany-core\Cargo.toml --lib memory_graph cargo check --manifest-path .\epiphany-core\Cargo.toml ``` Definition of done: -- The repo-model organ has typed bones and pure tests. +- The shared memory-graph organ has typed bones and pure tests. - The next ticket can add deterministic scanning without debating what a node, edge, summary, freshness record, or context packet is supposed to be. diff --git a/state/ledgers.msgpack b/state/ledgers.msgpack index 39b28f76843935f7139c39dc0f143057148b8f0c..7b4f00bf9855ee925371f175d1583af1a32e5b91 100644 GIT binary patch delta 394 zcmZvYu}T9$5QaIMyn^L`SO|B?C8)$$idv;mtSrR6y-9BIZg-iTi(VB$8#_U;un~L% zK@cGiB?y9@<7sKN3&Z^1_s`>d=i#GsyR!6k`n-A{MJHE_=yefA)ou)w{{#6YQ`K0y@LrU&8mbcDPMrHIjAXGh6*c34fs;o0LHQd zXAyS2@rAO@xSj5|JE6`3lVXH(!$q4K1Gy4o2!$g~7^#XSL)V#_vvD?V8M=u<3((}U zn+O7FCgk6_c;L+BXf=W-H!&rKZ8*qDi5p?1jA$wOWwj}Dm6a%2xN-=A(&oyN!*+q{Qy`yNI{-)~|KTZ{~h%ybP*bqy^-3=FIcjIB(~ Fq5$&Y5eonS diff --git a/state/map.yaml b/state/map.yaml index 6933fcaf..2f07b03c 100644 --- a/state/map.yaml +++ b/state/map.yaml @@ -75,7 +75,7 @@ active_subgoals: - The May 2026 teardown/rebuild migration in `notes/epiphany-ideal-architecture-rebuild-plan.md` is closed unless a concrete regression reopens it. Preserve the new foundation: Epiphany view/coordinator policy lives in `epiphany-core`, runtime-spine owns job/result lifecycle identity through `runtime_links`, duplicate acceptance is receipt-keyed, public findings stay typed/sealed, view expansion requires a core contract, and app-server remains routing/adaptation rather than a second policy throne. - Treat Phase 5 update/distill/propose/promote/read/notification as complete enough unless a concrete regression appears. - Preserve read-only boundaries for retrieve, distill, propose, freshness, context, graphQuery, roleResult, reorientResult, and the canonical view lenses for scene, jobs, roles, planning, pressure, reorient, crrc, and coordinator. - - Use `notes/repo-fractal-dataflow-cache-plan.md` as the source-grounded repo-model direction when replacing ad hoc grep/source orientation: typed CultCache repo graph documents own project shape; Qdrant stores semantic embeddings for nodes, edges, summaries, and context cuts as a rebuildable contribution cache, not as canonical truth. The roadmap's first implementable ticket is typed RepoModel documents plus pure `epiphany-core::repo_model` validation/context-cut tests, explicitly before scanner or Qdrant work. + - Use `notes/epiphany-memory-graph-unified-plan.md` as the source-grounded memory architecture direction. The repo fractal dataflow graph and agent memory are one `EpiphanyMemoryGraph` substrate with repo, role, short-term, incubation, agency, candidate-intervention, and evidence profiles. Qdrant stores rebuildable semantic embeddings for typed graph documents; it is never canonical memory. The first implementable ticket is shared memory graph documents plus pure validation/context-cut tests before repo scanners, sleep runners, CultCache store wiring, Qdrant cache plumbing, CultNet advertisement, or prompt integration. - Keep the landed Phase 6 coordinator, limited safe-boundary CRRC automation, and pre-compaction checkpoint intervention ready for the next bridge-equipped dogfood run. - Preserve the Ghostlight-derived heartbeat initiative seam: every lane, including coordinator/Self and Face, has a role dossier and competes for heartbeat slots through readiness/reaction timing; idle winners ruminate instead of inventing project work, and sleep windows distill the accumulated thought pressure into reviewed self-memory or doctrine. - Treat Ghostlight as reference lineage, not a sibling runtime to preserve. The useful organ is Epiphany-native character-loop protocol: any role can project its typed dossier plus visible stimulus into a local turn packet, with Face as the first public-surface actor. @@ -114,7 +114,7 @@ accepted_design: - `thread/epiphany/retrieve`, `thread/epiphany/distill`, `thread/epiphany/propose`, `thread/epiphany/freshness`, `thread/epiphany/context`, `thread/epiphany/graphQuery`, and `thread/epiphany/view` stay read-only. - `thread/epiphany/index` writes the retrieval catalog, not durable Epiphany understanding; catalog is auspex output, not doctrine. - Qdrant is the preferred persistent semantic backend; BM25 remains the bootstrap/fallback/control path. - - `notes/repo-fractal-dataflow-cache-plan.md` applies the mapped-fractal-domain/cache lesson to Epiphany and now includes an implementation roadmap: build a `RepoModel` organ where typed domain/node/edge/summary/context documents own repo architecture and dataflow, while Qdrant indexes embeddings for fast selection. If deleting Qdrant deletes understanding, the design is wrong; the fallback must be slower but still correct typed graph traversal. Start with typed documents and pure validation/context-cut tests, not vector plumbing. + - `notes/epiphany-memory-graph-unified-plan.md` applies the mapped-fractal-domain/cache lesson to Epiphany and corrects the organ boundary: repo architecture/dataflow and agent memory are profiles over one `EpiphanyMemoryGraph`, not separate graph engines. `notes/repo-fractal-dataflow-cache-plan.md` and `notes/agent-memory-fractal-cache-plan.md` are profile plans beneath that substrate. If deleting Qdrant deletes understanding, the design is wrong; the fallback must be slower but still correct typed graph traversal. Start with shared typed documents and pure validation/context-cut tests, not vector plumbing or profile-specific producers. - `epiphany-core` owns prompt rendering, rollout replay helpers, retrieval/indexing, distillation, proposal, promotion policy, scene projection, freshness derivation, pressure derivation, job/progress view derivation, planning view derivation, reorientation resume/regather verdict policy, pure CRRC recommendation policy, fixed-lane coordinator decision policy, role-board projection policy, and role-result interpretation/self-persistence review policy where possible. - Vendored Codex keeps the thin typed host seam: protocol types, thread bridges, app-server protocol, request routing, watcher lifecycle, notifications, and rollout integration. Epiphany projection and coordinator policy should keep moving into `epiphany-core` / `epiphany-codex-bridge` instead of living in `codex_message_processor.rs`. - The current algorithmic map is the source-grounded description of live control flow. @@ -291,10 +291,12 @@ rejected_paths: current_status: summary: The Codex starvation pass has five typing receipts, fifty-six processor/foundation cuts, a real physical extraction in `epiphany-codex-bridge`, a root `epiphany-state-model` crate for durable Epiphany state, a source-grounded Codex auth spine inventory, the native `epiphany-openai-adapter` document boundary, a buildable Codex auth/model wrapper in `epiphany-openai-codex-spine`, the first native runtime caller in `epiphany-openai-runtime`, native heartbeat launch planning/job opening in `epiphany-core`, native state-update contract/validation/application in `epiphany-core`, native exact retrieval path matching in `epiphany-core`, and the route-facing mutation/view/coordinator projection services in `epiphany-codex-bridge`. `codex-protocol` no longer defines the durable `Epiphany*` state structs; it re-exports the native model for legacy rollout/API compatibility while `epiphany-core` imports the native model directly. `epiphany-core` now has no direct vendored Codex dependency: pressure derivation accepts native `EpiphanyTokenUsageSnapshot`, Codex rollout replay moved to `codex-core::epiphany_rollout`, and exact retrieval search uses Epiphany-owned workspace walking/path scoring instead of `codex-file-search`. The bridge owns Codex-facing Epiphany adapters for scene, jobs, context/planning/graph query, retrieval/indexing projection, result projection, runtime-spine result adaptation, launch/prompt doctrine, mutation document mechanics, mutation service update/promote/role-accept/reorient-accept/role-launch/reorient-launch/job-launch/job-interrupt application, live-state projection, pressure/pre-compaction rendering, freshness/reorientation projection, CRRC/coordinator/role-board projection, watcher invalidation adaptation, full `thread/epiphany/view` response construction, freshness/context/graph-query response construction, role/reorient result response construction, distill/propose patch-response construction, coordinator status/view construction, and acceptance/evidence signal helpers. Vendored app-server no longer retains root Epiphany modules; `codex_message_processor.rs` still contains route dispatch, JSON-RPC response shaping, watcher registration/snapshotting calls, and notification emission, but its root no longer imports the evacuated Epiphany policy/protocol registry or owns the watcher invalidation manager. Codex product surfaces for apps, skills, plugins, marketplace, and external agent config import are cut from the live app-server/protocol path: list routes return empty lists only where compatibility shells still exist, mutation/detail routes report disabled where they survive, `externalAgentConfig/*` routes and import-completed notification are gone, helper/import modules and stale implementation-shape tests were deleted, runtime feature enablement no longer supports `apps` or `plugins`, config writes no longer emit plugin-toggle analytics, plugin/skill cache invalidation on config writes is gone, plugin startup warmups and app-list refreshes were removed, and app-server no longer directly depends on `codex-plugin`, `codex-chatgpt`, or `codex-core-plugins`. `Config::to_mcp_config` now keeps user-declared MCP servers but no longer accepts `PluginsManager`, folds plugin MCP servers into the runtime, enables Codex apps, enables skill MCP dependency install, or advertises plugin capability summaries; `TurnContext::apps_enabled()` now returns false so model turns cannot awaken Codex Apps through ChatGPT auth or feature flags; session startup and per-turn context no longer load skills, turns receive an empty `SkillLoadOutcome`, internal `Op::ListSkills` is gone from `codex-protocol` / `codex-core` / TUI command routing, the Codex skill watcher has been collapsed to a no-op marker with no file watcher, skill-root registration, cache-clear task, or `SkillsUpdateAvailable` session listener, model turns no longer run skill mention collection, skill env-var prompts, skill MCP dependency installation, skill instruction injection, or implicit skill invocation analytics from shell tools, and the turn prompt/router path no longer loads plugin/app capability summaries, injects available-skill/available-plugin/plugin instruction fragments, parses plugin/app mentions, records app/plugin invocation telemetry, or filters MCP tools through Codex app connectors. Skill mutation/list/notification is gone from the app-server edge too: `skills/list`, `skills/config/write`, `skills/changed`, `Op::ListSkills`, `EventMsg::ListSkillsResponse`, `EventMsg::SkillsUpdateAvailable`, the app-server handlers, TUI slash/startup/list/manage-skill surfaces, generated schema entries, core skill-list suite, and fake live-reload/developer-message tests were deleted. Skill mention plumbing is also gone from the live TUI/protocol edge: no `UserInput::Skill` variant, app-server schema branch, TUI skill popup, `skills_all` cache, composer `set_skill_mentions`, skill mention insertion, `skill://` / `SKILL.md` history decoding, or stale skill popup snapshot remains. The `codex-core-skills` crate, core `skills` re-export shim, no-op `skills_watcher`, session `SkillsManager` service, and per-turn `TurnSkillsContext` are deleted; core/protocol no longer keep legacy `` or `` prompt compatibility, memory usage classification, snapshot normalization, or cleanup paths. Remaining product-shaped skill residue is also cut from memory templates, app-server docs/tests/warnings, TUI tooltips/placeholders/warnings/comments, app-server v1 docs, agent-tool schema text, stale `UserInput::Skill` test arms, and obsolete feature flags/schema entries. App-instruction prompt compatibility is cut too: no `APPS_INSTRUCTIONS` protocol constants, no `` snapshot canonicalization, and MCP fixture tests now use neutral `demo_mcp` / `calendar_mcp` names instead of `codex_apps` / `/api/codex/apps` branding. The TUI app connector product path is also deleted: no `/apps` slash command, app-link view, connector AppEvent variants, app connector cache/popup/refresh loop, app mention insertion, `app://` mention codec support, stale app-popup tests, or TUI browser-opening shim remains. The `app/list` route and `app/list/updated` notification are gone from app-server protocol, generated schemas, app-server dispatch, test helpers, and docs; `AppInfo`/app-list metadata schema structs are deleted; the orphaned `codex-connectors` crate and `core::connectors` shim are deleted from the workspace; Codex app analytics no longer tracks `codex_app_mentioned` / `codex_app_used` facts, events, reducer arms, client APIs, or dedupe cache state; `[apps.*]`, `Feature::Apps`, `include_apps_instructions`, app config/read schemas, app managed requirements, app prompt tests, and app file-MCP tests are gone; `Feature::Plugins`, `Feature::RemotePlugin`, `plugin://` structured mentions, plugin instruction tags, and stale plugin popup snapshots are gone too; `[plugins]` / `[marketplaces]`, `PluginConfig`, `MarketplaceConfig`, marketplace edit helpers, stale plugin/external-agent README route claims, and generated schema entries are removed from config; and the bundled `codex-skills` sample-skill crate plus embedded imagegen/openai-docs/plugin-creator/skill-creator/skill-installer payloads are deleted from the workspace. The vendored `vendor/codex/sdk` subtree is deleted as non-spine app-server client/release scaffolding, including stale generated Python SDK route types for removed product surfaces; the root pnpm workspace no longer lists `sdk/typescript`. Live MCP tool approval is named `McpToolApproval`, not `AppToolApproval`, and remains under MCP server config; live structured user-input mentions are MCP-shaped only. `codex_message_processor.rs` is about 1,009 lines after route/lifecycle/projection/resume-helper/config-helper extraction; root `message_processor.rs` is about 1,292 lines after the external-agent/config-product cuts; `epiphany_read_routes.rs` is about 579 lines after the full view and read-response extractions; `epiphany_mutation_routes.rs` is about 694 lines after route-load/notification helper cuts and product-surface starvation; the old vendored `epiphany_state_helpers.rs` rollout-state loader is deleted and its remaining function lives in `epiphany-codex-bridge::state`; and `codex_thread.rs` is about 1,425 lines after state-update law moved native. `epiphany-openai-adapter` defines CultCache `DatabaseEntry` request/event/receipt/auth-status documents with no Codex app-server or generic JSON dependency. `epiphany-openai-codex-spine` owns the native OpenAI model transport path: it exposes `default_codex_home` and `auth_manager`, projects `epiphany-openai-auth-spine` `AuthManager` / `CodexAuth` into typed `EpiphanyOpenAiAdapterStatus`, builds a local serializable OpenAI Responses request body instead of Codex protocol request/item/config types, opens HTTP Responses streams through `codex-client`, parses Responses SSE frames into typed stream events and receipts, and no longer directly or transitively depends on Codex's Responses client, model-provider stack, `codex-protocol`, `codex-api`, `codex-model-provider`, or `codex-model-provider-info`; it intentionally reaches retained vendored `codex-login` only through `epiphany-openai-auth-spine`. Auth identity, credential storage, and ChatGPT token refresh remain owned by the Codex-compatible auth organ; external bearer command auth and agent identity remain retained Codex auth capabilities to expose only if a typed Epiphany surface needs them. `epiphany-openai-runtime` no longer depends directly on `codex-login`; it writes OpenAI status/request/stream-event/receipt records plus runtime sessions/jobs/results/events into `epiphany-core::runtime_spine_cache`, can emit the assistant text artifact for legacy result consumers, is now the default executor for startup-only repo birth specialists, and now has `run-worker` to load a durable `EpiphanyRuntimeWorkerLaunchRequest`, call the typed OpenAI runtime through the spine, persist typed `EpiphanyRuntimeRoleWorkerResult` / `EpiphanyRuntimeReorientWorkerResult` documents, and complete the original heartbeat/specialist runtime job without Codex worker execution; `codex exec` survives only as explicit fallback in repo-birth startup. `epiphany-mvp-coordinator` now invokes `epiphany-openai-runtime run-worker` for launched runtime job ids and no longer creates a second coordinator-owned shadow runtime job/result as a compensator. `epiphany-core::plan_runtime_spine_heartbeat_launch` validates launch contracts/conflicts and projects durable job bindings/runtime links; `epiphany-core::open_runtime_spine_heartbeat_job` owns the initialize-runtime/ensure-session/create-job sequence and now persists `EpiphanyRuntimeWorkerLaunchRequest` CultCache records keyed by runtime job id, with indexed kind/contract fields plus MessagePack-encoded typed launch documents instead of discarding work intent after queue receipt creation; `roleResult` and `reorientResult` now require typed worker result documents; generic job summaries are lifecycle receipts, not reviewable findings; `epiphany-core::EpiphanyStateUpdate`, `epiphany_state_update_validation_errors`, and `apply_epiphany_state_update` own state-update document shape and mutation law while `codex-core` calls them for compatibility. Vendored `CodexThread::epiphany_launch_job` is now mostly revision check, persistence validation, launch request handoff, and rollout/session writeback for the compatibility route. Vendored `codex-core` no longer has `epiphany_distillation`, `epiphany_promotion`, `epiphany_proposal`, or `epiphany_retrieval` husk modules; it re-exports native surfaces from `epiphany-core` directly, and now owns only the real Codex rollout adapter because it is a host seam. `epiphany-runtime-spine` advertises OpenAI adapter status/model-request/stream-event/receipt CultNet schemas and mutation contracts, so the native caller now has a real storage route instead of only JSON-RPC. MCP itself is correctly JSON at its protocol edge; the target is typed Epiphany intent/result/receipt wrappers around MCP, not replacing MCP JSON. The current OpenAI request path no longer pulls `codex-api`; `codex-login` remains deliberately sealed behind `epiphany-openai-auth-spine`, and the next spine decision is another Epiphany-in-vendor evacuation surface unless a concrete typed auth-status/runtime need requires exposing external bearer or agent-identity auth. latest_cut: > - `notes/repo-fractal-dataflow-cache-plan.md` now has an implementation - roadmap for the repo-model feature. The first ticket is typed RepoModel - documents in `epiphany-state-model` plus pure `epiphany-core::repo_model` - id/validation/freshness/context-cut tests, with scanner, CultCache store, - Qdrant cache, CultNet advertisement, and coding-lane integration staged - afterward. - next_action: Continue the Codex starvation campaign unless the user directs the next slice into repo modeling. If repo modeling starts, first add typed repo-model document structs and pure `epiphany-core::repo_model` validation/context-cut tests before connecting Qdrant. + `notes/epiphany-memory-graph-unified-plan.md` now corrects the repo + fractal graph and agent memory roadmaps into one shared + `EpiphanyMemoryGraph` substrate with domain profiles for repo + architecture/dataflow, role self-memory, short-term thought, incubation, + agency pressure, candidate interventions, and evidence. The first ticket + is shared typed memory graph documents plus pure validation/context-cut + tests, with scanners, sleep consolidation, CultCache store wiring, Qdrant + cache, CultNet advertisement, and prompt integration staged afterward. + next_action: Continue the Codex starvation campaign unless the user directs the next slice into memory graph work. If memory graph work starts, first add shared `EpiphanyMemoryGraph` typed document structs and pure `epiphany-core::memory_graph` id/validation/freshness/context-cut tests before adding repo profile producers, agent sleep consolidation, or Qdrant.