Map repo fractal context cache

This commit is contained in:
Meta Vens 2026-05-18 00:58:26 +01:00
commit 4872154af5
4 changed files with 393 additions and 5 deletions

View file

@ -39,6 +39,7 @@ remembers doctrine; the branch remembers the blade.
- `epiphany-openai-runtime` worker result ingress now parses assistant output directly into typed role/reorient ingress structs selected from the durable launch document, stores nested `statePatch` and `selfPatch` as typed MessagePack in `EpiphanyRuntimeRoleWorkerResult`, and no longer passes worker result data through generic `serde_json::Value`. JSON survives there only as the OpenAI text-output edge before typed deserialization.
- Agent self-memory review/application now has typed `AgentSelfPatch` core APIs. JSON-value selfPatch functions are compatibility wrappers for CLI/artifact ingress; native callers should parse once at the edge and then call `review_agent_self_patch_document` / `apply_agent_self_patch_document`. Role-result self-persistence no longer exports a JSON-value reviewer; the live API is `review_role_self_patch_document`.
- `epiphany-openai-codex-spine` no longer uses Codex's Responses client, model-provider layer, `codex-api`, `codex-model-provider`, `codex-model-provider-info`, or `codex-protocol` for outbound model requests. It reaches `codex-login` only through `epiphany-openai-auth-spine` so credential loading/refresh stays Codex-compatible while request/status/event/runtime surfaces stay typed Epiphany documents.
- `notes/repo-fractal-dataflow-cache-plan.md` is the current repo-model direction drawn from `Fractal Domains And The Cache That Bites`: the repo's semantic architecture/dataflow graph should be a typed CultCache document organ, with Qdrant storing rebuildable embeddings for nodes, edges, conservative summaries, and context cuts. Qdrant is a contribution cache, not truth. The coding lane should eventually receive typed repo context packets for orientation and use source reads mainly for exact patch verification.
- Durable Epiphany state types now live in root crate `epiphany-state-model`. `codex-protocol` re-exports them only so old rollout/API callers keep compiling; do not add new durable `Epiphany*` state structs under `vendor/codex`. `epiphany-core` imports the native model directly and no longer depends on `codex-protocol`. Pressure derives from native `EpiphanyTokenUsageSnapshot`; Codex token telemetry is mapped at the bridge. Codex rollout/event reconstruction now lives in `codex-core::epiphany_rollout`, where the Codex `RolloutItem` / `EventMsg` types belong. Exact retrieval path matching is native to `epiphany-core`, so `epiphany-core` has no direct vendored Codex crate dependency.
- State-update contract proof and job launch/interrupt DTO ownership now live in native `epiphany-core`. The old giant `codex_thread.rs` Epiphany state-update test fixture was cut, and `epiphany-codex-bridge` no longer imports Epiphany job launch request types from `codex_core`; vendored `CodexThread` may call the native mutation/launch law for compatibility, but it must not regain ownership of state-update semantics, heartbeat launch-plan policy, or Epiphany DTOs.
- MCP remains JSON because MCP is a JSON protocol. The latest processor cut only moved MCP refresh/OAuth/status/resource/tool-call route handling into `codex_message_processor/mcp_routes.rs`, reducing the root processor to about 8,977 lines; it did not claim MCP payloads are CultNet or make MCP an Epiphany-owned runtime contract.

View file

@ -0,0 +1,384 @@
# Repo Fractal Dataflow Cache Plan
This note applies the lesson from
`E:\Projects\gamecult-site\GameCult\Blog\fractal-domains-cache-that-bites.md`
to Epiphany's repo-modeling architecture.
The lesson is not "add embeddings." That is the cheap altar. The lesson is:
```text
semantic intent
-> owned domain
-> graph grammar
-> conservative summaries
-> contribution cache
-> selected context packet
-> coding agent
```
The cache only deserves to exist after the domain owns the object. For
Epiphany, the object is not a file chunk. The object is the repo's live
architecture/dataflow graph.
## Current Mechanism
Epiphany currently has two separate organs:
- `epiphany-core/src/retrieval.rs` indexes workspace text chunks, writes a
manifest, stores semantic chunk vectors in Qdrant, and falls back to
query-time BM25 when the persistent index is missing or stale.
- `epiphany-core/src/surfaces/graph_context.rs` reads accepted typed graph
state from `EpiphanyThreadState` and derives bounded node/edge/path/frontier
context. It does not search source, infer missing architecture, or update
embeddings.
That split is conceptually sound but incomplete. Retrieval is fast text memory.
GraphQuery is accepted map memory. Neither is the resident repo model that can
answer, "what exact architectural neighborhood does this task touch?"
So the coding agent still has to grep, read docs, and reconstruct project shape
when the accepted graph is sparse or stale. That is the xenos whale carcass in a
better coat.
## Desired Mechanism
Introduce an Epiphany-owned `RepoModel` organ:
```text
repo files / docs / manifests / build metadata / git history
-> typed repo scanner
-> typed code/document domains
-> fractal architecture/dataflow graph documents
-> conservative node and edge summaries
-> Qdrant embeddings for nodes, edges, summaries, and candidate neighborhoods
-> query planner chooses a stable context cut
-> coding agent receives exact typed context packet
```
The durable truth is CultCache document state. Qdrant is the contribution cache.
It makes selection fast. It does not become the source of truth.
## Core Inputs
- Source files, docs, manifests, build config, generated schema files, and
tests from the workspace.
- Git history and current dirty state.
- Existing accepted Epiphany architecture/dataflow graph nodes, edges, links,
frontier, checkpoints, observations, and evidence.
- Optional IDE/editor bridge facts from Rider/Unity when they expose structure
the filesystem cannot see cleanly.
- User objective and active lane intent.
## Durable State Stores
Keep truth in typed CultCache documents:
- `epiphany.repo_model.snapshot`: repo identity, scan revision, source hashes,
root domains, and model status.
- `epiphany.repo_model.node`: typed architecture/dataflow/code/domain nodes.
- `epiphany.repo_model.edge`: typed ownership, call, dataflow, config,
dependency, runtime, test, and evidence edges.
- `epiphany.repo_model.summary`: conservative summaries for subtree/domain
cuts, with freshness and confidence.
- `epiphany.repo_model.frontier`: active task neighborhood, dirty refs, open
questions, and selected context cuts.
- `epiphany.repo_model.embedding_manifest`: Qdrant collection names, model id,
vector dimensions, indexed document ids, source hashes, and stale regions.
Qdrant stores vectors and retrieval payloads for fast selection:
- node vectors
- edge vectors
- summary vectors
- code/document excerpt vectors
- task-context packet vectors
- optional historical probe/click/acceptance signals
Qdrant payloads may carry keys, hashes, score metadata, and compact labels. They
must not be the only home of graph truth.
## Core Transformations
1. **Domain Mapping**
- Map the repo into explicit domains: crate, module, binary, test suite,
schema catalog, prompt template, runtime spine, bridge adapter, vendored
Codex organ, docs, and state store.
- Each domain owns the facts that give its children meaning.
2. **Graph Grammar**
- Emit typed nodes and edges from parsers and bounded model-assisted
distillers.
- Nodes represent stable concepts: module authority, state document,
runtime contract, adapter, route, test seam, schema, prompt, or external
bridge.
- Edges represent real relationships: owns, reads, writes, derives,
adapts, persists, launches, verifies, renders, imports, exports, depends
on, invalidates, and tests.
3. **Summary Construction**
- Every node/subtree/domain gets a conservative summary: purpose, owned
invariant, inputs, outputs, child count, confidence, dirty hash set,
public API refs, and blast radius.
- If a subtree cannot summarize its children with bounded honesty, it is not
eligible as a collapsed context packet.
4. **Embedding Projection**
- Embed the durable node, edge, summary, and selected excerpt documents.
- Store vectors in Qdrant keyed by typed document id and source hash.
- Refresh only dirty regions when source hashes change.
5. **Context Cut Planning**
- Given task intent, active frontier, dirty refs, and query text, select a
stable cut through the repo graph.
- Prefer high-confidence summaries for broad context and descend into exact
nodes/files/tests only where projected task relevance or uncertainty earns
it.
6. **Context Packet Rendering**
- Emit a typed `EpiphanyRepoContextPacket` containing graph nodes, edges,
summaries, code refs, required exact reads, known stale regions, and test
seams.
- The coding agent consumes that packet before touching source.
## Core Outputs
- `EpiphanyRepoContextPacket`: the exact architecture neighborhood for the
current work.
- `EpiphanyRepoFreshness`: stale regions, dirty graph paths, embedding drift,
and required reindex/regather actions.
- `EpiphanyRepoModelStatus`: indexed domains, graph confidence, Qdrant
readiness, source hash coverage, and failed extraction receipts.
- `EpiphanyRepoModelPatchCandidate`: reviewable graph updates, never automatic
architecture truth.
## Module Ownership
Target module network:
- `epiphany-state-model`
- owns typed document structs for repo model snapshots, nodes, edges,
summaries, embedding manifests, context packets, freshness, and receipts.
- `epiphany-core::repo_model`
- owns domain mapping, graph grammar, summary construction, freshness,
context-cut planning, validation, and patch proposal policy.
- `epiphany-core::retrieval`
- remains text/chunk retrieval and low-level Qdrant/Ollama backend adapter
until split.
- It should not own repo architecture semantics.
- `epiphany-core::surfaces::graph_context`
- becomes a view over accepted graph/model state, not a substitute for the
repo modeler.
- `epiphany-runtime-spine`
- advertises CultNet read/mutation contracts for repo-model status, context
query, indexing, and reviewable patch candidates.
- `epiphany-codex-bridge`
- compatibility wrapper only. It may adapt old `thread/epiphany/retrieve` and
graphQuery calls to native contracts while consumers migrate.
- Qdrant/Ollama
- vector cache and embedding service. Fast shrine. No throne.
## Invariants
- CultCache documents own repo shape.
- Qdrant is cache/index, not canonical truth.
- Every node has an owner domain, stable id, source hash/freshness identity, and
a reason to exist.
- Every edge names a real relationship and points to existing typed node ids.
- Every summary must be conservative enough to stand in for its children when
the context budget does not permit descent.
- Model-assisted graph extraction produces patch candidates, not accepted truth.
- Context packets may omit detail only when they carry a bounded summary and a
confidence/freshness signal.
- The coding agent may still perform exact reads for patch work, but ordinary
orientation should come from the repo model packet, not ad hoc grep.
- Tests must align to module ownership: parsers, graph grammar, summaries,
freshness, Qdrant adapter, and context planner each get their own mock seam.
## Architectural Smells To Avoid
### Qdrant As Truth
Current temptation: put rich payloads into Qdrant and call that the graph.
Real need: fast semantic selection.
What breaks if deleted: latency and semantic recall quality, not truth.
Verdict: if deleting Qdrant loses the architecture map, ownership is wrong.
Simpler architecture: typed repo model in CultCache; Qdrant stores vectors keyed
to typed document ids.
### Chunk Retrieval Pretending To Be Understanding
Current mechanism: `retrieval.rs` indexes 24-line chunks and returns excerpts.
Real need: source lookup and fallback context.
What breaks if deleted: search convenience.
Verdict: not enough for architecture. Chunks are leaves, not domains.
Simpler architecture: repo domains and graph nodes summarize source; chunk
retrieval supports exact evidence and descent.
### Accepted Graph Too Sparse To Orient Work
Current mechanism: `EpiphanyThreadState.graphs` stores accepted nodes/edges and
`graphQuery` walks them.
Real need: durable reviewed project understanding.
What breaks if deleted: the accepted map and review-gated graph memory.
Verdict: essential, but not sufficient. It needs a repo-model substrate that can
propose and refresh neighborhoods.
Simpler architecture: accepted graph remains doctrine; repo modeler maintains
source-grounded candidate graph/summary documents with review gates.
### Model Extraction Without Conservative Summaries
Current temptation: ask a model to summarize files/modules and store the result.
Real need: human-scale graph construction.
What breaks if deleted: automation speed.
Verdict: model extraction is useful only when bounded by source hashes,
evidence refs, confidence, and review.
Simpler architecture: parser/scanner emits deterministic structure first; model
lanes propose semantic summaries; summaries must cite refs and pass validation.
### Context Packets As Pretty Retrieval Results
Current temptation: rank chunks and paste them into the prompt.
Real need: give the coding agent enough shape to act without wandering.
What breaks if deleted: prompt convenience.
Verdict: a packet that lacks ownership, invariants, freshness, and test seams is
not architecture context.
Simpler architecture: packets are typed graph cuts with summaries, exact refs,
staleness, open questions, and required verification hooks.
## Ranked Teardown Plan
### Keep
- Existing typed `EpiphanyThreadState.graphs`, observations, evidence,
frontier, and checkpoints as reviewed durable project truth.
- Existing Qdrant/Ollama workspace retrieval as a backend capability.
- Existing `graphQuery` and context surfaces as read-only bounded graph views.
- Existing rule that `thread/epiphany/index` writes retrieval catalog, not
durable understanding.
### Cut
- Any future design where semantic retrieval results directly mutate accepted
graph state.
- Any Qdrant payload that becomes the only place a node, edge, or summary
exists.
- Prompt-time grep/source-dump orientation as the normal path once repo-model
packets exist.
### Collapse
- Retrieval freshness, graph freshness, and embedding manifest freshness should
collapse into one repo-model freshness surface once the repo model owns source
hash coverage.
- Separate "retrieve query" and "graph query" operator workflows should
collapse into "ask repo model for a context packet" for ordinary agent work,
while exact retrieve remains a leaf-level evidence tool.
### Split
- Split `epiphany-core/src/retrieval.rs` into:
- text corpus/chunking
- embedding backend
- Qdrant backend
- retrieval ranking
- manifest/freshness
- repo-model embedding cache adapter
- Split graph truth from graph candidate generation:
- accepted graph stays in durable Epiphany state
- repo modeler owns source-derived candidate graph documents and patch
proposals
### Rebuild
- Build `epiphany-core::repo_model` as the new architecture organ before adding
more retrieval features.
- Build typed repo-model CultNet contracts so Aquarium and coding lanes can ask
for context packets without Codex JSON-RPC.
## First Slice
Do not start by embedding everything. First prove the typed shape.
1. Add typed repo-model document structs in `epiphany-state-model`:
- snapshot
- domain
- node
- edge
- summary
- embedding manifest
- context packet
- freshness
2. Add `epiphany-core::repo_model` with pure in-memory validation and context
cut planning over mocked documents.
3. Add unit tests:
- node ids are stable from domain/path/symbol
- edges cannot reference missing nodes
- dirty source hashes mark affected summaries stale
- context planner returns parent summary when child detail is unnecessary
- context planner descends when uncertainty or task relevance crosses the
threshold
4. Only then connect Qdrant as a cache keyed to typed document ids.
## Mock Boundaries
Interview-grade seams:
- fake repo source scanner
- fake git dirty snapshot
- fake symbol extractor
- fake summary extractor
- fake embedding provider
- fake vector index
- fake clock
- fake evidence store
- fake context budget
Tests should shame the ownership boundary:
- repo model can build context packets without Qdrant
- Qdrant can disappear and leave slower but correct graph traversal
- stale source hashes force re-summarization before packet confidence is high
- model-assisted extraction cannot accept graph truth without evidence/review
- a packet names the exact tests or verification seams relevant to its graph cut
## Success Criteria
The target is not "faster search." The target is:
- The coding agent starts with a typed context packet for the task.
- That packet names ownership, dataflow, invariants, exact refs, stale regions,
and test seams.
- The agent uses grep/source reads mainly for exact patch verification, not
basic orientation.
- Qdrant reduces latency but can be rebuilt from typed documents and source
hashes.
- Accepted architecture truth remains review-gated.
The renderer from the article does not wake every branch of the fractal tree.
Epiphany should not wake the whole repo every time it wants to move one bolt.

Binary file not shown.

File diff suppressed because one or more lines are too long