Epiphany/notes/epiphany-fork-implementation-plan.md

Epiphany Fork Implementation Plan

Status

Updated on 2026-04-25 after recognizing that Phase 5 is complete enough to stop feeding itself tiny safety slices. The update/distill/propose/promote/read/notification control plane is now the regression-guarded baseline, not an excuse to keep rearranging the furniture until everyone dies of responsible boredom.

This note tracks Epiphany as a fork of Codex with an opinionated modeling architecture. The point is not to offer Codex another collaboration preset. The point is to make the harness force the model to carry explicit structure about the codebase, the active subgoal, the evidence trail, and the machine it is modifying.

• Phase 1 durable Epiphany thread state
• Phase 2 prompt integration
• a minimal Phase 3 typed client read surface via Thread.epiphanyState
• a verified Phase 4 slice 1 query-time hybrid retriever across protocol, core, app-server protocol, and app-server
• a verified Phase 4 slice 2 follow-up that adds explicit persistent semantic indexing through thread/epiphany/index while keeping thread/epiphany/retrieve read-only
• the heavy Epiphany-owned prompt/replay/retrieval implementation has now been extracted into the repo-owned epiphany-core crate, leaving vendored Codex with thin host adapters plus the typed integration seam
• a first typed state-update slice that adds explicit loaded-thread-only thread/epiphany/update, appends observations/evidence, replaces bounded map/scratch/churn fields, bumps the state revision, and persists an immediate rollout EpiphanyState snapshot
• a first typed distillation/proposal slice that adds read-only loaded-thread-only thread/epiphany/distill, producing deterministic observation/evidence patches for explicit promotion through thread/epiphany/update
• a first richer promotion-policy slice that treats map/churn/frontier/checkpoint replacements as higher-risk state edits: accepted promotions must be tied to explicit observations and patch evidence, and structural graph/churn mistakes are rejected before the durable update path
• a first typed map/churn proposal slice that adds read-only loaded-thread-only thread/epiphany/propose, producing candidate graph frontier/churn patches from verified observations with code refs for explicit promotion through thread/epiphany/promote
• a first proposal-quality hardening slice that reuses existing architecture graph nodes by exact code-ref, same-path overlap, or deterministic path-node id before creating new candidate nodes, and records whether a proposal refines, expands, or updates the map in churn
• a frontier-focus hardening slice that expands proposal frontier focus through existing graph links and marks named incident graph edges active, so prompt rendering gets the relevant architecture/dataflow context instead of one lonely node with a tiny hat
• an evidence-selection/churn hardening slice that requires selected proposal observations to cite accepting recent_evidence and derives proposal diff_pressure from the candidate map delta, touched paths, selected observation count, and unresolved write risk
• a graph-semantic proposal hardening slice that scores selected observations/evidence, uses the strongest selected signal for proposal wording, and can reuse unanchored architecture graph nodes through strict unique semantic overlap while refusing ambiguous matches and leaving code-anchored nodes under concrete ref/path/id matching
• an automatic observation-selection slice that lets read-only thread/epiphany/propose omit observationIds; epiphany-core then chooses a bounded coherent path cluster from existing verified, evidence-backed observations using source/evidence quality and current graph-frontier focus
• a map-delta judgment slice that distinguishes exact-ref refinement, same-path broadening, deterministic-node reuse, semantic unanchored-node reuse, and new candidate surfaces before setting graph freshness and diff_pressure
• a source-output distillation slice that keeps thread/epiphany/distill read-only while summarizing noisy tool/command/shell/model output into salient lines and defaulting evidence kind to tool-output or model-output
• a source-output salience-ranking slice that ranks final test result, failure, error, exit-code, passed, and finished lines above generic warnings, while still preserving warnings when warnings are the only useful signal
• a risky-delta promotion policy slice that keeps thread/epiphany/promote as the only gate but requires medium/high/expanded/broadening/semantic/update churn deltas to carry explicit warning rationale plus strong verifier evidence kind
• an expansion-freshness promotion hardening slice that prevents a graph-expansion patch from bypassing warning requirements by falsely claiming low diff_pressure
• a token-aware verifier-kind promotion hardening slice that prevents substring accidents such as contest from satisfying the risky-delta test verifier-kind requirement
• a reusable richer Phase 5 app-server smoke harness at tools/epiphany_phase5_smoke.py that verifies source-output distillation, read-only proposal, verifier-only promotion notification evidence changes, risky-delta rejection, expanded-low-pressure rejection, substring verifier-kind rejection, and accepted graph/churn promotion through the loaded-thread seam
• a first live typed-state notification slice: experimental thread/epiphany/stateUpdated fires after successful direct thread/epiphany/update and accepted thread/epiphany/promote, carrying the updated epiphanyState, typed source, event-level revision, and typed changedFields while rejected promotions remain silent and non-mutating; accepted promotions always report evidence when verifier evidence is appended
• a direct write-response hardening slice: successful thread/epiphany/update and accepted thread/epiphany/promote responses now include response-level revision and changedFields, matching the notification metadata without forcing the initiating caller to wait for or diff the live event
• a rejected-promotion silence smoke hardening slice: the reusable Phase 5 app-server smoke now asserts rejected thread/epiphany/promote calls do not emit thread/epiphany/stateUpdated, preserving the "no mutation, no bell" contract
• a direct-update integrity hardening slice: thread/epiphany/update now validates appended observation/evidence graph records and structural replacement fields before mutation, rejects empty/duplicate/reused append ids, missing evidence references, and invalid graph/frontier/checkpoint/subgoal/churn replacement shapes, and the reusable smoke asserts invalid direct updates leave revision unchanged with no thread/epiphany/stateUpdated notification
• a write/read surface consistency hardening slice: successful direct update and accepted promote responses plus their thread/epiphany/stateUpdated notifications now reuse the same live Epiphany state projection as thread/read, so clients get retrieval-summary backfill consistently instead of receiving a raw just-written state in one surface and a richer read state in another
• current phase: Phase 6 reflection boundary / scene-surface precursor

The next job is no longer to prove the retriever exists, sketch the persistent follow-up in prose, invent the first red-pen path, build the first observation proposal surface, make promotion notice broken map/churn replacements, ship the first read-only map/churn proposal surface, teach proposal to avoid duplicating already-mapped code surfaces, make proposal frontier focus follow existing graph links, make selected observations evidence-backed with non-haunted churn pressure, rescue unanchored graph nodes with strict semantic matching, let proposal choose a bounded ready observation set when ids are omitted, make proposal pressure understand match kinds, make the distiller summarize noisy source output, make source-output salience prefer final results over generic warnings, make promotion notice risky deltas, make expansion freshness resistant to low-pressure underreporting, prevent substring verifier-kind accidents, prove that richer chain over app-server by hand, add the first live state notification after successful writes, make that notification say which revision and state sections changed, reject malformed direct updates, or make write responses match thread/read projection. Those parts are landed or live-smoked.

The next job is to move one layer outward: design and land a meaningful Phase 6 reflection/scene-surface or job-state precursor that lets a client inspect and steer the typed Epiphany machine without making the GUI/client the source of truth. Use tools/epiphany_phase5_smoke.py as a regression guardrail when touching the Phase 5 control plane, not as a ritual that justifies another tiny safety slice.

Workflow trap to avoid: do not leave handoff or plan notes that say only "continue the next smallest hardening slice" after a phase has reached a coherent baseline. Bounded slices are how work lands safely; they are not the project direction. A good handoff should name the next organ, not merely prescribe one more scrape of sandpaper on the same seam.

Summary

Phase 4 lands as a bounded internal retrieval/indexing slice:

• add typed retrieval state and shard/index summaries to Epiphany state
• add one additive app-server retrieval query surface for loaded threads
• add one explicit semantic indexing surface for loaded threads
• support hybrid retrieval from day one:
  • exact/path/symbol/lexical results
  • semantic chunk results
• keep the slice internal/dev-usable first while the fork architecture hardens

After this slice:

• Epiphany can ask structured repo questions through one typed surface
• GUI work has a real data source later
• retrieval can evolve without making prompt text or shell transcripts the canonical knowledge path

This slice should still avoid:

• watcher-driven invalidation
• automatic observation promotion
• GUI implementation
• specialist-agent scheduling
• user-facing activation flows

Phase 4 Principle

Do not build "vector search" as a separate novelty organ off to the side.

Build a hybrid repo retrieval subsystem with one typed query shape and one typed result shape. Exact lookup and semantic lookup should be different modes of the same machine, not rival religions.

Recovered bounded decision

After rehydrating from the repo state, the first bounded Phase 4 slice was:

• query-time
• loaded-thread-first
• read-only
• hybrid from day one

Concrete choice for the first implementation:

• reuse existing codex_file_search substrate for exact/path-ish lookup
• use workspace-local BM25 chunk search as the initial semantic backend
• defer embeddings, vector stores, persistent indexing, and watcher-driven invalidation

Updated follow-up direction after landing the verified slice:

• keep the BM25 implementation as the proven baseline/fallback
• for the first persistent semantic backend, prefer Qdrant over JSON/blob/postgres-thrash designs, but do not make it a hard requirement for basic Epiphany use
• do not turn retrieval into vector-only soup; exact/path lookup remains first-class and BM25 remains useful as bootstrap/control/fallback when Qdrant is unavailable
• live app-server smoke verified the explicit Qdrant path against real local services using an ephemeral loaded thread rooted at epiphany-core; the smoke indexed 5 files into 198 chunks, created Qdrant collection epiphany_workspace_fa24bab116f8d229, and retrieved persistent semantic chunks from src/prompt.rs
• env/default backend configuration is good enough for the current slice; defer a first-class backend config surface until a real operator pain proves it deserves to exist

Preflight note:

• use tracked source as the corpus estimate, not raw working-tree disk usage
• at vendor/codex HEAD d45ab10, tracked codex-rs source is 3642 files / 31.09 MB
• raw working-tree size under codex-rs can be heavily inflated by build artifacts and will lie to the design

Current landed implementation note:

• the protocol scaffold still exists in vendor/codex/codex-rs/protocol/src/protocol.rs
• the heavier Epiphany-owned implementation now lives in:
  • E:/Projects/EpiphanyAgent/epiphany-core/src/prompt.rs
  • E:/Projects/EpiphanyAgent/epiphany-core/src/rollout.rs
  • E:/Projects/EpiphanyAgent/epiphany-core/src/retrieval.rs
• the vendored host seam is now wired through:
  • core/src/epiphany_retrieval.rs
  • core/src/epiphany_rollout.rs
  • core/src/context/epiphany_state_instructions.rs
  • core/src/codex_thread.rs
  • core/src/session/tests.rs
  • app-server-protocol/src/protocol/common.rs
  • app-server-protocol/src/protocol/v2.rs
  • app-server-protocol/src/export.rs
  • app-server/src/codex_message_processor.rs
• the retrieval/indexing/core-extraction baseline is now formatted, verified, committed, and pushed on main at 80c29e0:
  • cargo fmt --all passed
  • targeted no-run compile for codex-core, codex-app-server-protocol, and codex-app-server passed
  • targeted Phase 4 tests passed in core, app-server protocol, and app-server
  • full cargo test -p codex-app-server-protocol passed after regenerating stable schema fixtures
• stable schema fixtures are the safe checked-in state for this slice; write_schema_fixtures --experimental rewrites the same tree and cannot be left behind as the repo state

Key Changes

1. Extend Epiphany state with retrieval metadata

Touch:

• vendor/codex/codex-rs/protocol/src/protocol.rs

Add minimal retrieval metadata to EpiphanyThreadState, for example:

• retrieval
  • workspace_root
  • index_revision
  • status
  • semantic_available
  • last_indexed_at
  • shards
  • dirty_paths

Keep this metadata summary-focused. Do not dump raw embeddings, giant posting lists, or per-file sludge into thread state.

2. Add a core retrieval facade

Touch:

• E:/Projects/EpiphanyAgent/epiphany-core/src/retrieval.rs
• vendor/codex/codex-rs/core/src/epiphany_retrieval.rs

Responsibilities:

• the repo-owned crate should accept a loaded-thread-derived workspace root plus query params
• it should run exact retrieval and semantic retrieval behind one interface
• it should return typed ranked results with file paths, line anchors, excerpts, and retrieval mode metadata
• the vendored Codex file should stay a thin adapter or re-export layer, not the place where the heavy organ grows back out of spite

Result types should be explicit and bounded:

• exact hits
• semantic hits
• optional graph-linked ids later

Do not make the first cut depend on prompt parsing or transcript scraping.

3. Keep the first semantic backend small and local

The first semantic backend should be workspace-local BM25 chunk retrieval built at query time over the tracked source corpus.

Requirements:

• keep chunk text line-anchored and path-aware
• keep the implementation bounded enough to understand in one sitting
• track retrieval/index freshness metadata explicitly in typed state
• do not introduce a monolithic persistent blob just to feel sophisticated

If the first semantic backend needs to be intentionally narrow, that is fine. A small honest subsystem beats a magical blob with terrible write behavior.

4. Add one additive app-server query surface

Touch:

• vendor/codex/codex-rs/app-server-protocol/src/protocol/v2.rs
• vendor/codex/codex-rs/app-server/src/codex_message_processor.rs
• vendor/codex/codex-rs/app-server/README.md

Add one typed method for retrieval, tentatively:

• thread/epiphany/retrieve

Suggested request fields:

• threadId
• query
• optional cwd
• optional modes or preferSemantic
• optional limit
• optional pathPrefixes

Suggested response fields:

• query
• indexSummary
• results

Keep it additive and read-only. That part is now landed.

Follow-up addition that is now verified and pushed:

• thread/epiphany/index

Current bounded shape of that follow-up:

• it is the explicit write path for persistent semantic indexing
• it persists manifest metadata under codex_home
• it targets Qdrant as the preferred persistent semantic backend
• it uses local Ollama embeddings by default
• it leaves exact/path lookup first-class
• it keeps BM25 alive as the fallback/control path when the persistent backend is stale, missing, or unavailable
• it now runs through the sibling epiphany-core crate so modified Codex alone is not quietly the whole product

Follow-up addition now landed in the working tree:

• thread/epiphany/update

Current bounded shape of that follow-up:

• it is the explicit write path for durable Epiphany state, not a retrieval side effect
• it requires a loaded thread
• it optionally checks expectedRevision
• it can append observations and evidence
• it can replace bounded typed fields: objective, active subgoal, subgoals, invariants, graphs, graph frontier/checkpoint, scratch, churn, and mode
• it increments the Epiphany revision, updates lastUpdatedTurnId when a reference turn exists, writes live SessionState, persists RolloutItem::EpiphanyState, and flushes the rollout
• rollout replay now accepts an out-of-band Epiphany snapshot before the first real user turn so pre-turn seed updates survive resume

Follow-up addition now landed after the update/promote surfaces:

• thread/epiphany/stateUpdated

Current bounded shape of that follow-up:

• it is an experimental server notification, not another write path
• it carries threadId plus the full updated epiphanyState
• it carries source: "update" after successful direct thread/epiphany/update
• it carries source: "promote" after accepted thread/epiphany/promote
• it carries event-level revision so clients can sequence updates without cracking open the full state first
• it carries typed changedFields derived from the accepted patch surface so clients can react to top-level state-section changes without diffing the entire epiphanyState
• successful update/promote responses and notifications publish the same client-visible live state projection as thread/read; if the durable state has no persisted retrieval metadata yet, the non-durable retrieval summary backfill is present in those response/notification states too
• accepted promotion notifications include evidence even when the patch evidence list was empty, because verifier evidence is appended to durable state after policy acceptance
• it does not fire for rejected promotions, because rejection still does not mutate state
• successful thread/epiphany/update responses now also carry response-level revision and changedFields
• accepted thread/epiphany/promote responses now also carry response-level revision and changedFields; rejected promotion responses omit revision/change metadata and still carry reasons plus no state
• the richer app-server smoke now asserts direct update responses/notifications publish revision/changed-fields 1/observations,evidence, verifier-only accepted promotion responses/notifications publish 2/observations,evidence, accepted graph/churn promotion responses/notifications publish 3/graphs,graphFrontier,observations,evidence,churn, and all successful write response/notification states include the same retrieval-summary backfill exposed by thread/read
• the richer app-server smoke also asserts rejected promotion attempts leave rejectedPromotionNotificationCount at 0
• the richer app-server smoke now also asserts malformed direct thread/epiphany/update observation/evidence appends and structural replacement patches return invalid-request errors, leave revision at 2, and leave invalidDirectUpdateNotificationCount / invalidReplacementUpdateNotificationCount at 0

Follow-up addition now landed after the update surface:

• thread/epiphany/distill

Current bounded shape of that follow-up:

• it is a read-only proposal path, not a second state writer
• it requires a loaded thread so the response can include the current expectedRevision
• it normalizes one explicit source/status/text observation plus optional subject, evidence kind, and code refs
• it returns deterministic observation/evidence records inside a ThreadEpiphanyUpdatePatch
• callers must pass the patch to thread/epiphany/update to make it durable

Follow-up addition now landed after the promotion gate:

• richer thread/epiphany/promote policy for state replacement patches

Current bounded shape of that follow-up:

• it keeps the wire protocol stable
• it still rejects failed verifier evidence without mutation
• it now passes replacement fields into epiphany-core policy evaluation instead of reducing them to a generic boolean
• map/churn/frontier/checkpoint replacements must include explicit observations and patch evidence, so state edits are tied to a verified observation trail
• subgoals, invariants, graphs, graph links, frontier ids, checkpoint ids, and churn fields get lightweight structural validation before accepted patches are sent to thread/epiphany/update
• it still does not infer graph edits automatically; callers must submit explicit typed replacements

5. Reuse existing exact-search substrate where practical

Codex already has fuzzy/exact-ish repo search substrate. Reuse what is useful instead of building a second exact-search toy from scratch.

But do not confuse that with the whole answer. Phase 4 exists because exact file search alone is not enough.

6. Test the machine, not just the types

Add tests for:

1. protocol serde for the new retrieval request/response shapes
2. core ranking/merging behavior for mixed exact + semantic results
3. app-server handling for the new request
4. retrieval metadata inclusion in EpiphanyThreadState
5. a smoke test that proves semantic retrieval can find a concept that exact-name search would miss

Assumptions

• Phase 4 is still internal/dev-usable only.
• Retrieval should be thread/workspace aware, not a global cross-repo soup.
• The first retrieval slice can be read-only.
• Watcher-driven invalidation belongs later; this slice only needs enough metadata to admit freshness honestly.

Immediate Next Step

Treat the current retrieval baseline and explicit indexing follow-up as landed Phase 4. Treat the explicit update path, distillation proposal path, read-only map/churn proposal path, verifier-backed promotion gate, structural map/churn promotion validation, direct-update append/replacement integrity validation, graph-node reuse, linked frontier focus, evidence-backed selection, map-delta churn pressure, selected-observation prioritization, strict unanchored-node semantic reuse, automatic bounded observation-set selection, match-kind-aware map-delta judgment, source-output-aware observation distillation, source-output salience ranking, risky-delta promotion policy, expansion-freshness promotion hardening, token-aware verifier-kind promotion hardening, reusable richer app-server smoke harness, and sourced/revisioned/changed-field thread/epiphany/stateUpdated notification seam with write/read retrieval-summary projection consistency as the active baseline. The next machine gap is no longer "can Epiphany retrieve code?", "can it propose one durable observation patch?", "can it draft one bounded map/churn candidate?", "can it avoid duplicating an existing graph node when an observation points at already-mapped code?", "can it focus linked graph context?", "can it reject selected observations that are not backed by accepting recent evidence?", "can it use graph language when no concrete refs exist yet?", "can it choose a proposal-ready observation set when ids are omitted?", "can it distinguish exact refinement from same-path broadening or semantic anchoring?", "can it summarize noisy tool/model output as typed evidence?", "can it rank final result/error/finished lines above generic warnings?", "can promotion policy reason over richer deltas without silently writing them?", "can direct update reject malformed append and replacement patches before writing?", "can expansion freshness resist low-pressure underreporting?", "can verifier kind matching avoid substring accidents?", "does the richer Phase 5 chain behave through the app-server seam?", "can successful writes notify a client what changed state and why?", or "do successful write responses/notifications expose the same retrieval-summary-backed state projection as thread/read?" It can.

1. treat the current verified-and-landed query-time hybrid retriever as the Phase 4 slice 1 baseline
2. treat the verified and live-smoked thread/epiphany/index slice as the bounded persistent-semantic follow-up
3. keep the new epiphany-core boundary honest instead of letting vendored Codex re-accumulate the heavy implementation
4. do not add durable retrieval-summary writes from thread/epiphany/retrieve without a clean out-of-band rollout/update semantic
5. declare Phase 5 complete enough unless a concrete bug, regression, or real-use gap appears
6. use the Phase 5 smoke as a guardrail before modifying the existing control plane, not as a reason to keep sanding that same control plane forever
7. choose the next meaningful Phase 6 precursor:
   • a typed scene/read model for graph, invariants, evidence, intent, churn, and retrieval status, derived from existing Epiphany state
   • or a minimal typed job-state/progress surface for indexing/remap/verification work that a future GUI can render
8. keep the source-of-truth rule intact: GUI/client surfaces reflect and steer typed state, but they do not manufacture canonical understanding

Do not start with automatic graph invalidation or specialist-agent scheduling. Do not pretend shell transcripts are a retrieval strategy. GUI reflection is no longer forbidden as a category, but it must start as a thin reflection over the engine, not as a pretty parallel brain.