GameCult/Epiphany
Archived
Watch
9
0
Fork
You've already forked Epiphany
 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

Land Epiphany Phase 4 hybrid retrieval slice

Browse source
This commit is contained in:
Meta Vens 2026-04-24 02:45:02 +01:00
commit 360dfea6e4
39 changed files with 10378 additions and 62 deletions
Download patch file Download diff file Expand all files Collapse all files

73
notes/codex-epiphany-mode-plan.md
View file

@ -2,13 +2,14 @@
## Status
Planned on 2026-04-23 after landing and verifying:
Updated on 2026-04-24 after verifying the first bounded Phase 4 slice in the working tree:
- Phase 1 durable Epiphany thread state
- Phase 2 prompt integration
- a minimal Phase 3 typed client read surface via `Thread.epiphanyState`
- a verified Phase 4 slice 1 query-time hybrid retriever across protocol, core, app-server protocol, and app-server
The next job is to give Epiphany a real repo retrieval organ instead of making every future role rediscover the workspace with shell commands and optimism.
The next job is no longer to prove the retriever exists. That part is done. The next job is to keep the landing zone honest: do not casually turn `thread/epiphany/retrieve` into a durable Epiphany-state writer just because the state object has a `retrieval` field, and do not regress into another blob-shaped persistent semantic store when the time comes to add one.
## Summary
@ -41,6 +42,51 @@ Do not build "vector search" as a separate novelty organ off to the side.
Build a **hybrid repo retrieval subsystem** with one typed query shape and one typed result shape. Exact lookup and semantic lookup should be different modes of the same machine, not rival religions.
## Recovered bounded decision
After rehydrating from the repo state and the lone dirty working-tree diff, the first bounded Phase 4 slice should be:
- query-time
- loaded-thread-first
- read-only
- hybrid from day one
Concrete choice for the first implementation:
- reuse existing `codex_file_search` substrate for exact/path-ish lookup
- use workspace-local BM25 chunk search as the initial semantic backend
- defer embeddings, vector stores, persistent indexing, and watcher-driven invalidation
Updated follow-up direction after landing the verified slice:
- keep the BM25 implementation as the proven baseline/fallback
- for the first persistent semantic backend, prefer Qdrant over JSON/blob/postgres-thrash designs, but do not make it a hard requirement for basic Epiphany use
- do not turn retrieval into vector-only soup; exact/path lookup remains first-class and BM25 remains useful as bootstrap/control/fallback when Qdrant is unavailable
Preflight note:
- use tracked source as the corpus estimate, not raw working-tree disk usage
- at `vendor/codex` HEAD `d45ab10`, tracked `codex-rs` source is `3642` files / `31.09 MB`
- raw working-tree size under `codex-rs` can be heavily inflated by build artifacts and will lie to the design
Current in-flight implementation note:
- the protocol scaffold still exists in `vendor/codex/codex-rs/protocol/src/protocol.rs`
- it is now wired through the working tree across:
- `core/src/epiphany_retrieval.rs`
- `core/src/codex_thread.rs`
- `core/src/session/tests.rs`
- `app-server-protocol/src/protocol/common.rs`
- `app-server-protocol/src/protocol/v2.rs`
- `app-server-protocol/src/export.rs`
- `app-server/src/codex_message_processor.rs`
- the slice is now formatted and verified in the working tree:
- `cargo fmt --all` passed
- targeted no-run compile for `codex-core`, `codex-app-server-protocol`, and `codex-app-server` passed
- targeted Phase 4 tests passed in core, app-server protocol, and app-server
- full `cargo test -p codex-app-server-protocol` passed after regenerating stable schema fixtures
- stable schema fixtures are the safe checked-in state for this slice; `write_schema_fixtures --experimental` rewrites the same tree and cannot be left behind as the repo state
## Key Changes
### 1. Extend Epiphany state with retrieval metadata
@ -89,14 +135,14 @@ Do not make the first cut depend on prompt parsing or transcript scraping.
### 3. Keep the first semantic backend small and local
The first semantic backend should be workspace-local, sharded, and incremental-friendly in storage layout, even if the initial indexing policy is still crude.
The first semantic backend should be workspace-local BM25 chunk retrieval built at query time over the tracked source corpus.
Requirements:
- do not store everything in one giant JSON blob
- prefer per-workspace or per-subsystem shards
- track index revision and freshness explicitly
- keep chunk text line-anchored and path-aware
- keep the implementation bounded enough to understand in one sitting
- track retrieval/index freshness metadata explicitly in typed state
- do not introduce a monolithic persistent blob just to feel sophisticated
If the first semantic backend needs to be intentionally narrow, that is fine. A small honest subsystem beats a magical blob with terrible write behavior.
@ -154,12 +200,15 @@ Add tests for:
## Immediate Next Step After This Plan
Implement the smallest useful retrieval baseline:
Treat the current retrieval baseline as real and ship it cleanly:
1. extend `EpiphanyThreadState` with retrieval metadata
2. add a core hybrid retrieval facade
3. expose one typed `thread/epiphany/retrieve` read method
4. document it in `app-server/README.md`
5. verify with targeted protocol/core/app-server tests
1. clean up, stage, and push the current verified query-time hybrid retriever as the Phase 4 slice 1 landing zone
2. do not add durable retrieval-summary writes from `thread/epiphany/retrieve` without a clean out-of-band rollout/update semantic
3. if a later follow-up is needed, keep it tight:
- retrieval freshness/persistence semantics
- probably a dedicated Epiphany update path
- Qdrant-backed persistent semantic indexing as the preferred backend, with BM25 fallback for no-Qdrant or degraded mode
- not GUI work
- not watcher-driven invalidation
Do not start with GUI. Do not start with automatic graph invalidation. Do not pretend shell transcripts are a retrieval strategy.

92
notes/epiphany-current-algorithmic-map.md
View file

@ -2,17 +2,17 @@
This note is not a full machine map for Codex. We already have that in [E:/Projects/EpiphanyAgent/notes/codex-repository-algorithmic-map.md](E:/Projects/EpiphanyAgent/notes/codex-repository-algorithmic-map.md).
This one is narrower and meaner: it documents where the current Epiphany prototype already diverges from upstream Codex in the code we have actually landed.
This one is narrower and meaner: it documents where the current Epiphany prototype already diverges from upstream Codex in the code we actually have in the working tree.
Scope: current state after Phase 1, Phase 2, and the minimal Phase 3 typed read surface. It does **not** describe planned retrieval, invalidation, GUI reflection, or multi-agent scheduling as if they already exist.
Scope: current state after Phase 1, Phase 2, the minimal Phase 3 typed read surface, and the first bounded Phase 4 repo-local hybrid retrieval slice. It does **not** describe planned invalidation, GUI reflection, or multi-agent scheduling as if they already exist.
## Mental model in one sentence
Current Epiphany is a thin but real overlay on Codex's thread harness: Codex now carries a structured thread-state snapshot through persistence, replay, prompt assembly, and hydrated client thread payloads.
Current Epiphany is a thin but real overlay on Codex's thread harness: Codex now carries structured thread state through persistence, replay, prompt assembly, hydrated client thread payloads, and one typed repo-local hybrid retrieval surface.
## Divergence summary
Today Epiphany differs from plain Codex in four shipped places:
Today Epiphany differs from plain Codex in five currently active places:
1. **Durable state exists**
- Codex rollout/history now has a first-class `EpiphanyThreadState` snapshot.
@ -26,6 +26,9 @@ Today Epiphany differs from plain Codex in four shipped places:
4. **Replay semantics respect rollback and compaction**
- the "latest Epiphany state" is not "last one in the file." It is reconstructed using turn/rollback boundaries.
5. **Repo-local hybrid retrieval now exists behind one typed surface**
- loaded threads can answer structured repo queries through exact/path lookup plus BM25 chunk retrieval.
That is already a meaningful divergence from stock Codex, even if the more glamorous organs are still missing.
## The delta flow
@ -41,6 +44,9 @@ flowchart TD
B --> H["app-server stored-thread reads reconstruct latest surviving Epiphany state from rollout"]
G --> I["Client can read typed state"]
H --> I
D --> J["CodexThread exposes epiphany_retrieve() and epiphany_retrieval_state()"]
J --> K["app-server routes thread/epiphany/retrieve"]
K --> L["Hybrid exact/path + BM25 chunk results"]
```
The important thing is that the same state is now flowing through three layers:
@ -49,7 +55,7 @@ The important thing is that the same state is now flowing through three layers:
- prompt construction
- typed client reads
That is the current Epiphany spine.
That is the current Epiphany spine, and retrieval now hangs off it instead of living as shell archaeology.
## Divergence 1: durable typed Epiphany state
@ -207,19 +213,75 @@ We do **not** yet have:
- `thread/epiphany/read`
- `thread/epiphany/update`
- `thread/epiphany/retrieve`
- `thread/epiphany/stateUpdated`
- `thread/epiphany/evidenceAppended`
So the client can now see the state, but cannot yet steer or subscribe to it as a first-class live surface.
## Divergence 5: repo-local hybrid retrieval
### What changed
Loaded threads now expose a typed retrieval surface that combines exact/path-ish lookup with semantic chunk retrieval:
- [E:/Projects/EpiphanyAgent/vendor/codex/codex-rs/protocol/src/protocol.rs](E:/Projects/EpiphanyAgent/vendor/codex/codex-rs/protocol/src/protocol.rs)
- [E:/Projects/EpiphanyAgent/vendor/codex/codex-rs/core/src/epiphany_retrieval.rs](E:/Projects/EpiphanyAgent/vendor/codex/codex-rs/core/src/epiphany_retrieval.rs)
- [E:/Projects/EpiphanyAgent/vendor/codex/codex-rs/core/src/codex_thread.rs](E:/Projects/EpiphanyAgent/vendor/codex/codex-rs/core/src/codex_thread.rs)
- [E:/Projects/EpiphanyAgent/vendor/codex/codex-rs/app-server-protocol/src/protocol/common.rs](E:/Projects/EpiphanyAgent/vendor/codex/codex-rs/app-server-protocol/src/protocol/common.rs)
- [E:/Projects/EpiphanyAgent/vendor/codex/codex-rs/app-server-protocol/src/protocol/v2.rs](E:/Projects/EpiphanyAgent/vendor/codex/codex-rs/app-server-protocol/src/protocol/v2.rs)
- [E:/Projects/EpiphanyAgent/vendor/codex/codex-rs/app-server/src/codex_message_processor.rs](E:/Projects/EpiphanyAgent/vendor/codex/codex-rs/app-server/src/codex_message_processor.rs)
Current behavior:
- `EpiphanyThreadState` now has retrieval metadata via `retrieval`
- `CodexThread` exposes:
- `epiphany_retrieval_state()`
- `epiphany_retrieve(...)`
- app-server protocol exposes experimental `thread/epiphany/retrieve`
- the retrieval machine is hybrid from day one:
- exact/path-ish hits via existing `codex_file_search`
- semantic hits via workspace-local BM25 chunk search over a gitignore-respecting corpus
- live `thread.epiphanyState` hydration backfills a lightweight retrieval summary when the thread has Epiphany state but no persisted retrieval metadata yet
### How this differs from plain Codex
Plain Codex has pieces of repo search substrate scattered around:
- fuzzy/exact-ish file search
- BM25 usage in tool-search paths
- shell access if the operator wants to rummage manually
What it does not have is one Epiphany-owned retrieval organ that says:
- this query is for repo understanding
- this thread/workspace is the scope
- exact and semantic lookup are the same machine
- results come back in a typed shape instead of as transcript confetti
### Why it matters
This is the first time Epiphany can ask a structured repo question without falling back to file-by-file terminal spelunking.
That changes the shape of the harness. The map is no longer just a notebook the prompt can read. It now has a first retrieval organ hanging off the same typed spine.
### Important limits
This slice is intentionally small and honest:
- query-time only
- loaded-thread-first
- read-only
- local BM25 rather than embeddings
- no watcher-driven invalidation
- no persistent vector store
- no explicit retrieval update/write path yet
## What has *not* diverged yet
This matters because the spec is ahead of the code in a few obvious places.
### Not yet shipped
- no retrieval subsystem
- no code-intelligence layer
- no watcher-driven semantic invalidation
- no observation-promotion machinery beyond the data model
@ -250,6 +312,7 @@ input
-> rollout + epiphany_state snapshot
-> replay-aware epiphany restoration
-> hydrated client thread payload with epiphanyState
-> optional typed repo retrieval through thread/epiphany/retrieve
```
That is the real delta.
@ -258,7 +321,7 @@ The transcript is no longer the only thing with continuity.
## Where the divergence is already meaningful
Even without retrieval or GUI, the divergence is already structurally important in three ways:
Even without invalidation or GUI, the divergence is already structurally important in four ways:
1. **Understanding has a durable object now**
- not just prose and replayed vibes
@ -269,12 +332,19 @@ Even without retrieval or GUI, the divergence is already structurally important
3. **Clients can read that understanding directly**
- not by scraping the prompt or reverse-engineering transcript fragments
4. **The harness has one typed repo retrieval organ now**
- not just shell archaeology plus vibes
That is not the full Epiphany vision yet, but it is no longer cosmetic either.
## Recommended next companion note
Once Phase 4 lands, this file should get a new section:
If the next bounded slice lands, the useful follow-on section for this file is probably not "more retrieval exists."
- `Divergence 5: repo-local hybrid retrieval`
It is more likely one of these:
That will probably be the moment the divergence starts feeling less like a careful extension and more like a different operating philosophy living inside the same harness.
- retrieval freshness/persistence semantics
- structured observation promotion from retrieval/tool output into durable Epiphany state
- Qdrant-backed persistent semantic indexing as a preferred backend, with BM25 fallback instead of a hard dependency
Those would be the next places where the machine stops being a better notebook and starts being a more opinionated harness.

130
notes/fresh-workspace-handoff.md
View file

@ -45,6 +45,91 @@ Current architectural/spec notes:
- `notes/epiphany-current-algorithmic-map.md`
- `notes/epiphany-core-harness-surfaces.md`
## Current In-Flight Work
Phase 4 repo-local hybrid retrieval is now verified in the working tree. It is still uncommitted, but it is no longer a vague scaffold or a maybe.
Current working-tree implementation spans:
- `vendor/codex/codex-rs/protocol/src/protocol.rs`
- `vendor/codex/codex-rs/core/src/epiphany_retrieval.rs`
- `vendor/codex/codex-rs/core/src/codex_thread.rs`
- `vendor/codex/codex-rs/core/src/lib.rs`
- `vendor/codex/codex-rs/core/Cargo.toml`
- `vendor/codex/codex-rs/core/src/session/tests.rs`
- `vendor/codex/codex-rs/app-server-protocol/src/protocol/common.rs`
- `vendor/codex/codex-rs/app-server-protocol/src/protocol/v2.rs`
- `vendor/codex/codex-rs/app-server-protocol/src/export.rs`
- `vendor/codex/codex-rs/app-server/src/codex_message_processor.rs`
- `vendor/codex/codex-rs/app-server/README.md`
- regenerated stable fixtures under `vendor/codex/codex-rs/app-server-protocol/schema/`
What is wired right now:
- `EpiphanyThreadState` already has retrieval metadata fields and types in protocol
- `codex-core` now has a new `epiphany_retrieval.rs` module
- the core retriever is query-time and hybrid:
- exact/path-ish lookup via `codex_file_search`
- semantic lookup via BM25 chunk search over a local gitignore-respecting corpus
- `CodexThread` now exposes:
- `epiphany_retrieval_state()`
- `epiphany_retrieve(...)`
- app-server protocol now declares experimental `thread/epiphany/retrieve`
- app-server message handling now routes that method to the core retriever
- live hydrated `thread.epiphanyState` reads now backfill a lightweight retrieval summary when the thread already has Epiphany state but no persisted retrieval summary yet
- basic protocol/core/app-server unit tests were added for the new types and mapping layer
Important recovery facts:
- existing exact/path lookup substrate already exists through `codex_file_search`
- app-server fuzzy search currently uses it in `vendor/codex/codex-rs/app-server/src/fuzzy_file_search.rs`
- `codex-core` already depends on `bm25`
- `vendor/codex/codex-rs/core/src/tools/handlers/tool_search.rs` already shows how BM25 is used in-tree
- tracked corpus preflight for `vendor/codex/codex-rs` at `vendor/codex` HEAD `d45ab10` is `3642` files / `31.09 MB`
- raw working-tree size under `codex-rs` is polluted by build artifacts and should not be used as the retrieval corpus estimate
Recovered bounded design decision:
- the first Phase 4 slice should be a query-time hybrid retriever
- reuse `codex_file_search` for exact/path-ish lookup
- use workspace-local BM25 chunk search as the initial semantic backend
- keep the first slice loaded-thread-only and read-only
- defer embeddings, vector stores, persistent indexing, and watcher-driven invalidation for later slices
Updated follow-up direction:
- the verified BM25 slice is still the right first landing zone
- the first persistent semantic store after that should target Qdrant rather than another blob-thrashing JSON/Postgres embedding store
- Qdrant should be the preferred persistent semantic backend later, not a hard requirement for basic Epiphany use
- exact/path lookup stays first-class
- BM25 should stay around as bootstrap/fallback/control for users who do not want Qdrant or when Qdrant is unavailable instead of turning the machine into vector-only confidence soup
What got verified:
- `cargo fmt --all` passed
- `cargo test -p codex-core -p codex-app-server-protocol -p codex-app-server --lib --no-run` passed with `CARGO_TARGET_DIR=C:\Users\Meta\.cargo-target-codex`
- targeted Phase 4 tests passed:
- `cargo test -p codex-core --lib retrieve_workspace_`
- `cargo test -p codex-app-server-protocol --lib thread_epiphany_retrieve`
- `cargo test -p codex-app-server --lib map_epiphany_retrieve_response_preserves_summary_and_results`
- full `cargo test -p codex-app-server-protocol` passed after regenerating stable schema fixtures
Notable verification fixes:
- `codex_message_processor.rs` needed the live-thread retrieval helper call to stop pretending `&CodexThread` needed `.as_ref()`
- `core/src/session/tests.rs` needed `retrieval: None` in the prompt fixture Epiphany state
- the exact-hit retrieval test fixture needed a genuinely matchable file name (`session_checkpoint.rs`)
- `EpiphanyThreadState` needed an explicit exemption in the protocol export test because sparse durable thread state intentionally uses optional fields
Important schema caveat:
- `cargo run -p codex-app-server-protocol --bin write_schema_fixtures -- --experimental` rewrites the same checked-in schema tree as stable generation
- leaving the repo in that state makes stable protocol tests fail
- the safe checked-in state for this slice is the stable fixture generation output
- after inspection, do not bolt durable Epiphany-state writes onto `thread/epiphany/retrieve` just because the retrieval summary exists:
- current Epiphany snapshots are persisted on real user-turn boundaries
- out-of-band durable retrieval writes would need new rollout semantics and are a larger machine than this slice deserves
## What Already Landed
Phase 1 is done.
@ -119,8 +204,8 @@ Use it when you need the answer to "how does current Epiphany differ from plain
If a future session wakes up from compaction and starts bluffing, this is the part to staple to its forehead.
- Phase 1, Phase 2, and the minimal Phase 3 read surface are all landed and verified.
- The latest pushed implementation anchor before this handoff sync is `640e063`:
- `Document pre-compaction Epiphany workflow`
- The latest local implementation anchor before the current in-flight retrieval slice is `d45ab10`:
- `Add Epiphany delta algorithmic map`
- `vendor/codex` is ordinary tracked repo content, not a submodule.
- Phase 2 means Codex now **reads** Epiphany state during turn construction:
- `SessionState.epiphany_state` is the internal activation signal
@ -135,6 +220,15 @@ If a future session wakes up from compaction and starts bluffing, this is the pa
- The next phase is **Phase 4 repo-local hybrid retrieval**:
- give Epiphany a real repo retrieval subsystem instead of repeated file-by-file shell archaeology
- keep it typed, additive, and GUI-friendly
- Phase 4 slice 1 is now verified in the working tree:
- protocol/core/app-server wiring exists and passes targeted verification
- stable app-server protocol schema fixtures were regenerated
- `write_schema_fixtures --experimental` is a trap if it is left as the checked-in tree
- The recovered bounded implementation choice for the first slice is:
- exact/path lookup via existing `codex_file_search`
- semantic lookup via local BM25 chunk search
- query-time, read-only, loaded-thread-first behavior
- no embeddings or persistent vector store yet
- Important Windows verification footgun still stands:
- use `CARGO_TARGET_DIR=C:\Users\Meta\.cargo-target-codex` for `codex-core` work on this machine
- Snapshot hygiene note:
@ -181,6 +275,13 @@ Verified:
- `cargo test -p codex-core --lib latest_epiphany_state_from_rollout_items`
- `cargo test -p codex-app-server --lib load_epiphany_state_from_rollout_path_reads_latest_snapshot`
- `cargo test -p codex-app-server-protocol --lib serialize_client_response`
- `cargo fmt --all` passed for the Phase 4 retrieval slice
- `cargo test -p codex-core -p codex-app-server-protocol -p codex-app-server --lib --no-run` passed for the Phase 4 retrieval slice when `CARGO_TARGET_DIR=C:\Users\Meta\.cargo-target-codex`
- targeted Phase 4 tests passed:
- `cargo test -p codex-core --lib retrieve_workspace_`
- `cargo test -p codex-app-server-protocol --lib thread_epiphany_retrieve`
- `cargo test -p codex-app-server --lib map_epiphany_retrieve_response_preserves_summary_and_results`
- full `cargo test -p codex-app-server-protocol` passed after restoring stable schema fixtures
Important Windows footgun:
@ -192,23 +293,20 @@ Without that, you get to learn about `symlink_dir failed: ... A required privile
## Recommended Next Implementation
Do **Phase 4 repo-local hybrid retrieval** next.
Do not restart verification from superstition. The current Phase 4 slice is already verified in the working tree.
The durable state seam exists, the turn loop now reads it, and clients can now load typed Epiphany state directly. The next clean move is to stop making every future agent rediscover the repo with shell spelunking.
The durable state seam exists, the turn loop reads it, clients can load typed Epiphany state directly, and the first retrieval slice is now real. The next clean move is to ship that verified slice cleanly instead of widening it before Qdrant is even online.
Planned slice:
Current next implementation move:
1. define retrieval state and shard/index summaries in typed Epiphany state
2. add a bounded hybrid retrieval surface that combines exact search with semantic chunk lookup
3. keep the first retrieval slice additive and internal/dev-usable first
4. leave watcher-driven invalidation, heavy GUI reflection, and specialist-agent scheduling for later slices
Main files for the next slice:
- `vendor/codex/codex-rs/protocol/src/`
- `vendor/codex/codex-rs/core/src/`
- `vendor/codex/codex-rs/app-server/src/`
- `vendor/codex/codex-rs/app-server-protocol/src/`
1. clean up, stage, commit, and push the current verified query-time hybrid retriever as the Phase 4 slice 1 landing zone
2. do not casually add durable retrieval-summary writes to `thread/epiphany/retrieve`
3. if a follow-up is needed later, keep it narrow:
- retrieval persistence/freshness semantics
- probably via a dedicated Epiphany update path
- Qdrant-backed persistent semantic indexing as the preferred backend, with BM25 fallback for no-Qdrant or degraded mode
- not GUI work
- not watcher-driven invalidation
## What Not To Do Next