214 lines
8.1 KiB
Markdown
214 lines
8.1 KiB
Markdown
# Codex Epiphany Phase 4 Implementation Plan
|
|
|
|
## Status
|
|
|
|
Updated on 2026-04-24 after verifying the first bounded Phase 4 slice in the working tree:
|
|
|
|
- Phase 1 durable Epiphany thread state
|
|
- Phase 2 prompt integration
|
|
- a minimal Phase 3 typed client read surface via `Thread.epiphanyState`
|
|
- a verified Phase 4 slice 1 query-time hybrid retriever across protocol, core, app-server protocol, and app-server
|
|
|
|
The next job is no longer to prove the retriever exists. That part is done. The next job is to keep the landing zone honest: do not casually turn `thread/epiphany/retrieve` into a durable Epiphany-state writer just because the state object has a `retrieval` field, and do not regress into another blob-shaped persistent semantic store when the time comes to add one.
|
|
|
|
## Summary
|
|
|
|
Land Phase 4 as a **bounded internal retrieval slice**:
|
|
|
|
- add typed retrieval state and shard/index summaries to Epiphany state
|
|
- add one additive app-server retrieval query surface for loaded threads
|
|
- support hybrid retrieval from day one:
|
|
- exact/path/symbol/lexical results
|
|
- semantic chunk results
|
|
- keep the slice internal/dev-usable first
|
|
|
|
After this slice:
|
|
|
|
- Epiphany can ask structured repo questions through one typed surface
|
|
- GUI work has a real data source later
|
|
- retrieval can evolve without making prompt text or shell transcripts the canonical knowledge path
|
|
|
|
This slice should still avoid:
|
|
|
|
- watcher-driven invalidation
|
|
- automatic observation promotion
|
|
- GUI implementation
|
|
- specialist-agent scheduling
|
|
- user-facing activation flows
|
|
|
|
## Phase 4 Principle
|
|
|
|
Do not build "vector search" as a separate novelty organ off to the side.
|
|
|
|
Build a **hybrid repo retrieval subsystem** with one typed query shape and one typed result shape. Exact lookup and semantic lookup should be different modes of the same machine, not rival religions.
|
|
|
|
## Recovered bounded decision
|
|
|
|
After rehydrating from the repo state and the lone dirty working-tree diff, the first bounded Phase 4 slice should be:
|
|
|
|
- query-time
|
|
- loaded-thread-first
|
|
- read-only
|
|
- hybrid from day one
|
|
|
|
Concrete choice for the first implementation:
|
|
|
|
- reuse existing `codex_file_search` substrate for exact/path-ish lookup
|
|
- use workspace-local BM25 chunk search as the initial semantic backend
|
|
- defer embeddings, vector stores, persistent indexing, and watcher-driven invalidation
|
|
|
|
Updated follow-up direction after landing the verified slice:
|
|
|
|
- keep the BM25 implementation as the proven baseline/fallback
|
|
- for the first persistent semantic backend, prefer Qdrant over JSON/blob/postgres-thrash designs, but do not make it a hard requirement for basic Epiphany use
|
|
- do not turn retrieval into vector-only soup; exact/path lookup remains first-class and BM25 remains useful as bootstrap/control/fallback when Qdrant is unavailable
|
|
|
|
Preflight note:
|
|
|
|
- use tracked source as the corpus estimate, not raw working-tree disk usage
|
|
- at `vendor/codex` HEAD `d45ab10`, tracked `codex-rs` source is `3642` files / `31.09 MB`
|
|
- raw working-tree size under `codex-rs` can be heavily inflated by build artifacts and will lie to the design
|
|
|
|
Current in-flight implementation note:
|
|
|
|
- the protocol scaffold still exists in `vendor/codex/codex-rs/protocol/src/protocol.rs`
|
|
- it is now wired through the working tree across:
|
|
- `core/src/epiphany_retrieval.rs`
|
|
- `core/src/codex_thread.rs`
|
|
- `core/src/session/tests.rs`
|
|
- `app-server-protocol/src/protocol/common.rs`
|
|
- `app-server-protocol/src/protocol/v2.rs`
|
|
- `app-server-protocol/src/export.rs`
|
|
- `app-server/src/codex_message_processor.rs`
|
|
- the slice is now formatted and verified in the working tree:
|
|
- `cargo fmt --all` passed
|
|
- targeted no-run compile for `codex-core`, `codex-app-server-protocol`, and `codex-app-server` passed
|
|
- targeted Phase 4 tests passed in core, app-server protocol, and app-server
|
|
- full `cargo test -p codex-app-server-protocol` passed after regenerating stable schema fixtures
|
|
- stable schema fixtures are the safe checked-in state for this slice; `write_schema_fixtures --experimental` rewrites the same tree and cannot be left behind as the repo state
|
|
|
|
## Key Changes
|
|
|
|
### 1. Extend Epiphany state with retrieval metadata
|
|
|
|
Touch:
|
|
|
|
- `vendor/codex/codex-rs/protocol/src/protocol.rs`
|
|
|
|
Add minimal retrieval metadata to `EpiphanyThreadState`, for example:
|
|
|
|
- `retrieval`
|
|
- `workspace_root`
|
|
- `index_revision`
|
|
- `status`
|
|
- `semantic_available`
|
|
- `last_indexed_at`
|
|
- `shards`
|
|
- `dirty_paths`
|
|
|
|
Keep this metadata summary-focused. Do not dump raw embeddings, giant posting lists, or per-file sludge into thread state.
|
|
|
|
### 2. Add a core retrieval facade
|
|
|
|
Touch:
|
|
|
|
- `vendor/codex/codex-rs/core/src/`
|
|
|
|
Add a focused retrieval module, tentatively:
|
|
|
|
- `epiphany_retrieval.rs`
|
|
|
|
Responsibilities:
|
|
|
|
- accept a loaded thread plus query params
|
|
- resolve the relevant workspace root
|
|
- run exact retrieval and semantic retrieval behind one interface
|
|
- return typed ranked results with file paths, line anchors, excerpts, and retrieval mode metadata
|
|
|
|
Result types should be explicit and bounded:
|
|
|
|
- exact hits
|
|
- semantic hits
|
|
- optional graph-linked ids later
|
|
|
|
Do not make the first cut depend on prompt parsing or transcript scraping.
|
|
|
|
### 3. Keep the first semantic backend small and local
|
|
|
|
The first semantic backend should be workspace-local BM25 chunk retrieval built at query time over the tracked source corpus.
|
|
|
|
Requirements:
|
|
|
|
- keep chunk text line-anchored and path-aware
|
|
- keep the implementation bounded enough to understand in one sitting
|
|
- track retrieval/index freshness metadata explicitly in typed state
|
|
- do not introduce a monolithic persistent blob just to feel sophisticated
|
|
|
|
If the first semantic backend needs to be intentionally narrow, that is fine. A small honest subsystem beats a magical blob with terrible write behavior.
|
|
|
|
### 4. Add one additive app-server query surface
|
|
|
|
Touch:
|
|
|
|
- `vendor/codex/codex-rs/app-server-protocol/src/protocol/v2.rs`
|
|
- `vendor/codex/codex-rs/app-server/src/codex_message_processor.rs`
|
|
- `vendor/codex/codex-rs/app-server/README.md`
|
|
|
|
Add one typed method for retrieval, tentatively:
|
|
|
|
- `thread/epiphany/retrieve`
|
|
|
|
Suggested request fields:
|
|
|
|
- `threadId`
|
|
- `query`
|
|
- optional `cwd`
|
|
- optional `modes` or `preferSemantic`
|
|
- optional `limit`
|
|
- optional `pathPrefixes`
|
|
|
|
Suggested response fields:
|
|
|
|
- `query`
|
|
- `indexSummary`
|
|
- `results`
|
|
|
|
Keep it additive and read-only. No write/update/index-control methods yet unless they turn out to be mechanically unavoidable.
|
|
|
|
### 5. Reuse existing exact-search substrate where practical
|
|
|
|
Codex already has fuzzy/exact-ish repo search substrate. Reuse what is useful instead of building a second exact-search toy from scratch.
|
|
|
|
But do not confuse that with the whole answer. Phase 4 exists because exact file search alone is not enough.
|
|
|
|
### 6. Test the machine, not just the types
|
|
|
|
Add tests for:
|
|
|
|
1. protocol serde for the new retrieval request/response shapes
|
|
2. core ranking/merging behavior for mixed exact + semantic results
|
|
3. app-server handling for the new request
|
|
4. retrieval metadata inclusion in `EpiphanyThreadState`
|
|
5. a smoke test that proves semantic retrieval can find a concept that exact-name search would miss
|
|
|
|
## Assumptions
|
|
|
|
- Phase 4 is still internal/dev-usable only.
|
|
- Retrieval should be thread/workspace aware, not a global cross-repo soup.
|
|
- The first retrieval slice can be read-only.
|
|
- Watcher-driven invalidation belongs later; this slice only needs enough metadata to admit freshness honestly.
|
|
|
|
## Immediate Next Step After This Plan
|
|
|
|
Treat the current retrieval baseline as real and ship it cleanly:
|
|
|
|
1. clean up, stage, and push the current verified query-time hybrid retriever as the Phase 4 slice 1 landing zone
|
|
2. do not add durable retrieval-summary writes from `thread/epiphany/retrieve` without a clean out-of-band rollout/update semantic
|
|
3. if a later follow-up is needed, keep it tight:
|
|
- retrieval freshness/persistence semantics
|
|
- probably a dedicated Epiphany update path
|
|
- Qdrant-backed persistent semantic indexing as the preferred backend, with BM25 fallback for no-Qdrant or degraded mode
|
|
- not GUI work
|
|
- not watcher-driven invalidation
|
|
|
|
Do not start with GUI. Do not start with automatic graph invalidation. Do not pretend shell transcripts are a retrieval strategy.
|