diff --git a/notes/codex-epiphany-mode-plan.md b/notes/codex-epiphany-mode-plan.md index 29388532..e2d3ad31 100644 --- a/notes/codex-epiphany-mode-plan.md +++ b/notes/codex-epiphany-mode-plan.md @@ -2,7 +2,7 @@ ## Status -Updated on 2026-04-24 after verifying the first bounded Phase 4 slice in the working tree: +Updated on 2026-04-24 after landing the first bounded Phase 4 slice on `main`: - Phase 1 durable Epiphany thread state - Phase 2 prompt integration @@ -69,7 +69,7 @@ Preflight note: - at `vendor/codex` HEAD `d45ab10`, tracked `codex-rs` source is `3642` files / `31.09 MB` - raw working-tree size under `codex-rs` can be heavily inflated by build artifacts and will lie to the design -Current in-flight implementation note: +Current landed implementation note: - the protocol scaffold still exists in `vendor/codex/codex-rs/protocol/src/protocol.rs` - it is now wired through the working tree across: @@ -80,7 +80,7 @@ Current in-flight implementation note: - `app-server-protocol/src/protocol/v2.rs` - `app-server-protocol/src/export.rs` - `app-server/src/codex_message_processor.rs` -- the slice is now formatted and verified in the working tree: +- the slice is now formatted, verified, committed, and pushed on `main`: - `cargo fmt --all` passed - targeted no-run compile for `codex-core`, `codex-app-server-protocol`, and `codex-app-server` passed - targeted Phase 4 tests passed in core, app-server protocol, and app-server @@ -200,9 +200,9 @@ Add tests for: ## Immediate Next Step After This Plan -Treat the current retrieval baseline as real and ship it cleanly: +Treat the current retrieval baseline as real and only grow it when the next bounded follow-up is ready: -1. clean up, stage, and push the current verified query-time hybrid retriever as the Phase 4 slice 1 landing zone +1. treat the current verified-and-landed query-time hybrid retriever as the Phase 4 slice 1 baseline 2. do not add durable retrieval-summary writes from `thread/epiphany/retrieve` without a clean out-of-band rollout/update semantic 3. if a later follow-up is needed, keep it tight: - retrieval freshness/persistence semantics diff --git a/notes/fresh-workspace-handoff.md b/notes/fresh-workspace-handoff.md index 822fbe28..3f881abe 100644 --- a/notes/fresh-workspace-handoff.md +++ b/notes/fresh-workspace-handoff.md @@ -20,12 +20,13 @@ Important consequence: - `vendor/codex` is now tracked directly in the parent repo as ordinary files - it is **not** a submodule anymore -Recent anchor commits before the current in-flight work: +Recent anchor commits: - `2042687e3035c5a86d7f6aa66306d87abcc10f2d` - vendored Codex and landed the Phase 1 Epiphany core state slice - `c823815` - persisted the Phase 2 implementation plan and refreshed the project handoff/state - `efd1420` - landed and pushed the Phase 2 prompt-integration slice - `640e063` - documented the pre-compaction Epiphany workflow and synced the repo memory before compaction +- `360dfea` - landed and pushed the Phase 4 hybrid retrieval slice Do not trust this note for the exact current HEAD; use `git log --oneline -1` if you need the live commit. @@ -45,11 +46,11 @@ Current architectural/spec notes: - `notes/epiphany-current-algorithmic-map.md` - `notes/epiphany-core-harness-surfaces.md` -## Current In-Flight Work +## Current Landed Baseline -Phase 4 repo-local hybrid retrieval is now verified in the working tree. It is still uncommitted, but it is no longer a vague scaffold or a maybe. +Phase 4 repo-local hybrid retrieval is now landed on `main` at `360dfea` and pushed to `origin/main`. It is no longer a vague scaffold, a maybe, or a working-tree-only event. -Current working-tree implementation spans: +Current landed implementation spans: - `vendor/codex/codex-rs/protocol/src/protocol.rs` - `vendor/codex/codex-rs/core/src/epiphany_retrieval.rs` @@ -204,8 +205,8 @@ Use it when you need the answer to "how does current Epiphany differ from plain If a future session wakes up from compaction and starts bluffing, this is the part to staple to its forehead. - Phase 1, Phase 2, and the minimal Phase 3 read surface are all landed and verified. -- The latest local implementation anchor before the current in-flight retrieval slice is `d45ab10`: - - `Add Epiphany delta algorithmic map` +- The current landed Phase 4 anchor is `360dfea`: + - `Land Epiphany Phase 4 hybrid retrieval slice` - `vendor/codex` is ordinary tracked repo content, not a submodule. - Phase 2 means Codex now **reads** Epiphany state during turn construction: - `SessionState.epiphany_state` is the internal activation signal @@ -217,11 +218,11 @@ If a future session wakes up from compaction and starts bluffing, this is the pa - stored thread reads reconstruct from rollout with the same rollback/compaction semantics as core - there are still no dedicated Epiphany update RPCs or live `thread/epiphany/*` notifications - The next phase is **not** GUI work. -- The next phase is **Phase 4 repo-local hybrid retrieval**: - - give Epiphany a real repo retrieval subsystem instead of repeated file-by-file shell archaeology - - keep it typed, additive, and GUI-friendly -- Phase 4 slice 1 is now verified in the working tree: - - protocol/core/app-server wiring exists and passes targeted verification +- The current landed phase is **Phase 4 slice 1 repo-local hybrid retrieval**: + - Epiphany now has a real typed repo retrieval subsystem instead of repeated file-by-file shell archaeology + - the next bounded follow-up is persistent semantic indexing, not proving the retriever exists again +- Phase 4 slice 1 is now landed on `main` and was verified before landing: + - protocol/core/app-server wiring exists and passed targeted verification - stable app-server protocol schema fixtures were regenerated - `write_schema_fixtures --experimental` is a trap if it is left as the checked-in tree - The recovered bounded implementation choice for the first slice is: @@ -295,18 +296,18 @@ Without that, you get to learn about `symlink_dir failed: ... A required privile Do not restart verification from superstition. The current Phase 4 slice is already verified in the working tree. -The durable state seam exists, the turn loop reads it, clients can load typed Epiphany state directly, and the first retrieval slice is now real. The next clean move is to ship that verified slice cleanly instead of widening it before Qdrant is even online. +The durable state seam exists, the turn loop reads it, clients can load typed Epiphany state directly, and the first retrieval slice is now real and landed. The next clean move is the bounded persistent-semantic follow-up after reboot, not pretending the already-shipped slice still needs ceremony. Current next implementation move: -1. clean up, stage, commit, and push the current verified query-time hybrid retriever as the Phase 4 slice 1 landing zone -2. do not casually add durable retrieval-summary writes to `thread/epiphany/retrieve` -3. if a follow-up is needed later, keep it narrow: +1. keep the current verified-and-landed query-time hybrid retriever as the Phase 4 slice 1 baseline +2. after reboot and Qdrant availability, keep the next follow-up narrow: - retrieval persistence/freshness semantics - probably via a dedicated Epiphany update path - Qdrant-backed persistent semantic indexing as the preferred backend, with BM25 fallback for no-Qdrant or degraded mode - not GUI work - not watcher-driven invalidation +3. do not casually add durable retrieval-summary writes to `thread/epiphany/retrieve` ## What Not To Do Next @@ -320,7 +321,7 @@ Not yet: The next slice should stay small and mean: -- add typed retrieval scaffolding +- add persistent semantic retrieval/indexing scaffolding - keep GUI, mutation, and specialist-agent logic dark ## Verification Commands diff --git a/state/evidence.jsonl b/state/evidence.jsonl index d286dead..79fdeb82 100644 --- a/state/evidence.jsonl +++ b/state/evidence.jsonl @@ -53,3 +53,4 @@ {"ts": "2026-04-24T02:11:38+01:00", "type": "decision", "status": "ok", "note": "Deferred durable retrieval-summary persistence beyond the verified Phase 4 slice. EpiphanyState snapshots are still persisted on real user-turn boundaries, so teaching thread/epiphany/retrieve to append durable state now would require a new out-of-band rollout semantic or a fake turn boundary; keep the current slice as the landing zone instead of widening the machine casually."} {"ts": "2026-04-24T02:11:38+01:00", "type": "decision", "status": "ok", "note": "Adopted Qdrant as the intended first persistent semantic retrieval backend after the verified BM25 baseline. The rationale is practical rather than fashionable: the user is already moving away from a blob-thrashing Postgres/JSON embedding store in adjacent infrastructure, so Epiphany should target a real vector DB for persistent semantic indexing while keeping exact/path search first-class and retaining BM25 as bootstrap/fallback/control rather than going vector-only."} {"ts": "2026-04-24T02:43:41+01:00", "type": "decision", "status": "ok", "note": "Resolved the remaining backend policy fork before shipping Phase 4 slice 1: Qdrant is the preferred later persistent semantic backend, but not a hard runtime requirement for basic Epiphany use. The verified BM25 retriever remains the no-hassle bootstrap/fallback/control path for users who do not want Qdrant or when Qdrant is unavailable."} +{"ts": "2026-04-24T03:21:18+01:00", "type": "verification", "status": "ok", "note": "Collapsed the temporary phase branch after push so the landed Phase 4 slice now lives directly on main at 360dfea with a clean worktree. Synced map, scratch, plan, and handoff state so the persisted next action is the post-reboot Qdrant-backed persistent-semantic follow-up rather than stale push prep."} diff --git a/state/map.yaml b/state/map.yaml index 95db0089..fbfcb64d 100644 --- a/state/map.yaml +++ b/state/map.yaml @@ -45,9 +45,9 @@ data_flow: - commit or backtrack active_subgoals: - - Clean up and ship the verified Phase 4 repo-local hybrid retrieval slice without widening scope. - - Keep the canonical map, handoff note, evidence log, and checked-in stable schema state aligned with the verified landing zone. - - Preserve BM25 as the no-hassle fallback while treating Qdrant as the preferred later persistent semantic backend. + - Keep the landed Phase 4 slice as the retrieval baseline and keep the repo memory aligned with reality. + - After reboot and Qdrant availability, design the next bounded persistent semantic follow-up without widening the current machine casually. + - Preserve exact/path lookup and BM25 fallback while treating Qdrant as the preferred persistent semantic backend. accepted_design: - external typed state files @@ -96,5 +96,5 @@ rejected_paths: reason: audited candidates were either terminal-only, stale/aspirational, or incoherent wrappers; the comparison clones were removed from vendor to avoid building on untrusted drift current_status: - summary: Workspace initialized with protocol files, typed state, a tiny CLI, a git repo, fresh-workspace handoff notes, architecture rationale, a Codex algorithmic map note, an Epiphany core-harness surface spec, and a rolling implementation plan against vendored Codex. Phase 1 durable Epiphany state, Phase 2 prompt integration, and a minimal Phase 3 typed client read surface are all implemented and verified with a real Rust/MSVC toolchain; vendored Codex is tracked directly in the parent repo; app-server Thread payloads can now carry typed Epiphany state without scraping prompt text; pre-compaction persistence is now an explicit part of the intended Epiphany workflow; and the first bounded Phase 4 repo-local hybrid retrieval slice is now wired and verified in the working tree across protocol, core, app-server protocol, and app-server, including stable schema fixture regeneration and the Windows `CARGO_TARGET_DIR` verification path. - next_action: Clean up and push the current verified query-time hybrid retriever as the Phase 4 slice 1 landing zone. Do not bolt durable EpiphanyState writes onto `thread/epiphany/retrieve` without a clean out-of-band rollout semantic; after the push and reboot, the next persistent semantic retrieval slice should target a Qdrant-backed store with explicit freshness/indexing semantics as the preferred semantic backend while BM25 remains the fallback when Qdrant is unavailable. + summary: Workspace initialized with protocol files, typed state, a tiny CLI, a git repo, fresh-workspace handoff notes, architecture rationale, a Codex algorithmic map note, an Epiphany core-harness surface spec, and a rolling implementation plan against vendored Codex. Phase 1 durable Epiphany state, Phase 2 prompt integration, and a minimal Phase 3 typed client read surface are all implemented and verified with a real Rust/MSVC toolchain; vendored Codex is tracked directly in the parent repo; app-server Thread payloads can now carry typed Epiphany state without scraping prompt text; pre-compaction persistence is now an explicit part of the intended Epiphany workflow; and the first bounded Phase 4 repo-local hybrid retrieval slice is now landed on `main` at `360dfea`, including stable schema fixture regeneration and the Windows `CARGO_TARGET_DIR` verification path. + next_action: After reboot and Qdrant availability, prototype the next bounded persistent semantic retrieval follow-up as a Qdrant-backed backend with explicit freshness/indexing semantics while keeping the landed query-time BM25 retriever as fallback and exact/path lookup first-class. Do not bolt durable EpiphanyState writes onto `thread/epiphany/retrieve` without a clean out-of-band rollout semantic. diff --git a/state/scratch.md b/state/scratch.md index 5a5dfb97..40a016eb 100644 --- a/state/scratch.md +++ b/state/scratch.md @@ -4,11 +4,11 @@ This file is intentionally disposable. ## Current Subgoal -- Clean up, persist, and ship the verified first bounded Phase 4 repo-local hybrid retrieval slice in vendored Codex. +- Rebase the repo memory onto the landed Phase 4 baseline and make the next Qdrant follow-up explicit. ## Working Notes -- The old lone-diff situation is gone. The current in-flight Phase 4 wiring now spans: +- The first bounded Phase 4 slice is landed on `main` at `360dfea`. The landed retrieval wiring spans: - `vendor/codex/codex-rs/protocol/src/protocol.rs` - `vendor/codex/codex-rs/core/src/epiphany_retrieval.rs` - `vendor/codex/codex-rs/core/src/codex_thread.rs` @@ -39,6 +39,7 @@ This file is intentionally disposable. - `cargo test -p codex-core -p codex-app-server-protocol -p codex-app-server --lib --no-run` passed with `CARGO_TARGET_DIR=C:\Users\Meta\.cargo-target-codex` - targeted Phase 4 tests passed in core, app-server protocol, and app-server - full `cargo test -p codex-app-server-protocol` passed after regenerating stable schema fixtures + - the slice is now committed and pushed on `main` as `360dfea` - Small fallout that had to be patched: - `codex_message_processor.rs` had an unnecessary `live_thread.as_ref()` - `core/src/session/tests.rs` needed `retrieval: None` in the prompt fixture Epiphany state @@ -50,7 +51,7 @@ This file is intentionally disposable. - the checked-in repo state must stay on the stable fixture generation path - despite the giant `git status` scream wall, the real schema content diff is small and expected: - `git diff --numstat` shows actual content changes in `15` generated schema files - - plus the new untracked Epiphany TypeScript schema files + - plus the newly tracked Epiphany TypeScript schema files - the broader wall is mostly line-ending/worktree noise, not extra logical surface area - After inspecting the live code path, do not widen `thread/epiphany/retrieve` into a durable Epiphany-state write right now: - durable `EpiphanyState` snapshots are currently persisted on real user-turn boundaries