152 lines
14 KiB
Markdown
152 lines
14 KiB
Markdown
# Scratch
|
|
|
|
This file is intentionally disposable.
|
|
|
|
## Current Subgoal
|
|
|
|
- Move from structural map/churn promotion validation toward richer observation-to-map proposal machinery without hidden writes.
|
|
|
|
## Working Notes
|
|
|
|
- The landed Phase 4 retrieval/indexing/core-extraction baseline is now `80c29e0` on `main`. The older `360dfea` commit is only the first hybrid retrieval anchor, not the current repo state.
|
|
- The extraction boundary is now the important truth:
|
|
- `epiphany-core/src/retrieval.rs` owns the heavy hybrid retrieval/indexing engine
|
|
- `epiphany-core/src/prompt.rs` owns the Epiphany prompt-state renderer
|
|
- `epiphany-core/src/rollout.rs` owns the latest-Epiphany-state replay helper used for Phase 3 stored-thread hydration
|
|
- `epiphany-core/src/lib.rs` re-exports the stable surface used by vendored Codex
|
|
- `vendor/codex/codex-rs/core/src/epiphany_retrieval.rs` is now a thin re-export wrapper
|
|
- `vendor/codex/codex-rs/core/src/epiphany_rollout.rs` is now a thin wrapper that supplies the Codex-specific user-turn-boundary predicate
|
|
- `vendor/codex/codex-rs/core/src/context/epiphany_state_instructions.rs` is now a thin `ContextualUserFragment` adapter around `epiphany_core::render_epiphany_state(...)`
|
|
- `vendor/codex/codex-rs/core/Cargo.toml` now depends on the sibling crate by path
|
|
- What still must live in vendored Codex because that is where the typed host seam exists:
|
|
- protocol types in `vendor/codex/codex-rs/protocol/src/protocol.rs`
|
|
- `CodexThread` bridge methods in `vendor/codex/codex-rs/core/src/codex_thread.rs`
|
|
- the user-turn-boundary rule in `vendor/codex/codex-rs/core/src/context_manager/history.rs`
|
|
- app-server protocol surfaces in `vendor/codex/codex-rs/app-server-protocol/src/protocol/common.rs` and `vendor/codex/codex-rs/app-server-protocol/src/protocol/v2.rs`
|
|
- request routing in `vendor/codex/codex-rs/app-server/src/codex_message_processor.rs`
|
|
- Why this shape is worth keeping:
|
|
- it shrinks the Apache-mixed surface
|
|
- it makes modified Codex alone less useful as a standalone Epiphany rebuild kit
|
|
- it preserves first-class typed integration instead of hiding the machine behind an opaque plugin blob
|
|
- Retrieval/indexing behavior is still the bounded machine we wanted:
|
|
- exact/path-ish lookup via `codex_file_search`
|
|
- semantic lookup via BM25 chunk search when persistent semantic state is unavailable or stale
|
|
- explicit `thread/epiphany/index` for Qdrant-backed semantic persistence
|
|
- read-only `thread/epiphany/retrieve`
|
|
- manifest metadata under `codex_home`
|
|
- local Ollama embeddings defaulting to `qwen3-embedding:0.6b`
|
|
- Corpus preflight for the tracked `vendor/codex/codex-rs` source at `vendor/codex` HEAD `d45ab10`: `3642` tracked files / `31.09 MB`.
|
|
- Do not use raw working-tree size as the retrieval denominator here; build artifacts and other sludge lie.
|
|
- Verification is now split cleanly across the repo-owned crate and the vendored host:
|
|
- `cargo fmt --manifest-path .\\epiphany-core\\Cargo.toml`
|
|
- `cargo test --manifest-path .\\epiphany-core\\Cargo.toml`
|
|
- `cargo fmt --all`
|
|
- `cargo test -p codex-core --lib epiphany`
|
|
- `cargo test -p codex-app-server-protocol --lib thread_epiphany_`
|
|
- `cargo test -p codex-app-server --lib map_epiphany_`
|
|
- `cargo test -p codex-core -p codex-app-server-protocol -p codex-app-server --lib --no-run` with `CARGO_TARGET_DIR=C:\Users\Meta\.cargo-target-codex`
|
|
- Current repo sanity task:
|
|
- keep docs from describing the already-landed Qdrant/indexing/core-extraction work as a dirty working-tree follow-up
|
|
- keep README, map, plan, handoff, and Epiphany algorithmic map in agreement about the current landing zone
|
|
- keep `notes/epiphany-current-algorithmic-map.md` focused on exact current control flow, not general fork-positioning prose
|
|
- Local dependency sanity:
|
|
- Qdrant at `http://127.0.0.1:6333` responds and currently has VoidBot collections
|
|
- Ollama at `http://127.0.0.1:11434` exposes the default `qwen3-embedding:0.6b` model
|
|
- Mechanical honesty still matters:
|
|
- do not widen `thread/epiphany/retrieve` into a durable Epiphany-state write without a clean out-of-band rollout/update semantic
|
|
- do not let `epiphany-core` sprawl into GUI or watcher machinery just because it now owns the bigger organ
|
|
- do not add metaphors or natural-language modeling prose to an algorithmic map until the relevant source is in context and the prose is anchored to concrete refs
|
|
- meticulous workflow discipline is not ceremony for its own sake; the useful loop is rehydrate, bound the slice, read the code being described, make the smallest coherent change, verify the seam that matters, and persist the learned state before charging onward
|
|
- Source-grounded Epiphany map audit on 2026-04-24 reread the cited protocol/session/rollout/prompt/thread/app-server/retrieval code before editing:
|
|
- sharpened the distinction between full core resume reconstruction and the narrower `epiphany-core` stored-thread replay helper
|
|
- removed unsupported broad wording around thread hydration surfaces and anchored the actual live/reconstructed paths
|
|
- made the retrieval branch explicit: exact search always runs, Qdrant is used only with a fresh compatible manifest/collection and working Ollama query embedding, otherwise query-time BM25 supplies semantic chunks
|
|
- corrected indexing wording so the explicit index path returns ready state on successful writes or an error, not a casual stale state
|
|
- Live app-server smoke on 2026-04-24:
|
|
- built `codex-app-server.exe` with `CARGO_TARGET_DIR=C:\Users\Meta\.cargo-target-codex`
|
|
- used an isolated `CODEX_HOME` under `.epiphany-smoke/codex-home`
|
|
- started an ephemeral loaded app-server thread rooted at `E:\Projects\EpiphanyAgent\epiphany-core`
|
|
- preflighted the bounded smoke corpus as 5 tracked files / 106,748 bytes
|
|
- `thread/epiphany/index` with real Qdrant/Ollama returned ready state: 5 files, 198 chunks, `qdrant-ollama-v1:qwen3-embedding:0.6b`
|
|
- Qdrant collection `epiphany_workspace_fa24bab116f8d229` was created and the manifest landed under the isolated Codex home
|
|
- `thread/epiphany/retrieve` returned persistent semantic chunks from `src/prompt.rs`, proving the fresh Qdrant path is used after explicit indexing
|
|
- env/default backend config is sufficient for this slice; do not add a first-class config surface yet
|
|
- unrelated app-server startup noise remains: project trust warning, plugin sync 403, and unauthenticated OpenAI websocket warning
|
|
- Typed state-update implementation on 2026-04-24:
|
|
- added experimental loaded-thread-only `thread/epiphany/update`
|
|
- protocol patch accepts optional `expectedRevision`, append-only observations/evidence, and bounded replacements for objective, active subgoal, subgoals, invariants, graphs, graph frontier/checkpoint, scratch, churn, and mode
|
|
- `CodexThread.epiphany_update_state(...)` rejects empty patches, checks revision when supplied, applies the patch to live `SessionState`, increments revision, sets `last_updated_turn_id` from the current reference turn when available, persists `RolloutItem::EpiphanyState`, and flushes rollout
|
|
- replay helpers now accept an out-of-band Epiphany snapshot before the first user turn so seed updates survive resume
|
|
- stable app-server schema fixtures were regenerated; the export test now treats Epiphany DTOs as sparse durable state, not conventional response envelopes
|
|
- verification passed:
|
|
- `cargo test --manifest-path .\\epiphany-core\\Cargo.toml rollout`
|
|
- `cargo test -p codex-core --lib epiphany`
|
|
- `cargo test -p codex-app-server-protocol --lib thread_epiphany_`
|
|
- `cargo test -p codex-app-server --lib map_epiphany_`
|
|
- `cargo test -p codex-app-server-protocol`
|
|
- live app-server stdio smoke passed after rebuilding `codex-app-server.exe`:
|
|
- started ephemeral loaded thread `019dbffc-c19a-75e0-bf35-c780bee59a68`
|
|
- called `thread/epiphany/update` with `expectedRevision: 0`
|
|
- response revision was `1`
|
|
- `thread/read` returned `epiphanyState.revision == 1`, objective `Smoke-test explicit Epiphany update persistence`, observation `obs-update-smoke`, and evidence `ev-update-smoke`
|
|
- smoke result: `.epiphany-smoke/update-smoke-result.json`
|
|
- wire footgun: app-server envelope fields are camelCase, but nested reused Epiphany core DTO fields such as `source_kind`, `understanding_status`, and `recent_evidence` are snake_case
|
|
- Typed distillation/proposal implementation on 2026-04-24:
|
|
- added repo-owned `epiphany-core/src/distillation.rs`
|
|
- added thin vendored core re-export at `vendor/codex/codex-rs/core/src/epiphany_distillation.rs`
|
|
- added experimental loaded-thread-only `thread/epiphany/distill`
|
|
- distill is read-only: it returns `expectedRevision` plus a `ThreadEpiphanyUpdatePatch`, and callers must pass that patch to `thread/epiphany/update` to make it durable
|
|
- `distill_observation(...)` normalizes required `sourceKind`, `status`, and `text`; rejects empty required fields; builds a bounded summary; fingerprints source/status/subject/text into stable observation/evidence ids; defaults test/smoke/verification sources to evidence kind `verification`; and carries code refs into both records
|
|
- first smoke exposed that returning `expectedRevision: null` for brand-new threads was too loose, so the handler now defaults absent Epiphany state to revision `0`
|
|
- verification passed:
|
|
- `cargo test --manifest-path .\\epiphany-core\\Cargo.toml distill`
|
|
- `cargo test -p codex-core --lib distill`
|
|
- `cargo test -p codex-app-server-protocol --lib thread_epiphany_`
|
|
- `cargo test -p codex-app-server --lib thread_epiphany`
|
|
- `cargo test -p codex-app-server-protocol`
|
|
- `cargo build -p codex-app-server --bin codex-app-server`
|
|
- live app-server stdio smoke passed after rebuilding `codex-app-server.exe`:
|
|
- started ephemeral loaded thread `019dc028-99ce-7b03-8f89-65b072cc2cca`
|
|
- called `thread/epiphany/distill` with smoke input and a code ref to `src/distillation.rs`
|
|
- distill returned `expectedRevision: 0`, observation `obs-c9bfcac23d66`, and evidence `ev-c9bfcac23d66`
|
|
- passing the patch to `thread/epiphany/update` advanced state to revision `1`
|
|
- `thread/read` returned the generated observation/evidence ids
|
|
- smoke result: `.epiphany-smoke/distill-smoke-result.json`
|
|
- Typed promotion gate implementation on 2026-04-24:
|
|
- added repo-owned `epiphany-core/src/promotion.rs`
|
|
- added thin vendored core re-export at `vendor/codex/codex-rs/core/src/epiphany_promotion.rs`
|
|
- added experimental loaded-thread-only `thread/epiphany/promote`
|
|
- promotion evaluates a patch plus required verifier evidence; failed policy returns `accepted: false` with reasons and no state mutation
|
|
- accepted policy appends verifier evidence and applies the patch through `CodexThread.epiphany_update_state(...)`, preserving the durable update path as the only state writer
|
|
- initial policy requires an actual mutation, nonempty verifier evidence, accepting verifier status (`ok`, `accepted`, `verified`, `pass`, `passed`), valid evidence records, unique ids, and evidence-linked observations
|
|
- live app-server stdio smoke passed:
|
|
- started ephemeral loaded thread `019dc087-6d3d-7323-9a2f-2487a591ef7a`
|
|
- distilled a promotion smoke patch
|
|
- `thread/epiphany/promote` rejected failed verifier evidence with no `epiphanyState`
|
|
- accepted verifier evidence promoted through update to revision `1`
|
|
- `thread/read` returned observation `obs-77248505f492`, distill evidence `ev-77248505f492`, and verifier evidence `ev-promote-smoke-verifier`
|
|
- smoke result: `.epiphany-smoke/promote-smoke-result.json`
|
|
- Richer promotion safety layer on 2026-04-24:
|
|
- `epiphany-core/src/promotion.rs` now receives the actual replacement fields instead of only a `has_state_replacements` boolean
|
|
- state replacement promotions must include explicit observations and patch evidence before verifier evidence can stamp them into the durable update path
|
|
- subgoal, invariant, graph, graph link, frontier, checkpoint, and churn replacements get lightweight structural validation
|
|
- `vendor/codex/codex-rs/app-server/src/codex_message_processor.rs` still keeps the host seam thin: it passes the patch fields into `evaluate_promotion`, then accepted promotions go through `CodexThread.epiphany_update_state(...)`
|
|
- verification passed:
|
|
- `cargo test --manifest-path .\\epiphany-core\\Cargo.toml`
|
|
- `cargo test -p codex-core --lib epiphany`
|
|
- `cargo test -p codex-app-server --lib epiphany`
|
|
- attempted full `cargo test -p codex-app-server --lib` compiled but hit a pre-existing-looking stack overflow in `in_process_start_uses_requested_session_source_for_thread_start`; Epiphany-scoped app-server tests passed
|
|
- Stack-overflow trace/fix on 2026-04-24:
|
|
- the original `in_process_start_uses_requested_session_source_for_thread_start` failure reproduced without `RUST_MIN_STACK`
|
|
- breadcrumbs showed `thread/start` completed in the processor and the overflow happened before response routing, pointing at app-server request/future stack pressure rather than Epiphany promotion semantics
|
|
- `RUST_MIN_STACK=67108864` made exact failures pass; later exact failures in guardian lifecycle and thread-start tracing tests confirmed the app-server test binary had crossed a Windows stack threshold
|
|
- `vendor/codex/codex-rs/.cargo/config.toml` now sets `RUST_MIN_STACK=67108864` so Cargo-driven tests get the required stack without shell folklore
|
|
- large app-server JSON/in-process request futures, `thread_start_task`, and the expanded `thread/epiphany/promote` arm are boxed to reduce dispatcher stack pressure
|
|
- full `cargo test -p codex-app-server --lib` now passes 232/232 with no shell `RUST_MIN_STACK` set
|
|
|
|
## Open Questions
|
|
|
|
- How much more can move into `epiphany-core` without sacrificing the typed Codex host seam that makes the integration first-class?
|
|
- How should verified observations propose map/churn edits without turning promotion into automatic graph fanfic?
|
|
|
|
Do not promote anything from here into the map unless it survives verification or repeated reuse without contradiction.
|