experiment: name: EpiphanyAgent status: scaffolded phase: manual_external_typed_state premise: > Test whether separating map, scratch, and output reduces architectural drift for coding tasks when using a strong general model. objective: primary: > Build Epiphany as an opinionated fork of Codex that forces models to keep explicit typed models of the codebase, active goal, evidence, and change machinery they are working on. immediate: - Finish the initial workspace scaffold. - Keep controller actions and write rules explicit. - Ground the Epiphany fork architecture against the real Codex codebase. - Use the first minimal runner or manual operating loop on a small real task. - Try the protocol on a small real task. constraints: - No frontier-scale training budget. - Start with external state and prompting. - Keep the protocol light enough to use during normal work. - Prefer evidence over vibes. invariants: - The map is canonical and slow-changing. - Scratch is temporary and allowed to be wrong. - Output must not silently redefine the map. - Accepted changes should have recorded evidence. - Rejected paths stay recorded. - Algorithmic-map natural language and metaphors must be grounded in freshly read source context and concrete code references. data_flow: observe: - task request - repository state - verifier results process: - update map only with stable understanding - think about one subgoal at a time - propose one bounded change - verify against task goals and invariants - record evidence - commit or backtrack active_subgoals: - Keep the extracted Phase 4 persistent-semantic follow-up aligned with reality in both code and repo memory. - Keep vendored Codex as a thin typed host seam while Epiphany-owned prompt and retrieval/indexing logic live in the repo-owned `epiphany-core` crate. - Preserve exact/path lookup and BM25 fallback while the Qdrant path stays explicit and bounded. - Build the next layer above the landed typed distill/promote/update paths: richer observation-to-map proposal machinery while keeping verifier-backed acceptance/rejection explicit. accepted_design: - external typed state files - bounded controller actions - verifier-guided acceptance - manual first, automation later - patch Codex via collaboration-mode presets before touching protocol enums - make Phase 1 preset selection, labeling, and instruction lookup preset-name-aware where needed because Epiphany initially rides on ModeKind::Default - for a GUI-centered experiment, build a dedicated wrapper around customized vendored Codex instead of adopting third-party OpenCodex forks - treat Epiphany as core-owned structured thread state in Codex, with the GUI as a reflection and steering layer rather than the source of truth - represent the Epiphany machine map as two linked graphs (architecture and dataflow) with rich natural-language node fields such as purpose, mechanism, and metaphor, plus line-anchored code references - make the graph model compaction-aware: preserve a durable spine, checkpoint and reselect a frontier, and allow scratch to be rewritten or discarded - for mapping work, distinguish a completed pass from actual understanding: require representative concrete transformation questions, record unresolved mechanism gaps durably, and carry those gates across compaction - treat Epiphany as a supervising harness over specialist agents sharing a typed state plane, with private role-scoped scratch by default and explicit ownership for map, evidence, and code-change domains - add a repo-local hybrid retrieval subsystem to Epiphany: combine exact path/symbol/lexical search with semantic chunk retrieval and graph-linked lookup, shard it by repo/subsystem, and track freshness from diffs - treat Codex's existing file watcher, subagent lifecycle, memory pipeline, and diff stream as reusable substrate, but add the missing semantic layer above them: typed repo understanding, freshness invalidation, structured tool-output distillation, and shared state across specialist agents - insert a structured observation layer between raw tool output and durable Epiphany state so evidence, invalidation, and graph updates are promoted from typed distillation rather than transcript osmosis - design Epiphany as GUI-first rather than transcript-first: the primary artifact is typed thread state, chat is one client of that state, user steering should go through typed updates, and long-running retrieval/remap/verification work should surface as explicit observable jobs - treat compaction as a role-specific state transition with safe points, explicit checkpoints, and resume packets; workers may compact aggressively between implementation passes, while mapper, verifier, and coordinator preserve different state at different boundaries - make the compact-rehydrate-continue cycle an explicit Epiphany architecture primitive: compact only from a checkpoint when possible, rehydrate from canonical map/handoff/evidence, restate the persisted next action, then continue with one bounded move - preserve handoff context in repo-local AGENTS and notes files - keep a durable repository-level algorithmic map of vendored Codex so GUI and Epiphany changes hook into the existing app-server/core boundary instead of reimplementing the turn loop - keep a dedicated Epiphany algorithmic map that documents the currently landed fork architecture separately from the full Codex machine map and the future-facing harness spec - keep the Codex algorithmic map honest about auxiliary side channels and limitations: file watching, unified diffs, compaction, subagent messaging, and memory summarization are real substrate, but they do not by themselves amount to semantic repo understanding - vendor Codex directly into the parent repo instead of keeping it as a gitlink/submodule so Epiphany harness changes can be reviewed and committed atomically with the surrounding project state - for Phase 2, treat the presence of SessionState.epiphany_state as the internal activation signal and inject a bounded Epiphany summary into the developer-context path during build_initial_context rather than inventing a public operator surface early - Phase 2 is landed: a dedicated EpiphanyStateInstructions fragment now renders bounded Epiphany thread state into the developer prompt path during build_initial_context, with omission/inclusion/resume coverage and a stable snapshot - Phase 3 read-surface slice is landed: app-server protocol Thread payloads now carry optional typed epiphany_state hydrated from live CodexThread state when available and otherwise reconstructed from rollout with rollback/compaction semantics; dedicated Epiphany update RPCs and live notifications remain deferred - Epiphany should be an opinionated software development agent: pre-compaction and pre-handoff persistence of map, evidence, and next action are part of the workflow, not optional operator folklore - compaction handling should be pressure-aware rather than reactive: when context pressure rises, Epiphany should narrow the active move and persist checkpoints before forced compaction actually hits - repo-local AGENTS instructions should carry the session bootstrap and re-entry protocol explicitly, including running epiphany_state.py status on fresh load and after compaction - implement the first Phase 4 retrieval slice as a query-time hybrid retriever: reuse existing codex_file_search substrate for exact/path lookup and use workspace-local BM25 chunk search as the first semantic backend so the retrieval seam is proven before adding persistence - for the first persistent semantic retrieval store after the verified BM25 baseline, prefer a real vector database such as Qdrant over JSON/blob/postgres-thrash designs; treat Qdrant as the preferred persistent semantic backend rather than a hard requirement, keep exact/path lookup as a first-class organ, and keep BM25 available as a bootstrap/fallback/control path when Qdrant is unavailable or not configured instead of turning retrieval into vector-only soup - land the first persistent-semantic follow-up as an explicit indexing/update path rather than a hidden side effect in retrieval: keep `thread/epiphany/retrieve` read-only, add additive `thread/epiphany/index`, persist Qdrant index metadata under `codex_home`, use local Ollama embeddings by default, and fall back to BM25 whenever the persistent backend is stale, missing, or unavailable - extract Epiphany-owned implementation into a sibling repo-owned crate where possible, leaving vendored Codex with thin host adapters plus the typed protocol/thread/app-server seams that make the integration first-class - keep the licensing path compatible with future commercial use: Project-Authored Material outside `vendor/codex/**` is public under PolyForm Noncommercial 1.0.0 plus separate commercial terms, and outside contributions require a CLA granting broad sublicensing/relicensing rights - treat Epiphany as a fork of Codex with an opinionated modeling architecture, not merely a Codex collaboration mode or preset - use natural language and metaphor in algorithmic maps only after source-grounding the relevant mechanism; the metaphor should compress verified control flow, not decorate a guess - treat meticulous workflow discipline as part of the product, not as slowdown: rehydrate before acting, source-ground maps before prose, keep each slice bounded, verify the real seam, then persist map/evidence/handoff before moving on - live app-server smoke verified the explicit Qdrant indexing path against a loaded ephemeral thread rooted at `epiphany-core`; env/default Qdrant/Ollama backend config is good enough for the current slice, so defer a first-class config surface until it hurts for a real reason - land Epiphany's first explicit typed state-update surface as `thread/epiphany/update`: loaded-thread-only, revision-checked when requested, append observations/evidence, replace bounded map/scratch/churn fields, persist an immediate `RolloutItem::EpiphanyState`, and keep retrieval read-only - land Epiphany's first typed distillation/proposal surface as `thread/epiphany/distill`: loaded-thread-only, read-only, deterministic observation/evidence patch generation with a concrete `expectedRevision`, and explicit promotion through `thread/epiphany/update` - land Epiphany's first explicit promotion gate as `thread/epiphany/promote`: loaded-thread-only, verifier-backed, rejects failed candidates without mutation, and applies accepted candidates through the existing durable update path - land the first richer map/churn promotion safety layer: state replacement promotions must include explicit observations and patch evidence, and subgoal/invariant/graph/frontier/checkpoint/churn replacements get structural validation in `epiphany-core` before reaching the durable update path - keep Windows app-server verification explicit about stack pressure: Cargo config sets `RUST_MIN_STACK=67108864`, and large JSON/in-process request futures are boxed so the test binary does not depend on shell folklore or tiny default test-thread stacks rejected_paths: - name: pinned_map_only reason: still one token stream with no typed write semantics - name: train_new_base_model_first reason: too expensive for a first falsifiable experiment - name: exact_name_open_codex_cli_as_gui_surface reason: ymichael/open-codex is terminal-focused and has no Web UI or desktop surface - name: read_opencodex_web_readme_claude_models_as_current_gui_support reason: the Web UI has stale/aspirational Claude/Gemini model definitions, but the new-session selector only exposes Codex and the serve backend rejects non-Codex agents - name: imdigitalashish_opencodex_as_primary_gui_surface reason: the repo is a Tauri wrapper around an archived/vendored opencode CLI, has mismatched GUI/API model IDs versus the CLI model registry, runs the non-interactive tool path with broad auto-approval, and commits generated/binary/local-state artifacts - name: third_party_opencodex_forks_as_gui_base reason: audited candidates were either terminal-only, stale/aspirational, or incoherent wrappers; the comparison clones were removed from vendor to avoid building on untrusted drift current_status: summary: Workspace initialized with protocol files, typed state, a tiny CLI, a git repo, fresh-workspace handoff notes, architecture rationale, a Codex algorithmic map note, an Epiphany core-harness surface spec, and a rolling implementation plan against vendored Codex. Phase 1 durable Epiphany state, Phase 2 prompt integration, a minimal Phase 3 typed client read surface, and the Phase 4 retrieval/indexing/core-extraction work are landed on `main` at `80c29e0`. The current working tree adds the first explicit typed state-update surface, first read-only typed distillation/proposal surface, first explicit verifier-backed promotion gate, and first richer map/churn promotion safety layer: experimental loaded-thread-only `thread/epiphany/update`, `thread/epiphany/distill`, and `thread/epiphany/promote`; typed patch application through `CodexThread.epiphany_update_state(...)`; deterministic observation/evidence proposal generation; promotion policy evaluation in `epiphany-core`; explicit observation/evidence requirements for state replacement promotions; structural validation for subgoal, invariant, graph, frontier, checkpoint, and churn replacements; immediate rollout persistence only on accepted promotion/update; and replay support for pre-turn out-of-band Epiphany snapshots. Live app-server stdio smokes verified update, distill-to-update, and promote policy end-to-end before the richer structural layer: failed verifier evidence is rejected without state mutation, accepted verifier evidence promotes through the durable update path, and thread/read returns revision 1 with the promoted observation/evidence. The current baseline includes explicit `thread/epiphany/index`, manifest-backed Qdrant semantic persistence under `codex_home`, local Ollama embeddings by default, stale/fresh retrieval metadata, and read-only retrieval that still preserves exact/path search plus BM25 fallback instead of turning the machine into vector-only soup. A live app-server smoke on 2026-04-24 verified the explicit indexing path against real local Qdrant/Ollama using an ephemeral loaded thread rooted at `epiphany-core`: 5 files indexed into 198 chunks, a Qdrant collection was created, and persistent semantic retrieval returned `src/prompt.rs` chunks from the fresh index. Env-scoped backend config is good enough for now. The heavy Epiphany-owned prompt renderer, rollout replay helper, retrieval/indexing engine, observation distiller, and promotion evaluator live in the repo-owned `epiphany-core` crate, while vendored Codex carries thin adapters and the typed host seam. The repo also now has an operative source-available/commercial licensing notice and a CLA gate for external contributions. next_action: Build the next bounded layer above `thread/epiphany/distill`, `thread/epiphany/promote`, and `thread/epiphany/update`: richer observation-to-map proposal machinery from verified observations, while keeping retrieval and distillation read-only and avoiding watcher, GUI, or specialist-agent machinery.