experiment: name: EpiphanyAgent status: scaffolded phase: manual_external_typed_state premise: > Test whether separating map, scratch, and output reduces architectural drift for coding tasks when using a strong general model. objective: primary: > Build a model-agnostic protocol that Codex can use in this workspace and that can later be evaluated on open models without requiring frontier-scale training. immediate: - Finish the initial workspace scaffold. - Keep controller actions and write rules explicit. - Ground the Codex epiphany-mode idea against the real open-source codebase. - Use the first minimal runner or manual operating loop on a small real task. - Try the protocol on a small real task. constraints: - No frontier-scale training budget. - Start with external state and prompting. - Keep the protocol light enough to use during normal work. - Prefer evidence over vibes. invariants: - The map is canonical and slow-changing. - Scratch is temporary and allowed to be wrong. - Output must not silently redefine the map. - Accepted changes should have recorded evidence. - Rejected paths stay recorded. data_flow: observe: - task request - repository state - verifier results process: - update map only with stable understanding - think about one subgoal at a time - propose one bounded change - verify against task goals and invariants - record evidence - commit or backtrack active_subgoals: - Clean up and ship the verified Phase 4 repo-local hybrid retrieval slice without widening scope. - Keep the canonical map, handoff note, evidence log, and checked-in stable schema state aligned with the verified landing zone. - Preserve BM25 as the no-hassle fallback while treating Qdrant as the preferred later persistent semantic backend. accepted_design: - external typed state files - bounded controller actions - verifier-guided acceptance - manual first, automation later - patch Codex via collaboration-mode presets before touching protocol enums - make Phase 1 preset selection, labeling, and instruction lookup preset-name-aware where needed because Epiphany initially rides on ModeKind::Default - for a GUI-centered experiment, build a dedicated wrapper around customized vendored Codex instead of adopting third-party OpenCodex forks - treat Epiphany as core-owned structured thread state in Codex, with the GUI as a reflection and steering layer rather than the source of truth - represent the Epiphany machine map as two linked graphs (architecture and dataflow) with rich natural-language node fields such as purpose, mechanism, and metaphor, plus line-anchored code references - make the graph model compaction-aware: preserve a durable spine, checkpoint and reselect a frontier, and allow scratch to be rewritten or discarded - for mapping work, distinguish a completed pass from actual understanding: require representative concrete transformation questions, record unresolved mechanism gaps durably, and carry those gates across compaction - treat Epiphany as a supervising harness over specialist agents sharing a typed state plane, with private role-scoped scratch by default and explicit ownership for map, evidence, and code-change domains - add a repo-local hybrid retrieval subsystem to Epiphany: combine exact path/symbol/lexical search with semantic chunk retrieval and graph-linked lookup, shard it by repo/subsystem, and track freshness from diffs - treat Codex's existing file watcher, subagent lifecycle, memory pipeline, and diff stream as reusable substrate, but add the missing semantic layer above them: typed repo understanding, freshness invalidation, structured tool-output distillation, and shared state across specialist agents - insert a structured observation layer between raw tool output and durable Epiphany state so evidence, invalidation, and graph updates are promoted from typed distillation rather than transcript osmosis - design Epiphany as GUI-first rather than transcript-first: the primary artifact is typed thread state, chat is one client of that state, user steering should go through typed updates, and long-running retrieval/remap/verification work should surface as explicit observable jobs - treat compaction as a role-specific state transition with safe points, explicit checkpoints, and resume packets; workers may compact aggressively between implementation passes, while mapper, verifier, and coordinator preserve different state at different boundaries - preserve handoff context in repo-local AGENTS and notes files - keep a durable repository-level algorithmic map of vendored Codex so GUI and Epiphany changes hook into the existing app-server/core boundary instead of reimplementing the turn loop - keep a dedicated Epiphany delta algorithmic map that documents the currently landed divergence from plain Codex separately from the full Codex machine map and the future-facing harness spec - keep the Codex algorithmic map honest about auxiliary side channels and limitations: file watching, unified diffs, compaction, subagent messaging, and memory summarization are real substrate, but they do not by themselves amount to semantic repo understanding - vendor Codex directly into the parent repo instead of keeping it as a gitlink/submodule so Epiphany harness changes can be reviewed and committed atomically with the surrounding project state - for Phase 2, treat the presence of SessionState.epiphany_state as the internal activation signal and inject a bounded Epiphany summary into the developer-context path during build_initial_context rather than inventing a public operator surface early - Phase 2 is landed: a dedicated EpiphanyStateInstructions fragment now renders bounded Epiphany thread state into the developer prompt path during build_initial_context, with omission/inclusion/resume coverage and a stable snapshot - Phase 3 read-surface slice is landed: app-server protocol Thread payloads now carry optional typed epiphany_state hydrated from live CodexThread state when available and otherwise reconstructed from rollout with rollback/compaction semantics; dedicated Epiphany update RPCs and live notifications remain deferred - Epiphany should be an opinionated software development agent: pre-compaction and pre-handoff persistence of map, evidence, and next action are part of the workflow, not optional operator folklore - compaction handling should be pressure-aware rather than reactive: when context pressure rises, Epiphany should narrow the active move and persist checkpoints before forced compaction actually hits - repo-local AGENTS instructions should carry the session bootstrap and re-entry protocol explicitly, including running epiphany_state.py status on fresh load and after compaction - implement the first Phase 4 retrieval slice as a query-time hybrid retriever: reuse existing codex_file_search substrate for exact/path lookup and use workspace-local BM25 chunk search as the first semantic backend so the retrieval seam is proven before adding persistence - for the first persistent semantic retrieval store after the verified BM25 baseline, prefer a real vector database such as Qdrant over JSON/blob/postgres-thrash designs; treat Qdrant as the preferred persistent semantic backend rather than a hard requirement, keep exact/path lookup as a first-class organ, and keep BM25 available as a bootstrap/fallback/control path when Qdrant is unavailable or not configured instead of turning retrieval into vector-only soup rejected_paths: - name: pinned_map_only reason: still one token stream with no typed write semantics - name: train_new_base_model_first reason: too expensive for a first falsifiable experiment - name: exact_name_open_codex_cli_as_gui_surface reason: ymichael/open-codex is terminal-focused and has no Web UI or desktop surface - name: read_opencodex_web_readme_claude_models_as_current_gui_support reason: the Web UI has stale/aspirational Claude/Gemini model definitions, but the new-session selector only exposes Codex and the serve backend rejects non-Codex agents - name: imdigitalashish_opencodex_as_primary_gui_surface reason: the repo is a Tauri wrapper around an archived/vendored opencode CLI, has mismatched GUI/API model IDs versus the CLI model registry, runs the non-interactive tool path with broad auto-approval, and commits generated/binary/local-state artifacts - name: third_party_opencodex_forks_as_gui_base reason: audited candidates were either terminal-only, stale/aspirational, or incoherent wrappers; the comparison clones were removed from vendor to avoid building on untrusted drift current_status: summary: Workspace initialized with protocol files, typed state, a tiny CLI, a git repo, fresh-workspace handoff notes, architecture rationale, a Codex algorithmic map note, an Epiphany core-harness surface spec, and a rolling implementation plan against vendored Codex. Phase 1 durable Epiphany state, Phase 2 prompt integration, and a minimal Phase 3 typed client read surface are all implemented and verified with a real Rust/MSVC toolchain; vendored Codex is tracked directly in the parent repo; app-server Thread payloads can now carry typed Epiphany state without scraping prompt text; pre-compaction persistence is now an explicit part of the intended Epiphany workflow; and the first bounded Phase 4 repo-local hybrid retrieval slice is now wired and verified in the working tree across protocol, core, app-server protocol, and app-server, including stable schema fixture regeneration and the Windows `CARGO_TARGET_DIR` verification path. next_action: Clean up and push the current verified query-time hybrid retriever as the Phase 4 slice 1 landing zone. Do not bolt durable EpiphanyState writes onto `thread/epiphany/retrieve` without a clean out-of-band rollout semantic; after the push and reboot, the next persistent semantic retrieval slice should target a Qdrant-backed store with explicit freshness/indexing semantics as the preferred semantic backend while BM25 remains the fallback when Qdrant is unavailable.