Epiphany/state/map.yaml

experiment:
  name: EpiphanyAgent
  status: scaffolded
  phase: manual_external_typed_state
  premise: >
    Test whether separating map, scratch, and output reduces architectural drift
    for coding tasks when using a strong general model.

objective:
  primary: >
    Build Epiphany as an opinionated fork of Codex that forces models to keep
    explicit typed models of the codebase, active goal, evidence, and change
    machinery they are working on.
  immediate:
    - Finish the initial workspace scaffold.
    - Keep controller actions and write rules explicit.
    - Ground the Epiphany fork architecture against the real Codex codebase.
    - Use the first minimal runner or manual operating loop on a small real task.
    - Try the protocol on a small real task.

constraints:
  - No frontier-scale training budget.
  - Start with external state and prompting.
  - Keep the protocol light enough to use during normal work.
  - Prefer evidence over vibes.

invariants:
  - The map is canonical and slow-changing.
  - Scratch is temporary and allowed to be wrong.
  - Output must not silently redefine the map.
  - Accepted changes should have recorded evidence.
  - Rejected paths stay recorded.
  - Algorithmic-map natural language and metaphors must be grounded in freshly read source context and concrete code references.

data_flow:
  observe:
    - task request
    - repository state
    - verifier results
  process:
    - update map only with stable understanding
    - think about one subgoal at a time
    - propose one bounded change
    - verify against task goals and invariants
    - record evidence
    - commit or backtrack

active_subgoals:
  - Keep the Phase 5 semantic distillation/promotion/proposal work aligned with reality in code and repo memory.
  - Keep vendored Codex as a thin typed host seam while Epiphany-owned prompt and retrieval/indexing logic live in the repo-owned `epiphany-core` crate.
  - Preserve exact/path lookup and BM25 fallback while the Qdrant path stays explicit and bounded.
  - Build the next layer above the landed typed distill/promote/update paths: richer observation-to-map proposal machinery while keeping verifier-backed acceptance/rejection explicit.

accepted_design:
  - external typed state files
  - bounded controller actions
  - verifier-guided acceptance
  - manual first, automation later
  - patch Codex via collaboration-mode presets before touching protocol enums
  - make Phase 1 preset selection, labeling, and instruction lookup preset-name-aware where needed because Epiphany initially rides on ModeKind::Default
  - for a GUI-centered experiment, build a dedicated wrapper around customized vendored Codex instead of adopting third-party OpenCodex forks
  - treat Epiphany as core-owned structured thread state in Codex, with the GUI as a reflection and steering layer rather than the source of truth
  - represent the Epiphany machine map as two linked graphs (architecture and dataflow) with rich natural-language node fields such as purpose, mechanism, and metaphor, plus line-anchored code references
  - make the graph model compaction-aware: preserve a durable spine, checkpoint and reselect a frontier, and allow scratch to be rewritten or discarded
  - for mapping work, distinguish a completed pass from actual understanding: require representative concrete transformation questions, record unresolved mechanism gaps durably, and carry those gates across compaction
  - treat Epiphany as a supervising harness over specialist agents sharing a typed state plane, with private role-scoped scratch by default and explicit ownership for map, evidence, and code-change domains
  - add a repo-local hybrid retrieval subsystem to Epiphany: combine exact path/symbol/lexical search with semantic chunk retrieval and graph-linked lookup, shard it by repo/subsystem, and track freshness from diffs
  - treat Codex's existing file watcher, subagent lifecycle, memory pipeline, and diff stream as reusable substrate, but add the missing semantic layer above them: typed repo understanding, freshness invalidation, structured tool-output distillation, and shared state across specialist agents
  - insert a structured observation layer between raw tool output and durable Epiphany state so evidence, invalidation, and graph updates are promoted from typed distillation rather than transcript osmosis
  - design Epiphany as GUI-first rather than transcript-first: the primary artifact is typed thread state, chat is one client of that state, user steering should go through typed updates, and long-running retrieval/remap/verification work should surface as explicit observable jobs
  - treat compaction as a role-specific state transition with safe points, explicit checkpoints, and resume packets; workers may compact aggressively between implementation passes, while mapper, verifier, and coordinator preserve different state at different boundaries
  - make the compact-rehydrate-continue cycle an explicit Epiphany architecture primitive: compact only from a checkpoint when possible, rehydrate from canonical map/handoff/evidence, restate the persisted next action, then continue with one bounded move
  - preserve handoff context in repo-local AGENTS and notes files
  - keep a durable repository-level algorithmic map of vendored Codex so GUI and Epiphany changes hook into the existing app-server/core boundary instead of reimplementing the turn loop
  - keep a dedicated Epiphany algorithmic map that documents the currently landed fork architecture separately from the full Codex machine map and the future-facing harness spec
  - keep the Codex algorithmic map honest about auxiliary side channels and limitations: file watching, unified diffs, compaction, subagent messaging, and memory summarization are real substrate, but they do not by themselves amount to semantic repo understanding
  - vendor Codex directly into the parent repo instead of keeping it as a gitlink/submodule so Epiphany harness changes can be reviewed and committed atomically with the surrounding project state
  - for Phase 2, treat the presence of SessionState.epiphany_state as the internal activation signal and inject a bounded Epiphany summary into the developer-context path during build_initial_context rather than inventing a public operator surface early
  - Phase 2 is landed: a dedicated EpiphanyStateInstructions fragment now renders bounded Epiphany thread state into the developer prompt path during build_initial_context, with omission/inclusion/resume coverage and a stable snapshot
  - Phase 3 read-surface slice is landed: app-server protocol Thread payloads now carry optional typed epiphany_state hydrated from live CodexThread state when available and otherwise reconstructed from rollout with rollback/compaction semantics; dedicated Epiphany update RPCs and live notifications remain deferred
  - Epiphany should be an opinionated software development agent: pre-compaction and pre-handoff persistence of map, evidence, and next action are part of the workflow, not optional operator folklore
  - compaction handling should be pressure-aware rather than reactive: when context pressure rises, Epiphany should narrow the active move and persist checkpoints before forced compaction actually hits
  - repo-local AGENTS instructions should carry the session bootstrap and re-entry protocol explicitly, including running epiphany_state.py status on fresh load and after compaction
  - implement the first Phase 4 retrieval slice as a query-time hybrid retriever: reuse existing codex_file_search substrate for exact/path lookup and use workspace-local BM25 chunk search as the first semantic backend so the retrieval seam is proven before adding persistence
  - for the first persistent semantic retrieval store after the verified BM25 baseline, prefer a real vector database such as Qdrant over JSON/blob/postgres-thrash designs; treat Qdrant as the preferred persistent semantic backend rather than a hard requirement, keep exact/path lookup as a first-class organ, and keep BM25 available as a bootstrap/fallback/control path when Qdrant is unavailable or not configured instead of turning retrieval into vector-only soup
  - land the first persistent-semantic follow-up as an explicit indexing/update path rather than a hidden side effect in retrieval: keep `thread/epiphany/retrieve` read-only, add additive `thread/epiphany/index`, persist Qdrant index metadata under `codex_home`, use local Ollama embeddings by default, and fall back to BM25 whenever the persistent backend is stale, missing, or unavailable
  - extract Epiphany-owned implementation into a sibling repo-owned crate where possible, leaving vendored Codex with thin host adapters plus the typed protocol/thread/app-server seams that make the integration first-class
  - keep the licensing path compatible with future commercial use: Project-Authored Material outside `vendor/codex/**` is public under PolyForm Noncommercial 1.0.0 plus separate commercial terms, and outside contributions require a CLA granting broad sublicensing/relicensing rights
  - treat Epiphany as a fork of Codex with an opinionated modeling architecture, not merely a Codex collaboration mode or preset
  - use natural language and metaphor in algorithmic maps only after source-grounding the relevant mechanism; the metaphor should compress verified control flow, not decorate a guess
  - treat meticulous workflow discipline as part of the product, not as slowdown: rehydrate before acting, source-ground maps before prose, keep each slice bounded, verify the real seam, then persist map/evidence/handoff before moving on
  - enforce immediate anti-cruft reverts for regression and benchmark work: if a fix attempt does not fix the regression or move the benchmark, revert it before trying the next hypothesis and record the rejected path when useful
  - render the anti-Jenga regression/benchmark discipline inside the Epiphany developer-state prompt so hydrated Epiphany turns carry the operational rule at runtime instead of relying on AGENTS prose alone
  - live app-server smoke verified the explicit Qdrant indexing path against a loaded ephemeral thread rooted at `epiphany-core`; env/default Qdrant/Ollama backend config is good enough for the current slice, so defer a first-class config surface until it hurts for a real reason
  - land Epiphany's first explicit typed state-update surface as `thread/epiphany/update`: loaded-thread-only, revision-checked when requested, append observations/evidence, replace bounded map/scratch/churn fields, persist an immediate `RolloutItem::EpiphanyState`, and keep retrieval read-only
  - land Epiphany's first typed distillation/proposal surface as `thread/epiphany/distill`: loaded-thread-only, read-only, deterministic observation/evidence patch generation with a concrete `expectedRevision`, and explicit promotion through `thread/epiphany/update`
  - land Epiphany's first explicit promotion gate as `thread/epiphany/promote`: loaded-thread-only, verifier-backed, rejects failed candidates without mutation, and applies accepted candidates through the existing durable update path
  - land the first richer map/churn promotion safety layer: state replacement promotions must include explicit observations and patch evidence, and subgoal/invariant/graph/frontier/checkpoint/churn replacements get structural validation in `epiphany-core` before reaching the durable update path
  - keep Windows app-server verification explicit about stack pressure: Cargo config sets `RUST_MIN_STACK=67108864` so the test binary does not depend on shell folklore or tiny default test-thread stacks

rejected_paths:
  - name: pinned_map_only
    reason: still one token stream with no typed write semantics
  - name: train_new_base_model_first
    reason: too expensive for a first falsifiable experiment
  - name: exact_name_open_codex_cli_as_gui_surface
    reason: ymichael/open-codex is terminal-focused and has no Web UI or desktop surface
  - name: read_opencodex_web_readme_claude_models_as_current_gui_support
    reason: the Web UI has stale/aspirational Claude/Gemini model definitions, but the new-session selector only exposes Codex and the serve backend rejects non-Codex agents
  - name: imdigitalashish_opencodex_as_primary_gui_surface
    reason: the repo is a Tauri wrapper around an archived/vendored opencode CLI, has mismatched GUI/API model IDs versus the CLI model registry, runs the non-interactive tool path with broad auto-approval, and commits generated/binary/local-state artifacts
  - name: third_party_opencodex_forks_as_gui_base
    reason: audited candidates were either terminal-only, stale/aspirational, or incoherent wrappers; the comparison clones were removed from vendor to avoid building on untrusted drift

current_status:
  summary: Workspace initialized with protocol files, typed state, handoff notes, architecture maps, and a rolling implementation plan against vendored Codex. Phase 1 durable state, Phase 2 prompt integration, the minimal Phase 3 typed read surface, and Phase 4 retrieval/indexing/core-extraction are landed on `main`. Phase 5 semantic distillation and promotion is now active: the current baseline includes explicit Qdrant indexing with BM25 fallback, typed `thread/epiphany/update`, read-only `thread/epiphany/distill`, verifier-backed `thread/epiphany/promote`, structural map/churn promotion validation in `epiphany-core`, and thin vendored Codex host seams. The Windows app-server stack-pressure regression was narrowed to test-thread stack size; repo-local Cargo config sets `RUST_MIN_STACK=67108864`, and boxed-future request-dispatch experiments were removed after config-only verification passed. The repo also has source-available/commercial licensing plus a CLA gate for external contributions.
  next_action: Continue Phase 5 by building the next bounded layer above `thread/epiphany/distill`, `thread/epiphany/promote`, and `thread/epiphany/update`: richer observation-to-map/churn proposal machinery from verified observations, while keeping retrieval and distillation read-only and avoiding watcher, GUI, or specialist-agent machinery.