Initial EpiphanyAgent scaffold and planning docs

.gitignore

@@ -0,0 +1,2 @@
+__pycache__/
+*.pyc

.gitmodules

@@ -0,0 +1,3 @@
+[submodule "vendor/codex"]
+	path = vendor/codex
+	url = https://github.com/openai/codex.git

AGENTS.md

@@ -0,0 +1,61 @@
+# EpiphanyAgent Instructions
+
+## Project Purpose
+
+This repo explores an Epiphany mode for AI coding agents: external typed state, explicit mental maps, bounded scratch work, verifier evidence, and aggressive anti-churn discipline.
+
+The motivating failure mode is that an agent can make many plausible local edits after global coherence has already failed. The project goal is to test whether explicit map/scratch/evidence channels reduce that drift.
+
+## Canonical State
+
+- Treat `state/map.yaml` as the canonical project map.
+- Treat `state/scratch.md` as disposable working memory for one bounded subgoal.
+- Treat `state/evidence.jsonl` as the durable log of what was learned, verified, rejected, or accepted.
+- Treat `notes/codex-epiphany-mode-plan.md` as the current patch plan for adding Epiphany mode to the vendored Codex repo.
+- Update `state/map.yaml` when project understanding changes.
+- Append evidence after meaningful research, implementation, verification, or rejected paths.
+
+## Current Recommendation
+
+Do not start with full protocol enum surgery.
+
+Start by implementing a lightweight preset-backed Epiphany mode in the vendored Codex repo:
+
+- add an `epiphany.md` collaboration-mode template
+- add an `Epiphany` collaboration preset backed by `ModeKind::Default`
+- patch the TUI to display the active preset name so the mode is visible to users
+- only add `ModeKind::Epiphany` later if the preset-backed experiment proves useful
+
+## Important Paths
+
+- Project root: `E:\Projects\EpiphanyAgent`
+- Vendored Codex repo: `E:\Projects\EpiphanyAgent\vendor\codex`
+- Patch plan: `E:\Projects\EpiphanyAgent\notes\codex-epiphany-mode-plan.md`
+- Handoff summary: `E:\Projects\EpiphanyAgent\notes\fresh-workspace-handoff.md`
+- State CLI: `E:\Projects\EpiphanyAgent\tools\epiphany_state.py`
+
+## Useful Commands
+
+Use the bundled Python runtime if `python` is not on PATH:
+
+```powershell
+& 'C:\Users\Meta\.cache\codex-runtimes\codex-primary-runtime\dependencies\python\python.exe' '.\tools\epiphany_state.py' status
+& 'C:\Users\Meta\.cache\codex-runtimes\codex-primary-runtime\dependencies\python\python.exe' '.\tools\epiphany_state.py' add-evidence --type research --status ok --note '...'
+```
+
+Useful Codex repo searches:
+
+```powershell
+rg -n "pub enum ModeKind|TUI_VISIBLE_COLLABORATION_MODES" .\vendor\codex\codex-rs\protocol\src\config_types.rs
+rg -n "builtin_collaboration_mode_presets|fn plan_preset|fn default_preset" .\vendor\codex\codex-rs\models-manager\src\collaboration_mode_presets.rs
+rg -n "collaboration_mode_label|collaboration_mode_indicator|set_collaboration_mask" .\vendor\codex\codex-rs\tui\src\chatwidget.rs
+```
+
+## Operating Discipline
+
+- Before substantial edits, restate the current mechanism and intended change.
+- Prefer one clear hypothesis per iteration.
+- Verify with checks that reflect the real goal, not just proxy success.
+- Revert or discard changes that do not clearly improve the target.
+- If the diff grows while understanding shrinks, stop implementation and switch to diagnosis.
+- Keep maps and prose together; do not replace useful maps with prose-only explanations.

README.md

@@ -0,0 +1,50 @@
+# EpiphanyAgent
+
+A cheap prototype for testing whether typed external state makes a strong model less likely to build an impressive pile of locally plausible nonsense.
+
+## Premise
+
+Most coding agents are one token stream pretending to be a planner, scratchpad, architect, and release engineer at the same time. This workspace separates those roles outside the model first.
+
+The goal is not to prove a new frontier architecture in a weekend. The goal is to find out whether explicit state channels and bounded controller actions reduce drift enough to matter.
+
+## Core Artifacts
+
+- `AGENTS.md`: project-specific instructions for future Codex sessions
+- `state/map.yaml`: canonical, slow-changing model of the task or system
+- `state/scratch.md`: temporary reasoning for one subgoal
+- `state/branches.json`: branch hypotheses and branch outcomes
+- `state/evidence.jsonl`: verifier outputs, acceptance reasons, and reversions
+- `protocol/controller-actions.md`: allowed actions and write rules
+- `notes/fresh-workspace-handoff.md`: concise summary for a fresh workspace
+- `notes/codex-epiphany-mode-plan.md`: concrete patch plan for vendored Codex
+- `notes/architecture-rationale.md`: why the map/scratch/evidence architecture exists
+- `tools/epiphany_state.py`: tiny CLI for inspecting and updating branch/evidence state
+- `eval-plan.md`: how to compare this protocol against plain prompting
+
+## Basic Loop
+
+1. Observe the task, repo state, and current verifier outputs.
+2. Update `state/map.yaml` only when understanding stabilizes.
+3. Use `state/scratch.md` for one bounded subgoal at a time.
+4. Propose one change or one answer.
+5. Verify against task success, map consistency, and simplicity.
+6. Record evidence.
+7. Commit, compare branches, or backtrack.
+
+## Why This Exists
+
+Pinned notes inside one giant prompt are better than nothing, but they are still one token soup. This workspace treats map, scratch, and output as different artifacts with different write rules.
+
+That may still turn out to be elaborate prompt theater. Fine. At least it will be falsifiable prompt theater.
+
+## Tiny CLI
+
+Examples:
+
+```powershell
+python .\tools\epiphany_state.py status
+python .\tools\epiphany_state.py add-evidence --type verify --status ok --note "Map and patch still agree"
+python .\tools\epiphany_state.py add-branch --id b1 --hypothesis "Smaller verifier beats elaborate verifier"
+python .\tools\epiphany_state.py close-branch --id b1 --status rejected --note "More ceremony, no better signal"
+```

eval-plan.md

@@ -0,0 +1,37 @@
+# Eval Plan
+
+The first question is not whether this protocol is elegant. The first question is whether it helps.
+
+## Conditions
+
+1. Plain prompting with no explicit map
+2. Plain prompting with a pinned architecture map in context
+3. External typed state with no verifier stack
+4. External typed state with verifier checks
+
+## Candidate Metrics
+
+- task success
+- regression rate after follow-up edits
+- total diff size
+- revert rate
+- contradiction rate between map and final patch
+- branch kill rate
+- human rating of architectural coherence
+
+## Good First Tasks
+
+- fix a bug without regressing adjacent behavior
+- refactor a medium file while preserving invariants
+- add one feature that touches multiple connected modules
+
+## Failure Smells
+
+- the map and the code disagree
+- scratch grows without being summarized or deleted
+- branches accumulate but are never killed
+- tests pass while architectural coherence gets worse
+
+## Exit Criteria For The Prototype
+
+The prototype is worth keeping if it reliably reduces drift or makes failure easier to diagnose without adding absurd overhead.

notes/architecture-rationale.md

@@ -0,0 +1,134 @@
+# Architecture Rationale
+
+## Core Epiphany
+
+The central failure mode is local plausibility without global coherence.
+
+An agent can keep making reasonable-looking edits, pass narrow tests, improve proxy metrics, and still lose the actual shape of the system. Once the mental model collapses, adding more local fixes often makes the tower worse.
+
+The response is to make global coherence a first-class object:
+
+- keep a persistent map
+- isolate scratch reasoning
+- treat public output and code edits as commits, not thoughts
+- verify against the map and the real goal
+- kill dead churn instead of preserving maybe-useful machinery
+
+## Proposed Model Shape
+
+The speculative model architecture is a typed-state controller around a language backbone:
+
+```mermaid
+flowchart TD
+    U["User / Environment"] --> P["Prompt + Observations Encoder"]
+    P --> C["Controller Policy"]
+    C --> M["Persistent Map Store"]
+    C --> T["Scratch Thought Workspace"]
+    C --> O["Output Stream"]
+    C --> S["Search Module"]
+    C --> B["Sparse Backbone LLM"]
+    M --> B
+    T --> B
+    P --> B
+    S --> B
+    B --> D["Drafts / Plans / Patches / Critiques"]
+    D --> V["Verifier Stack"]
+    V --> C
+    D --> S
+    D --> M
+    D --> O
+    O --> U
+```
+
+The important split:
+
+- `map`: slow-changing canonical system model, invariants, decisions, data flow, rejected paths
+- `scratch`: volatile local reasoning for one bounded subgoal
+- `output`: user-visible replies, tool calls, and code edits
+- `controller`: learned or prompted policy deciding when to model, think, verify, edit, backtrack, or speak
+- `verifier`: tests plus map coherence, simplicity, and real-objective checks
+- `search`: bounded branching and backtracking for hard decomposition choices
+
+## Why A Pinned Map Is Not Enough
+
+Pinning a map inside the context window is useful, but it is still one token soup:
+
+- it competes with everything else for attention
+- it has no typed write semantics
+- it can drift silently
+- it does not distinguish stable knowledge from temporary hypotheses
+
+The cheap prototype uses external files to fake typed state before any model architecture exists.
+
+## Cheap Prototype Strategy
+
+Do not train a new backbone first. That is where money goes to die.
+
+Instead:
+
+1. Externalize state in files.
+2. Use a controller protocol with a small action space.
+3. Make the model choose one action at a time.
+4. Record evidence after meaningful steps.
+5. Compare against plain prompting.
+
+Current state files:
+
+- `state/map.yaml`
+- `state/scratch.md`
+- `state/branches.json`
+- `state/evidence.jsonl`
+
+Current controller actions:
+
+- `update_map`
+- `think_subgoal`
+- `propose_patch`
+- `run_verify`
+- `compare_branches`
+- `backtrack`
+- `speak`
+
+## Evaluation Shape
+
+Compare four conditions:
+
+1. plain prompting
+2. pinned map in normal context
+3. external typed state without verifier discipline
+4. external typed state with verifier discipline
+
+Measure:
+
+- task success
+- regression rate after follow-up edits
+- total diff size
+- revert rate
+- contradiction rate between map and patch
+- branch kill rate
+- human rating of architectural coherence
+
+The best informal metric: after five iterations, does the system still make sense?
+
+## DeepSeek Research Signals
+
+These sources informed the architecture direction. The repo does not depend on them directly, but they are useful background.
+
+- DeepSeek-R1 suggests RL can elicit reflection and self-verification, but also shows raw emergent reasoning can become repetitive or sloppy: https://github.com/deepseek-ai/DeepSeek-R1
+- Generalist Reward Modeling suggests critique and principle generation can be scaled at inference time: https://arxiv.org/abs/2504.02495
+- DeepSeek-Prover-V1.5 uses intrinsic-reward tree search, supporting the idea that search helps branching reasoning tasks: https://github.com/deepseek-ai/DeepSeek-Prover-V1.5
+- DeepSeek-Prover-V2 leans on subgoal decomposition, supporting explicit decomposition rather than pure continuation: https://arxiv.org/abs/2504.21801
+- DeepSeekMath-V2 emphasizes verification quality, not just final-answer correctness: https://github.com/deepseek-ai/DeepSeek-Math-V2
+- DeepSeek-V2/V3 show the usefulness of sparse MoE and efficient attention for cheaper large-scale reasoning: https://arxiv.org/abs/2405.04434 and https://arxiv.org/abs/2412.19437
+- Engram argues for explicit conditional memory and lookup as a separate axis of sparsity, which matches the need for map-like persistent state: https://arxiv.org/abs/2601.07372
+
+## Codex-Specific Interpretation
+
+Codex cannot expose a new internal model architecture from this repo. The practical path is harness-level behavior:
+
+- add an Epiphany collaboration mode or preset
+- inject developer instructions that enforce external typed state
+- optionally add a tiny state helper tool
+- evaluate whether that improves behavior before touching protocol enums
+
+This is not a new brain. It is a better harness around the existing one.

notes/codex-epiphany-mode-plan.md

@@ -0,0 +1,173 @@
+# Codex Epiphany Phase 1 Implementation Plan
+
+This is the actionable implementation note for the first Epiphany patch slice against the real `openai/codex` tree vendored under `vendor/codex`.
+
+Phase 1 is deliberately **internal/dev-usable**, not user-facing. The goal is to give Codex a real durable Epiphany state seam in core, keep the app usable, and prove replay/persistence before touching prompts, presets, protocol notifications, or GUI reflection.
+
+## Summary
+
+Land Phase 1 as an **additive core slice** that persists and rehydrates structured Epiphany thread state without changing normal Codex behavior.
+
+After this slice:
+
+- non-Epiphany sessions still behave exactly as they do now
+- the app remains usable because existing clients ignore the new metadata
+- Epiphany state can be exercised and validated through automated core/rollout/resume tests
+
+This slice does **not** include:
+
+- prompt integration
+- app-server notifications
+- GUI surfaces
+- collaboration-mode presets
+- mutation gates
+- retrieval indexing
+- specialist-agent orchestration
+
+## Key Changes
+
+### 1. Add minimal durable Epiphany protocol types
+
+Touch:
+
+- `vendor/codex/codex-rs/protocol/src/protocol.rs`
+
+Add:
+
+- `EpiphanyThreadState`
+- `EpiphanyStateItem { turn_id: Option<String>, state: EpiphanyThreadState }`
+- `RolloutItem::EpiphanyState(EpiphanyStateItem)`
+
+Phase 1 schema should stay minimal but sufficient:
+
+- `revision`
+- `objective`
+- `active_subgoal_id`
+- `subgoals`
+- `invariants`
+- `graphs`
+- `graph_frontier`
+- `graph_checkpoint`
+- `scratch`
+- `observations`
+- `recent_evidence`
+- `churn`
+- `mode`
+- `last_updated_turn_id`
+
+Defer these to later phases:
+
+- `agents`
+- `jobs`
+- `retrieval`
+- separate append-only evidence/checkpoint rollout items
+- separate turn intent rollout item
+- any new `EventMsg` variants
+
+All new protocol types should derive the same serialization/schema traits as neighboring rollout types.
+
+### 2. Store Epiphany state in `SessionState`
+
+Touch:
+
+- `vendor/codex/codex-rs/core/src/state/session.rs`
+- `vendor/codex/codex-rs/core/src/session/mod.rs`
+
+Add:
+
+- `epiphany_state: Option<EpiphanyThreadState>` to `SessionState`
+- simple getter/setter helpers on session state
+
+Persistence rule:
+
+- persist one `RolloutItem::EpiphanyState(...)` per real user turn
+- write it immediately after the `TurnContextItem` baseline is persisted
+- skip the write when `epiphany_state` is `None`
+
+Do not change prompt assembly or tool behavior in this slice.
+
+### 3. Rehydrate Epiphany state on resume and fork
+
+Touch:
+
+- `vendor/codex/codex-rs/core/src/session/rollout_reconstruction.rs`
+- the session-start/resume path that consumes `RolloutReconstruction`
+
+Extend reconstruction to carry:
+
+- `epiphany_state: Option<EpiphanyThreadState>`
+
+Replay rule:
+
+- the newest surviving `EpiphanyStateItem` wins
+- rollback must discard Epiphany snapshots from rolled-back turns
+- compaction must not erase the latest surviving snapshot
+
+Treat `EpiphanyStateItem` as turn-scoped metadata in the same reverse scan that already respects rollback and compaction boundaries.
+
+### 4. Update rollout readers to tolerate the new variant
+
+Adding a `RolloutItem` variant will create exhaustiveness fallout across the repo. Update those sites explicitly instead of letting it turn into whack-a-mole.
+
+At minimum touch:
+
+- `vendor/codex/codex-rs/app-server-protocol/src/protocol/thread_history.rs`
+- `vendor/codex/codex-rs/rollout/src/policy.rs`
+- `vendor/codex/codex-rs/rollout/src/metadata.rs`
+- `vendor/codex/codex-rs/rollout/src/list.rs`
+- `vendor/codex/codex-rs/rollout/src/recorder.rs`
+- any additional exhaustive `RolloutItem` matches in `core`, `tui`, `exec`, and `protocol`
+
+Phase 1 behavior at those call sites:
+
+- thread-history/UI builders should ignore `EpiphanyState`
+- rollout metadata/listing code should treat it as non-rendered metadata
+- existing user-visible thread rendering must stay unchanged
+
+## Test Plan
+
+### Protocol and schema
+
+- serde round-trip for `EpiphanyThreadState`
+- serde round-trip for `EpiphanyStateItem`
+- any protocol/schema snapshots updated for the new `RolloutItem` variant
+
+### Core persistence
+
+- a session with populated Epiphany state writes `RolloutItem::EpiphanyState`
+- a session with `None` Epiphany state writes nothing extra
+- the Epiphany rollout item appears after the turn-context baseline for the turn
+
+### Replay correctness
+
+- resume restores the newest surviving Epiphany snapshot
+- rollback drops snapshots from discarded turns
+- compaction keeps the latest surviving Epiphany snapshot available after reconstruction
+
+### Compatibility
+
+- `app-server-protocol` thread reconstruction still builds the same visible turn history
+- rollout list/metadata readers still work with mixed old/new rollout files
+- untouched user flows remain unchanged in tests
+
+### Compile-fallout coverage
+
+- build/test the touched crates so all exhaustive `RolloutItem` matches are updated in one pass
+
+## Assumptions
+
+- Phase 1 is **internal/dev-usable**, not operator-usable or GUI-visible.
+- "Testable after Phase 1" means persistence/replay behavior is verifiable through automated tests and internal session wiring, not that users can activate Epiphany from the UI yet.
+- Phase 1 keeps evidence and checkpoint data embedded inside the baseline Epiphany snapshot instead of splitting them into separate rollout item variants.
+- Prompt integration, app-server typed updates, GUI scene loading, retrieval, jobs, and specialist-agent scheduling are later slices.
+
+## Immediate Next Step After This Plan
+
+Implement the protocol/core/replay slice only:
+
+1. add the minimal protocol types
+2. wire `SessionState`
+3. persist one snapshot per real user turn
+4. restore it through rollout reconstruction
+5. patch exhaustive `RolloutItem` matches
+6. add replay/persistence tests before moving on

notes/codex-repository-algorithmic-map.md

@@ -0,0 +1,490 @@
+# Codex Repository Algorithmic Map
+
+This is the working map for how vendored Codex moves data. It exists because the repo is large, the crate boundaries are real, and "just keep it in your head" is how you end up chewing drywall.
+
+Scope: this is the main request/response loop for the open-source Codex repository in `E:\Projects\EpiphanyAgent\vendor\codex`. It focuses on the steady-state path from user input to model prompt, tool execution, UI events, and persistence. It is not a full inventory of every crate.
+
+## Mental model in one sentence
+
+Codex is a threaded agent engine in `core` wrapped by transport layers (`app-server`, `protocol`) and user interfaces (`tui`, CLI entrypoints), with durable execution history written to rollouts and indexed into SQLite metadata.
+
+## Main actors
+
+| Actor | Job | Important crates |
+| --- | --- | --- |
+| CLI/TUI front door | Parse user intent, bootstrap config, start or connect to the engine, render events | `cli`, `tui` |
+| App server | Translate UI/CLI requests into core thread operations and stream results back over protocol transport | `app-server`, `protocol` |
+| Thread manager | Own shared managers and the registry of active threads | `core` |
+| Thread/session engine | Hold mutable thread state, build prompts, run turns, stream model events, persist history | `core` |
+| Model client | Talk to the model provider, maintain turn-scoped streaming state, reuse websocket state inside a turn | `core`, `backend-client`, provider crates |
+| Tool runtime | Expose model-visible tools, validate requests, execute shell/MCP/function calls, return tool outputs | `core`, `mcp-server`, `exec`, `exec-server` |
+| Watchers and live side channels | Watch files, emit coarse change notifications, surface turn diffs and other realtime status | `core`, `app-server`, `app-server-protocol` |
+| Memory and summarization | Extract rollout memories, maintain summaries, compact long history for future prompts | `core`, `rollout-trace`, `state` |
+| Persistence layer | Write durable thread rollouts, mirror metadata into SQLite, support listing/resume/fork/archive | `rollout`, `thread-store`, `state` |
+
+## Top-level flow
+
+```mermaid
+flowchart TD
+    A["User action in CLI or TUI"] --> B["CLI/TUI bootstrap"]
+    B --> C["App-server request or embedded app-server"]
+    C --> D["ThreadManager routes to CodexThread"]
+    D --> E["Session builds TurnContext"]
+    E --> F["Prompt assembly"]
+    F --> G["ModelClientSession streams ResponseEvents"]
+    G --> H["Assistant items"]
+    G --> I["Tool calls"]
+    I --> J["ToolRouter + ToolCallRuntime"]
+    J --> K["Tool outputs recorded as input items"]
+    K --> F
+    H --> L["Turn items + protocol events"]
+    L --> M["Rollout JSONL + SQLite metadata"]
+    L --> N["UI redraw / app-server notifications"]
+    L --> O["Compaction / summaries / replacement history"]
+    L --> P["Unified diff + watcher side channels"]
+```
+
+## Startup dataflow
+
+### 1. CLI chooses a surface
+
+- `cli/src/main.rs` is the top-level command parser.
+- Interactive usage usually lands in the TUI path.
+- Non-TUI commands branch into things like exec, review, sandbox, app server, or MCP server.
+
+The important bit: the CLI mostly decides which surface to launch. It is not the agent brain.
+
+### 2. TUI bootstraps the environment
+
+`tui/src/lib.rs` does the heavy bootstrap work:
+
+- load config
+- resolve auth constraints
+- set up logging and environment management
+- decide whether to launch an embedded app server or connect to a remote one
+- start the TUI client against that app-server target
+
+The TUI is a client shell around the engine, not the engine itself.
+
+### 3. App server becomes the transport seam
+
+`app-server` owns the JSON-RPC-ish transport loop and client/session coordination.
+
+Its job is to:
+
+- accept UI/client requests
+- resolve config overrides and trust checks
+- create or resume threads through `ThreadManager`
+- subscribe to thread events
+- forward thread snapshots, deltas, approvals, and status changes back to the client
+
+This is the cleanest seam for a custom GUI wrapper. A new GUI should talk to the app server, not reimplement the core turn loop.
+
+## Core runtime model
+
+### ThreadManager: the factory and registry
+
+`core/src/thread_manager.rs` owns the long-lived shared machinery:
+
+- models manager
+- skills manager
+- plugins manager
+- MCP connection manager
+- environment manager
+- active thread registry
+
+It creates new threads, resumes threads from rollouts, and forks existing threads by reconstructing history from persisted rollout items.
+
+### CodexThread: the per-thread facade
+
+`core/src/codex_thread.rs` is the stable handle the rest of the system uses for a live thread.
+
+It delegates thread operations into the underlying session:
+
+- submit input
+- steer input
+- fetch events
+- call MCP tools
+- flush rollout
+- expose config snapshots
+
+Think of it as the thread-shaped shell around a session.
+
+### Session: the actual mutable brain case
+
+`core/src/session/session.rs` is where the real per-thread state lives.
+
+A session owns:
+
+- current configuration and overrides
+- conversation/thread history
+- active turn state
+- event channels to the UI/app server
+- services bundle for auth, models, skills, plugins, MCP, analytics, and state DB
+- rollout recorder and trace recorder
+
+This is the core locus of mutation. If a thread changes, it changes here.
+
+## Turn dataflow: one full loop
+
+This is the central algorithmic loop.
+
+### 1. Input enters the thread
+
+Typical path:
+
+1. user message or command enters the TUI
+2. TUI sends an app-server request
+3. `CodexMessageProcessor` turns that into a core op
+4. `CodexThread.submit(...)` or `submit_with_trace(...)` forwards it into the session
+
+For a custom GUI, this is the same path you want. The GUI should produce thread operations, not hand-build prompts.
+
+### 2. Session creates a turn
+
+The session turns the incoming op into a concrete turn with:
+
+- sub-id / trace metadata
+- collaboration mode
+- approval and sandbox policy
+- cwd and environment
+- current date / timezone
+- developer instructions
+- user instructions
+- personality
+- feature flags
+- dynamic tools
+- model info and reasoning settings
+
+This bundle becomes `TurnContext` in `core/src/session/turn_context.rs`.
+
+That matters because the prompt is not built from raw user text alone. It is built from user text plus a large amount of execution policy and mode state.
+
+### 3. Context and history are recorded before sampling
+
+Before the model is asked anything, Codex records:
+
+- context updates
+- user input
+- initial turn items
+- skill/plugin injections
+- relevant history mutations
+
+This keeps the transcript, rollout, and UI state aligned even when a turn later branches into tools, retries, or cancellation.
+
+### 4. Tools are resolved for this turn
+
+`built_tools(...)` in `core/src/session/turn.rs` builds the model-visible tool surface for the turn.
+
+It gathers:
+
+- built-in tools
+- MCP tools from the connection manager
+- connector/tool-search surfaces when enabled
+- dynamic tools
+- permission-gated capabilities
+
+This becomes a `ToolRouter`, which is both:
+
+- the registry used at execution time
+- the model-visible tool schema exposed in the prompt
+
+### 5. Prompt assembly happens
+
+Prompt construction is easiest to see in `core/src/prompt_debug.rs` and `core/src/session/turn.rs`.
+
+The prompt is assembled from:
+
+- formatted input/history window
+- base instructions
+- collaboration mode developer instructions
+- optional personality
+- tool definitions
+- parallel-tool-call capability
+- optional output schema
+
+Useful frame: Codex does not have one static system prompt. It synthesizes a turn-scoped prompt from history plus runtime policy.
+
+### 6. The model stream begins
+
+`ModelClientSession` is the turn-scoped provider client.
+
+It:
+
+- opens or reuses provider transport for the turn
+- may prewarm websocket state
+- streams `ResponseEvent`s back into the session
+- retains turn-scoped routing state such as sticky turn tokens
+
+The stream is the heartbeat of the turn. Everything after this is driven by response items arriving incrementally.
+
+### 7. Stream events split into two classes
+
+As `ResponseEvent`s arrive, Codex handles them in `core/src/session/turn.rs` and `core/src/stream_events_utils.rs`.
+
+#### Class A: non-tool output
+
+These become visible assistant artifacts:
+
+- assistant messages
+- reasoning items
+- web search markers
+- image-generation items
+
+They are parsed into turn items, emitted to the UI, and recorded into history/rollout.
+
+#### Class B: tool calls
+
+If a completed response item is actually a tool request, `ToolRouter::build_tool_call(...)` converts it into an executable invocation.
+
+Then Codex:
+
+1. records the tool-call item immediately
+2. queues execution through `ToolCallRuntime`
+3. waits for the tool result
+4. turns the tool result into `ResponseInputItem`
+5. records that output back into history
+6. loops back into another sampling request if follow-up is needed
+
+This is the critical feedback loop. The model does not directly mutate the world. It emits tool requests, gets structured outputs, and continues sampling with those outputs now in-context.
+
+### 8. The turn repeats until follow-up stops
+
+Sampling can loop multiple times inside one user-visible turn:
+
+- model emits tool calls
+- tools execute
+- outputs are recorded
+- model samples again
+
+The turn ends when the model reaches a completed state with no more required tool follow-up.
+
+### 9. Final outputs settle into three places
+
+At the end of a turn, the result exists in three overlapping representations:
+
+- live turn items for UI rendering
+- conversation history for future prompts
+- rollout/state persistence for resume, listing, fork, and analysis
+
+That triplicate shape matters. Codex is always balancing live UX, future context, and durable replay.
+
+## Tool execution dataflow
+
+Tool execution is not one subsystem. It is a routing stack.
+
+### ToolRouter decides what the model asked for
+
+`core/src/tools/router.rs` converts completed model response items into tool invocations:
+
+- shell commands
+- function tools
+- MCP tools
+- custom tools
+- tool-search calls
+
+It also knows which tools are model-visible and whether they can run in parallel.
+
+### ToolCallRuntime actually runs the call
+
+`ToolCallRuntime` is the execution harness:
+
+- dispatch built-in tools
+- call MCP servers
+- run shell commands with approval/sandbox policy
+- handle cancellation
+- return a structured result payload
+
+Then the session records that payload as part of the same turn's transcript.
+
+Short version: the model proposes, the router interprets, the runtime executes, and the session turns the result back into prompt material.
+
+### Important caveat: tool outputs are still mostly transcript-shaped
+
+This is one of the important current limitations.
+
+Codex does have structured tool payload types. But much of the runtime still metabolizes tool execution back into response items and transcript/history material rather than promoting the result into a stronger shared state layer.
+
+In practice that means:
+
+- useful tool output often becomes more prompt material
+- later turns may need to rediscover or resummarize facts from history
+- there is only a weak membrane between "raw command exhaust" and "durable understanding"
+
+That is part of why Epiphany needs an explicit observation/distillation layer instead of trusting transcript osmosis.
+
+## Auxiliary side channels that matter
+
+The main turn loop is the heart of the machine, but Codex also has a few auxiliary channels that shape behavior around it.
+
+### File watching exists, but mostly as coarse notification substrate
+
+Codex already has a generic file-watcher layer and app-server fs-watch support.
+
+Today that mostly supports things like:
+
+- skills change notifications
+- client-visible file watch events
+- coarse "something changed here" awareness
+
+What it does **not** natively do is maintain a typed repo-understanding model or semantically invalidate graph claims, invariants, or code-intel shards when the code moves.
+
+So the watcher substrate is real, but its understanding layer is thin.
+
+### Turn diffs exist, but they are textual
+
+Codex also tracks per-turn file changes and can emit an aggregated unified diff snapshot to clients.
+
+That is useful for UI display and review, but it is still mostly text-level impact reporting.
+
+It does not, by itself, answer:
+
+- which mapped subsystem is now stale
+- which mechanism answer was invalidated
+- which graph node or invariant should be reopened
+
+So the diff channel is a useful sensor, not yet a semantic diagnosis layer.
+
+### Subagents exist, but coordination is mostly message/config based
+
+Codex already supports spawning subagents, sending them messages, waiting on them, and closing them.
+
+That is real multi-agent execution substrate.
+
+But the dominant collaboration model is still:
+
+- inherit parent config and tools
+- exchange prompts/messages
+- report completion/status back up the tree
+
+There is not yet a strong shared typed state plane for map truth, evidence authority, retrieval freshness, or invalidation ownership. The agents can cooperate, but mostly by talking through the same old language pipe.
+
+### Startup memory exists, but it is the wrong kind of memory for repo understanding
+
+Codex has a memories pipeline that extracts and consolidates useful summaries from rollouts.
+
+That helps with:
+
+- user/task continuity
+- startup summarization
+- durable memory-like guidance
+
+But it is not the same thing as a canonical repo-understanding store, and it is not a substitute for a graph/evidence/invalidation plane shared by specialists.
+
+So the machine already has memory organs, just not the ones we actually need for Epiphany.
+
+## Persistence model
+
+Codex has two persistence stories layered together.
+
+### 1. Rollout files are the durable thread log
+
+The rollout layer writes thread history as append-oriented session/thread artifacts under `codex_home`.
+
+These rollouts are the source used for:
+
+- resume
+- fork
+- archive
+- summary reconstruction
+- durable event history
+
+`ThreadManager` can rebuild a thread from rollout history, which is why rollout shape matters more than any one in-memory UI state.
+
+### 1.5. Compaction is part of the real dataflow
+
+Long-running threads do not just grow forever. Codex compacts history and preserves a replacement history that future prompts may carry forward.
+
+That matters because compaction is not some decorative cleanup pass. It changes what later turns see as their effective past.
+
+Today that compaction path is still mostly transcript- and summary-shaped. It preserves continuity, but not a first-class typed model of repo understanding.
+
+That is why our current external map/evidence files feel useful: they are compensating for a real hole in the native harness.
+
+### 2. SQLite mirrors metadata for indexing and control
+
+The `state` crate mirrors rollout-derived metadata into SQLite.
+
+That database supports things like:
+
+- thread metadata
+- backfill/extraction state
+- memory-mode pollution flags
+- job orchestration
+- fast listing/filtering without replaying every rollout file
+
+So the shape is:
+
+- rollout = durable narrative log
+- SQLite = queryable control/index layer
+
+One uncomfortable but useful truth:
+
+- rollout and SQLite are durable
+- transcript and metadata replay are real
+- but Codex still lacks a canonical typed repo-knowledge layer sitting above them
+
+So persistence is strong for history, weaker for understanding.
+
+### 3. Thread store abstracts local vs remote persistence
+
+`thread-store` defines storage-neutral thread persistence interfaces.
+
+That abstraction exists so callers can treat a `ThreadId` as the durable identity while storage backends decide whether the thread lives in local rollout files, remote storage, or something else.
+
+## UI and engine boundary
+
+This is the most important architectural boundary for EpiphanyAgent.
+
+The TUI is not where prompt assembly or tool execution fundamentally lives.
+
+Those live in `core`, with `app-server` acting as the transport/control seam.
+
+So a dedicated GUI wrapper should probably:
+
+- talk to app-server protocol
+- consume thread snapshots and turn events
+- send thread operations and settings updates
+- leave prompt building, tool execution, persistence, and model streaming inside Codex core
+
+That keeps the wrapper thin and avoids rebuilding the dangerous part twice.
+
+## Where Epiphany mode naturally hooks in
+
+Epiphany should hook in at the turn-construction and prompt-assembly layer, not by inventing a parallel execution engine.
+
+Natural hook points:
+
+1. collaboration preset resolution in the models/config path
+2. mode/preset visibility in the UI layer
+3. developer-instruction injection before prompt assembly
+4. possibly app-server/client settings so the chosen preset survives non-TUI flows
+
+The important thing is that Epiphany changes how turns are instructed and surfaced, not how the core agent loop fundamentally works.
+
+It also needs to patch the blind spots around that loop:
+
+- code-intelligence and hybrid retrieval
+- semantic invalidation from repo changes
+- structured distillation from tool output into durable facts
+- shared knowledge across specialists without forcing every role to reread the whole transcript
+
+## Anti-confusion summary
+
+If the repo starts feeling slippery again, remember this:
+
+- CLI chooses the surface.
+- TUI is a client shell.
+- App server is the transport seam.
+- ThreadManager owns shared managers and live threads.
+- Session owns mutable thread state.
+- TurnContext captures per-turn policy and environment.
+- Prompt assembly happens per turn.
+- Model streaming produces response items.
+- Response items either become UI/history artifacts or executable tool calls.
+- Tool outputs are fed back into the same turn.
+- File watchers, unified diffs, compaction, and memory summarization are real side channels around the main loop.
+- Those side channels are useful substrate, but they are not yet a semantic repo-understanding layer.
+- Rollouts persist the narrative; SQLite indexes the metadata.
+
+That is the machine. Everything else is detail, decoration, or damage.

notes/fresh-workspace-handoff.md

@@ -0,0 +1,91 @@
+# Fresh Workspace Handoff
+
+## What This Repo Is
+
+`EpiphanyAgent` is a prototype workspace for testing whether AI coding agents behave better when their mental model is externalized into typed state:
+
+- `map`: canonical project/system understanding
+- `scratch`: temporary local reasoning
+- `evidence`: verification, acceptance, and rejection trail
+
+The project grew from a concrete failure mode: an agent can keep making plausible local changes while the global design quietly collapses. The intended fix is not "more prompting" in the abstract, but explicit state channels and a controller discipline around when to model, think, verify, edit, revert, and speak.
+
+## Current State
+
+The workspace has:
+
+- a root git repo
+- a vendored clone of `openai/codex` at `vendor/codex`
+- a small state CLI at `tools/epiphany_state.py`
+- typed state files under `state/`
+- controller action rules in `protocol/controller-actions.md`
+- an eval sketch in `eval-plan.md`
+- architecture rationale in `notes/architecture-rationale.md`
+- a Codex patch plan in `notes/codex-epiphany-mode-plan.md`
+
+The latest canonical status is in `state/map.yaml`.
+
+## Key Conclusion From Codex Repo Research
+
+The best insertion point for Epiphany mode is Codex's existing collaboration-mode preset system.
+
+Relevant files:
+
+- `vendor/codex/codex-rs/protocol/src/config_types.rs`
+- `vendor/codex/codex-rs/models-manager/src/collaboration_mode_presets.rs`
+- `vendor/codex/codex-rs/collaboration-mode-templates/src/lib.rs`
+- `vendor/codex/codex-rs/core/src/context/collaboration_mode_instructions.rs`
+- `vendor/codex/codex-rs/app-server/src/codex_message_processor.rs`
+- `vendor/codex/codex-rs/tui/src/collaboration_modes.rs`
+- `vendor/codex/codex-rs/tui/src/chatwidget.rs`
+- `vendor/codex/codex-rs/tui/src/bottom_pane/footer.rs`
+
+## Recommended Next Implementation
+
+Build Phase 1, not the full enum-backed mode:
+
+1. Add `codex-rs/collaboration-mode-templates/templates/epiphany.md`.
+2. Export it as `EPIPHANY` from `codex-rs/collaboration-mode-templates/src/lib.rs`.
+3. Add an `Epiphany` preset in `codex-rs/models-manager/src/collaboration_mode_presets.rs`.
+4. Keep the preset backed by `ModeKind::Default` initially.
+5. Patch the TUI to display the preset name so Epiphany is visible even though it uses Default semantics.
+6. Test manually before adding a real `ModeKind::Epiphany`.
+
+Why:
+
+- avoids protocol/schema churn
+- avoids SDK regeneration churn
+- keeps tool gating predictable
+- tests the real behavioral hypothesis first
+
+## What Not To Do First
+
+Do not immediately add `ModeKind::Epiphany` unless the user explicitly asks for the full protocol-level mode. That path touches more of the stack: protocol types, analytics, UI indicators, tool gating, generated schemas, SDK types, and snapshots.
+
+Do not build a large state-management subsystem before testing whether the instruction/preset path helps. A tiny helper tool may be useful later, but the first value test is the mode discipline itself.
+
+## Prototype Philosophy
+
+The useful behavior is:
+
+- keep a canonical map for nontrivial work
+- use scratch for one bounded subgoal
+- explain the current mechanism before broad edits
+- prefer one hypothesis per iteration
+- verify against real goals and map coherence
+- revert churn that does not move the needle
+- stop coding when understanding degrades
+
+The mode should not pretend the model has native typed channels. It should use explicit external files and write rules until a real architecture exists.
+
+## Useful Verification
+
+From the repo root:
+
+```powershell
+& 'C:\Users\Meta\.cache\codex-runtimes\codex-primary-runtime\dependencies\python\python.exe' '.\tools\epiphany_state.py' status
+Get-Content -Tail 5 '.\state\evidence.jsonl'
+git status --short
+```
+
+The vendored Codex repo is its own clone under `vendor/codex`. Treat it as source material for now unless the user asks to patch it.

protocol/controller-actions.md

@@ -0,0 +1,119 @@
+# Controller Actions
+
+The controller chooses exactly one primary action per step. This is meant to reduce the classic failure mode where the model is "thinking", rewriting the plan, editing files, and rationalizing the edits all at once.
+
+## Actions
+
+### `update_map`
+
+Purpose:
+Write stable understanding into `state/map.yaml`.
+
+Allowed writes:
+- objectives
+- constraints
+- invariants
+- accepted design
+- rejected paths
+- current status
+
+Rules:
+- Do not copy raw scratch into the map.
+- Do not write guesses as settled architecture.
+- If the map changes, say why.
+
+### `think_subgoal`
+
+Purpose:
+Use `state/scratch.md` for one bounded problem only.
+
+Allowed writes:
+- temporary decomposition
+- open questions
+- candidate checks
+- local reasoning that may be wrong
+
+Rules:
+- Scope it to one subgoal.
+- Prefer deletion or summarization over accumulation.
+- Scratch is disposable by design.
+
+### `propose_patch`
+
+Purpose:
+Make one bounded code or document change tied to a clear hypothesis.
+
+Allowed writes:
+- workspace files
+- `state/branches.json`
+
+Rules:
+- Name the hypothesis.
+- Keep the patch narrow enough to verify.
+- Do not silently redefine the map through code.
+
+### `run_verify`
+
+Purpose:
+Check whether the proposed change actually improved the thing we care about.
+
+Allowed writes:
+- `state/evidence.jsonl`
+- `state/branches.json`
+
+Checks may include:
+- tests
+- lints
+- benchmark deltas
+- invariant checks
+- coherence review against `state/map.yaml`
+- simplicity review
+
+### `compare_branches`
+
+Purpose:
+Choose between competing changes or kill all of them.
+
+Allowed writes:
+- `state/branches.json`
+- `state/evidence.jsonl`
+
+Rules:
+- Prefer the smallest branch that clearly satisfies the objective.
+- If two branches both feel dubious, reject both.
+
+### `backtrack`
+
+Purpose:
+Abandon a bad line of attack instead of decorating it further.
+
+Allowed writes:
+- `state/branches.json`
+- `state/evidence.jsonl`
+- `state/map.yaml`
+
+Rules:
+- Record why the branch was rejected.
+- Preserve lessons that belong in the map.
+- Do not keep dead machinery out of sentiment.
+
+### `speak`
+
+Purpose:
+Produce user-visible output.
+
+Rules:
+- Speak after verification when possible.
+- If speaking before verification, be explicit that the result is provisional.
+
+## Promotion Rule
+
+Information moves from scratch to map only if it survives verification, comparison, or repeated re-use without contradiction.
+
+## Stop Conditions
+
+Stop implementation and switch back to diagnosis when:
+- the diff is growing and understanding is shrinking
+- tests are green but the map and code disagree
+- the agent cannot explain the data flow stage by stage
+- new abstraction is compensating for confusion instead of reducing it

state/branches.json

@@ -0,0 +1,15 @@
+{
+  "branches": [
+    {
+      "id": "main",
+      "hypothesis": "External typed state will reduce drift relative to plain prompting on iterative coding tasks.",
+      "status": "active",
+      "artifacts": [
+        "state/map.yaml",
+        "state/scratch.md",
+        "state/evidence.jsonl"
+      ],
+      "notes": "Initial scaffold plus tiny CLI. No evaluated task yet."
+    }
+  ]
+}

state/evidence.jsonl

@@ -0,0 +1,26 @@
+{"ts":"2026-04-23","type":"scaffold","status":"ok","note":"Initialized EpiphanyAgent workspace with typed state files, controller action rules, and an initial experiment map."}
+{"ts": "2026-04-23T03:42:27+00:00", "type": "bootstrap", "status": "ok", "note": "Initialized repo, added typed-state scaffold, and verified the tiny CLI runs with bundled Python."}
+{"ts": "2026-04-23T03:48:53+00:00", "type": "research", "status": "ok", "note": "Cloned openai/codex, identified collaboration-mode presets as the clean insertion point, and wrote a concrete patch plan note for preset-backed versus enum-backed Epiphany mode."}
+{"ts": "2026-04-23T15:28:33+00:00", "type": "handoff", "status": "ok", "note": "Added root AGENTS.md and notes/fresh-workspace-handoff.md so a fresh workspace can continue from repo-local context."}
+{"ts": "2026-04-23T15:29:25+00:00", "type": "handoff", "status": "ok", "note": "Added architecture-rationale.md with the typed-state architecture, cheap prototype strategy, eval shape, and DeepSeek research context."}
+{"ts": "2026-04-23T15:47:17+00:00", "type": "research", "status": "ok", "note": "Orientation confirmed the preset-backed Codex insertion point and recorded that Phase 1 TUI selection/labeling must compare preset identity, not only ModeKind, because Epiphany rides on Default."}
+{"ts": "2026-04-23T15:48:11+00:00", "type": "research", "status": "ok", "note": "App-server collaboration mode list preserves preset name but drops developer_instructions; turn-start fallback currently matches only ModeKind, so duplicate Default-backed presets need explicit preset identity or explicit instructions in non-TUI flows."}
+{"ts": "2026-04-23T15:55:06+00:00", "type": "research", "status": "ok", "note": "Cloned stellarlinkco/codex to vendor/opencodex and ymichael/open-codex to vendor/open-codex-cli. The exact-name open-codex fork is terminal-only; vendor/opencodex is the GUI-relevant Rust/Web UI fork served by codex serve."}
+{"ts": "2026-04-23T16:17:47+00:00", "type": "research", "status": "ok", "note": "Checked OpenCodex Anthropic/Claude GUI claims. web README and MODEL_OPTIONS mention claude/sonnet/opus, but AgentSelector only exposes codex and serve backend rejects any spawned agent other than codex; Anthropic support appears limited to model_provider config, not GUI agent selection."}
+{"ts": "2026-04-23T16:26:11+00:00", "type": "research", "status": "rejected", "note": "Audited imdigitalashish/OpenCodex: it is a Tauri desktop wrapper around vendored opencode, not a Codex fork; the core coding-agent path depends on an external opencode binary and non-interactive CLI mode."}
+{"ts": "2026-04-23T16:26:11+00:00", "type": "verification", "status": "partial", "note": "Frontend npm ci and npm run build succeeded for imdigitalashish/OpenCodex; npm audit reports 3 high-severity advisories. Go and Cargo toolchains are not on PATH, so opencode and Tauri backend were not compiled."}
+{"ts": "2026-04-23T16:26:11+00:00", "type": "research", "status": "rejected", "note": "imdigitalashish/OpenCodex GUI model IDs are inconsistent with vendored opencode SupportedModels: the GUI advertises/writes Anthropic API IDs like claude-sonnet-4-5-20250929 while opencode accepts symbolic IDs such as claude-4-sonnet."}
+{"ts": "2026-04-23T16:31:48+00:00", "type": "decision", "status": "ok", "note": "Removed third-party OpenCodex comparison clones from vendor/opencodex, vendor/open-codex-cli, and vendor/opencodex-imdigitalashish. Decision: build a dedicated GUI wrapper around customized vendored Codex instead of trusting unaudited forks."}
+{"ts": "2026-04-23T18:02:11+01:00", "type": "research", "status": "ok", "note": "Mapped vendored Codex end-to-end dataflow in notes/codex-repository-algorithmic-map.md: CLI/TUI choose a surface, app-server is the transport seam, core/session owns prompt assembly and tool execution, and rollouts plus SQLite provide durable thread history and metadata."}
+{"ts": "2026-04-23T18:16:47+01:00", "type": "design", "status": "ok", "note": "Defined Epiphany as core-owned structured thread state in notes/epiphany-core-harness-surfaces.md. The spec anchors on existing TurnContextItem and PlanUpdate/TurnDiff patterns: persist Epiphany baselines in rollouts, stream typed updates through app-server, and let the GUI render graphs, invariants, evidence, intent, and churn without becoming the source of truth."}
+{"ts": "2026-04-23T18:24:36+01:00", "type": "design", "status": "ok", "note": "Refined the Epiphany harness spec so the machine map is two linked graphs rather than one: an ArchitectureGraph and a DataflowGraph, joined by explicit cross-links and grounded by line-anchored code refs. Replaced the vague node description field with purpose, mechanism, and metaphor so the internal representation carries richer language for both model alignment and human inspection."}
+{"ts": "2026-04-23T18:36:52+01:00", "type": "design", "status": "ok", "note": "Updated the Epiphany harness spec to handle repeated compactions during large repo mapping. Added an explicit spine/frontier/scratch model, graph checkpoints as re-entry breadcrumbs, dataflow sharding for large repositories, coverage/provenance status on graph elements, and more resilient code refs with symbol and last-seen revision metadata so resume/compaction does not collapse the map back into prose mush."}
+{"ts": "2026-04-23T19:41:43+00:00", "type": "research", "status": "ok", "note": "Inspected Codex session 019db84b-7edd-7843-abc9-76ad2c402bb4 rollout. Failure mode was repeated premature closure on mapping work: assistant repeatedly framed local pass completion as done while user had to restate bar (missing leaves, control flow not file map, hand-waved mechanisms). Four context compactions preserved replacement_history but likely weakened the persistent evaluation bar. Quality improved only when user asked a concrete mechanism question (planetary gravity to visible nebula terrain), which forced an honest admission of a missing seam. Implication for Epiphany: mapping tasks need explicit done criteria based on answering concrete transformation questions, plus durable coverage/frontier/gap state across compactions."}
+{"ts": "2026-04-23T19:54:42+00:00", "type": "decision", "status": "ok", "note": "Patched epiphany-core-harness-surfaces.md to make mapping done-criteria explicit. Added durable MechanismQuestion and MechanismGap concepts, separate MappingCompletionState from coverage, preserved active questions/gaps through frontier and checkpoint, and required concrete transformation-question answerability before marking mapping work done. Updated state/map.yaml to carry the same rule into accepted design and next action."}
+{"ts": "2026-04-23T20:01:26+00:00", "type": "decision", "status": "ok", "note": "Documented a multi-agent Epiphany harness model in epiphany-core-harness-surfaces.md. Added supervising-harness plus specialist-agent framing, agent registry and role metadata, ownership boundaries for mapper/worker/verifier/coordinator, private-by-default role scratch, subscription-triggered wakeups, protocol/UI surfaces for agent status, and phased specialist-agent scheduling. Updated state/map.yaml to record the shared-state-plane plus role-ownership design and to defer the patch-slice plan until after this broader design pass."}
+{"ts": "2026-04-23T20:08:53+00:00", "type": "decision", "status": "ok", "note": "Extended epiphany-core-harness-surfaces.md with a first-class repository retrieval subsystem. Added hybrid retrieval principle (exact path/symbol/lexical plus semantic chunk and graph-linked lookup), repo-local sharded index summaries, diff-driven freshness/invalidation, role-scoped retrieval policies, retrieval lifecycle hooks, protocol/app-server update events, typed retrieval query surface guidance, UI retrieval view, and phased rollout positioning before GUI reflection and specialist-agent scheduling. Updated state/map.yaml to treat retrieval as a core design decision and to keep the next action on broader harness design before a Phase 1 patch-slice plan."}
+{"ts": "2026-04-23T20:13:55+00:00", "type": "research", "status": "ok", "note": "Scanned vendored Codex for additional harness blind spots before patch planning. Findings: repo retrieval is still fuzzy-file/CLI-centric; file watchers are wired mainly to skills cache invalidation and fs notifications, not repo-understanding freshness; subagents inherit config and tools but do not share a typed collaboration state plane; the startup memories pipeline explicitly excludes sub-agent sessions; most tool outputs still persist into history as text payloads; and turn diff tracking exposes aggregated unified diff, not impacted symbols/subsystems or semantic invalidation. Implication: Epiphany should reuse watcher/diff/subagent substrate but add typed distillation, freshness tracking, and shared repo knowledge above it."}
+{"ts": "2026-04-23T20:23:25+00:00", "type": "decision", "status": "ok", "note": "Patched notes/epiphany-core-harness-surfaces.md after the Codex blind-spot scan. The spec now explicitly reuses existing Codex substrate (watchers, diff stream, subagent lifecycle, memory pipeline) while naming the missing semantic layer above it: code-intelligence-aware hybrid retrieval, semantic invalidation over repo changes, a structured observation/distillation layer between raw tool output and durable state, and a shared knowledge plane for specialist agents that is not replacement_history or prompt osmosis. Synced state/map.yaml with the same observation-layer design."}
+{"ts": "2026-04-23T20:33:19+00:00", "type": "decision", "status": "ok", "note": "Revised notes/codex-repository-algorithmic-map.md so the Codex machine map reflects both the main turn loop and the surrounding side channels/limitations. Added watcher and memory actors, extended the top-level flow with compaction and watcher/diff side channels, documented that tool outputs are still mostly transcript-shaped, and added explicit sections for file watching, textual diff tracking, message/config-based subagent coordination, startup memory summarization, and compaction as real dataflow. Conclusion: Codex has useful substrate around the loop, but still lacks a canonical typed repo-understanding layer above it."}
+{"ts": "2026-04-23T20:38:04+00:00", "type": "decision", "status": "ok", "note": "Patched epiphany-core-harness-surfaces.md with explicit GUI-first implications. The spec now states that structured thread state, not the transcript, is the primary artifact; chat is one client of that state; GUI steering should use typed updates; resume must restore a working scene; and long-running retrieval/remap/verification/specialist work should surface as typed jobs with progress. Added EpiphanyJobState to the thread model plus EpiphanyJobsUpdated protocol/app-server events, and expanded UI responsibilities and phase notes accordingly. Synced state/map.yaml with the same GUI-first design correction."}
+{"ts": "2026-04-23T20:56:56+00:00", "type": "decision", "status": "ok", "note": "Replaced notes/codex-epiphany-mode-plan.md with the actionable Phase 1 implementation plan. The repo now records an internal/dev-usable first slice focused on minimal Epiphany protocol types, SessionState storage, one Epiphany rollout snapshot per real user turn, replay through rollout reconstruction, exhaustive RolloutItem compatibility updates, and persistence/replay tests before prompt integration, presets, protocol notifications, or GUI work. Synced state/map.yaml so the next action is implementation rather than more design passes."}

state/map.yaml

@@ -0,0 +1,90 @@
+experiment:
+  name: EpiphanyAgent
+  status: scaffolded
+  phase: manual_external_typed_state
+  premise: >
+    Test whether separating map, scratch, and output reduces architectural drift
+    for coding tasks when using a strong general model.
+
+objective:
+  primary: >
+    Build a model-agnostic protocol that Codex can use in this workspace and that
+    can later be evaluated on open models without requiring frontier-scale
+    training.
+  immediate:
+    - Finish the initial workspace scaffold.
+    - Keep controller actions and write rules explicit.
+    - Ground the Codex epiphany-mode idea against the real open-source codebase.
+    - Use the first minimal runner or manual operating loop on a small real task.
+    - Try the protocol on a small real task.
+
+constraints:
+  - No frontier-scale training budget.
+  - Start with external state and prompting.
+  - Keep the protocol light enough to use during normal work.
+  - Prefer evidence over vibes.
+
+invariants:
+  - The map is canonical and slow-changing.
+  - Scratch is temporary and allowed to be wrong.
+  - Output must not silently redefine the map.
+  - Accepted changes should have recorded evidence.
+  - Rejected paths stay recorded.
+
+data_flow:
+  observe:
+    - task request
+    - repository state
+    - verifier results
+  process:
+    - update map only with stable understanding
+    - think about one subgoal at a time
+    - propose one bounded change
+    - verify against task goals and invariants
+    - record evidence
+    - commit or backtrack
+
+active_subgoals:
+  - Implement the Phase 1 internal/dev-usable Epiphany core state slice in vendored Codex: protocol types, SessionState storage, rollout persistence, replay, and compatibility tests.
+  - Use the tiny CLI and typed state files on a small real task.
+  - Decide whether the next executable artifact should be a patch runner or a verifier helper.
+  - Run the protocol on a small real task.
+
+accepted_design:
+  - external typed state files
+  - bounded controller actions
+  - verifier-guided acceptance
+  - manual first, automation later
+  - patch Codex via collaboration-mode presets before touching protocol enums
+  - make Phase 1 preset selection, labeling, and instruction lookup preset-name-aware where needed because Epiphany initially rides on ModeKind::Default
+  - for a GUI-centered experiment, build a dedicated wrapper around customized vendored Codex instead of adopting third-party OpenCodex forks
+  - treat Epiphany as core-owned structured thread state in Codex, with the GUI as a reflection and steering layer rather than the source of truth
+  - represent the Epiphany machine map as two linked graphs (architecture and dataflow) with rich natural-language node fields such as purpose, mechanism, and metaphor, plus line-anchored code references
+  - make the graph model compaction-aware: preserve a durable spine, checkpoint and reselect a frontier, and allow scratch to be rewritten or discarded
+  - for mapping work, distinguish a completed pass from actual understanding: require representative concrete transformation questions, record unresolved mechanism gaps durably, and carry those gates across compaction
+  - treat Epiphany as a supervising harness over specialist agents sharing a typed state plane, with private role-scoped scratch by default and explicit ownership for map, evidence, and code-change domains
+  - add a repo-local hybrid retrieval subsystem to Epiphany: combine exact path/symbol/lexical search with semantic chunk retrieval and graph-linked lookup, shard it by repo/subsystem, and track freshness from diffs
+  - treat Codex's existing file watcher, subagent lifecycle, memory pipeline, and diff stream as reusable substrate, but add the missing semantic layer above them: typed repo understanding, freshness invalidation, structured tool-output distillation, and shared state across specialist agents
+  - insert a structured observation layer between raw tool output and durable Epiphany state so evidence, invalidation, and graph updates are promoted from typed distillation rather than transcript osmosis
+  - design Epiphany as GUI-first rather than transcript-first: the primary artifact is typed thread state, chat is one client of that state, user steering should go through typed updates, and long-running retrieval/remap/verification work should surface as explicit observable jobs
+  - preserve handoff context in repo-local AGENTS and notes files
+  - keep a durable repository-level algorithmic map of vendored Codex so GUI and Epiphany changes hook into the existing app-server/core boundary instead of reimplementing the turn loop
+  - keep the Codex algorithmic map honest about auxiliary side channels and limitations: file watching, unified diffs, compaction, subagent messaging, and memory summarization are real substrate, but they do not by themselves amount to semantic repo understanding
+
+rejected_paths:
+  - name: pinned_map_only
+    reason: still one token stream with no typed write semantics
+  - name: train_new_base_model_first
+    reason: too expensive for a first falsifiable experiment
+  - name: exact_name_open_codex_cli_as_gui_surface
+    reason: ymichael/open-codex is terminal-focused and has no Web UI or desktop surface
+  - name: read_opencodex_web_readme_claude_models_as_current_gui_support
+    reason: the Web UI has stale/aspirational Claude/Gemini model definitions, but the new-session selector only exposes Codex and the serve backend rejects non-Codex agents
+  - name: imdigitalashish_opencodex_as_primary_gui_surface
+    reason: the repo is a Tauri wrapper around an archived/vendored opencode CLI, has mismatched GUI/API model IDs versus the CLI model registry, runs the non-interactive tool path with broad auto-approval, and commits generated/binary/local-state artifacts
+  - name: third_party_opencodex_forks_as_gui_base
+    reason: audited candidates were either terminal-only, stale/aspirational, or incoherent wrappers; the comparison clones were removed from vendor to avoid building on untrusted drift
+
+current_status:
+  summary: Workspace initialized with protocol files, typed state, a tiny CLI, a git repo, fresh-workspace handoff notes, architecture rationale, a Codex algorithmic map note, an Epiphany core-harness surface spec, and an actionable Phase 1 implementation plan against vendored Codex. Third-party OpenCodex comparison clones were audited and removed.
+  next_action: Implement the Phase 1 internal/dev-usable Epiphany core state slice: add the minimal protocol types, store Epiphany state in SessionState, persist one rollout snapshot per real user turn, rehydrate it through rollout reconstruction, patch exhaustive RolloutItem matches, and add persistence/replay compatibility tests before touching prompts, presets, or GUI surfaces.

state/scratch.md

@@ -0,0 +1,26 @@
+# Scratch
+
+This file is intentionally disposable.
+
+## Current Subgoal
+
+- Confirm the scaffold is coherent enough to use.
+- Decide whether to patch vendored Codex with the lightweight Epiphany preset now.
+- Decide whether the next executable artifact should be a patch runner or a verifier helper.
+
+## Working Notes
+
+- The protocol should be usable by any capable model because most of the structure lives outside the model.
+- The first test should be small enough to run repeatedly without becoming its own research program.
+- If the protocol becomes too ceremonial, it will lose to plain prompting for the most boring reason imaginable: friction.
+- The current CLI is intentionally small. It exists to keep state honest, not to impersonate a full agent runtime.
+- The Codex repo already has a cleaner-than-expected collaboration-mode preset seam, so a preset-backed Epiphany mode is a tractable first implementation.
+
+## Open Questions
+
+- What is the smallest useful verifier stack?
+- Which coding task will expose drift quickly?
+- How much branch management is enough before it turns into paperwork?
+- Should the first in-Codex implementation be instructions-only, or should it also add a tiny state helper tool?
+
+Do not promote anything from here into the map unless it survives verification or repeated reuse without contradiction.

tools/epiphany_state.py

@@ -0,0 +1,179 @@
+from __future__ import annotations
+
+import argparse
+import json
+from datetime import datetime, timezone
+from pathlib import Path
+from typing import Any
+
+
+ROOT = Path(__file__).resolve().parents[1]
+STATE_DIR = ROOT / "state"
+MAP_PATH = STATE_DIR / "map.yaml"
+BRANCHES_PATH = STATE_DIR / "branches.json"
+EVIDENCE_PATH = STATE_DIR / "evidence.jsonl"
+
+
+def read_text(path: Path) -> str:
+    return path.read_text(encoding="utf-8")
+
+
+def write_text(path: Path, content: str) -> None:
+    path.write_text(content, encoding="utf-8")
+
+
+def load_branches() -> dict[str, Any]:
+    return json.loads(read_text(BRANCHES_PATH))
+
+
+def save_branches(data: dict[str, Any]) -> None:
+    write_text(BRANCHES_PATH, json.dumps(data, indent=2) + "\n")
+
+
+def append_evidence(record: dict[str, Any]) -> None:
+    with EVIDENCE_PATH.open("a", encoding="utf-8") as handle:
+        handle.write(json.dumps(record, ensure_ascii=True) + "\n")
+
+
+def utc_stamp() -> str:
+    return datetime.now(timezone.utc).replace(microsecond=0).isoformat()
+
+
+def extract_map_field(name: str) -> str | None:
+    prefix = f"  {name}:"
+    for line in read_text(MAP_PATH).splitlines():
+        if line.startswith(prefix):
+            return line.split(":", 1)[1].strip()
+    return None
+
+
+def extract_active_subgoals() -> list[str]:
+    lines = read_text(MAP_PATH).splitlines()
+    results: list[str] = []
+    in_section = False
+    for line in lines:
+        if not in_section:
+            if line.strip() == "active_subgoals:":
+                in_section = True
+            continue
+        if line.startswith("  - "):
+            results.append(line[4:].strip())
+            continue
+        if line and not line.startswith(" "):
+            break
+    return results
+
+
+def cmd_status(_: argparse.Namespace) -> int:
+    branches = load_branches().get("branches", [])
+    active = [branch for branch in branches if branch.get("status") == "active"]
+    summary = extract_map_field("summary") or "(missing)"
+    next_action = extract_map_field("next_action") or "(missing)"
+    subgoals = extract_active_subgoals()
+
+    print(f"Workspace: {ROOT}")
+    print(f"Summary: {summary}")
+    print(f"Next action: {next_action}")
+    print(f"Active branches: {len(active)} / {len(branches)}")
+    if subgoals:
+        print("Active subgoals:")
+        for item in subgoals:
+            print(f"- {item}")
+    return 0
+
+
+def cmd_add_evidence(args: argparse.Namespace) -> int:
+    record: dict[str, Any] = {
+        "ts": utc_stamp(),
+        "type": args.type,
+        "status": args.status,
+        "note": args.note,
+    }
+    if args.branch:
+        record["branch"] = args.branch
+    append_evidence(record)
+    print("Appended evidence record.")
+    return 0
+
+
+def cmd_add_branch(args: argparse.Namespace) -> int:
+    data = load_branches()
+    branches = data.setdefault("branches", [])
+    if any(branch.get("id") == args.id for branch in branches):
+        raise SystemExit(f"Branch '{args.id}' already exists.")
+    branches.append(
+        {
+            "id": args.id,
+            "hypothesis": args.hypothesis,
+            "status": "active",
+            "artifacts": args.artifact or [],
+            "notes": args.note or "",
+        }
+    )
+    save_branches(data)
+    print(f"Added branch '{args.id}'.")
+    return 0
+
+
+def cmd_close_branch(args: argparse.Namespace) -> int:
+    data = load_branches()
+    for branch in data.get("branches", []):
+        if branch.get("id") != args.id:
+            continue
+        branch["status"] = args.status
+        if args.note:
+            branch["notes"] = args.note
+        save_branches(data)
+        print(f"Updated branch '{args.id}' to status '{args.status}'.")
+        return 0
+    raise SystemExit(f"Branch '{args.id}' was not found.")
+
+
+def build_parser() -> argparse.ArgumentParser:
+    parser = argparse.ArgumentParser(
+        description="Inspect and update EpiphanyAgent state files."
+    )
+    subparsers = parser.add_subparsers(dest="command", required=True)
+
+    status_parser = subparsers.add_parser("status", help="Show a compact state summary.")
+    status_parser.set_defaults(func=cmd_status)
+
+    evidence_parser = subparsers.add_parser(
+        "add-evidence", help="Append one JSONL evidence record."
+    )
+    evidence_parser.add_argument("--type", required=True)
+    evidence_parser.add_argument("--status", required=True)
+    evidence_parser.add_argument("--note", required=True)
+    evidence_parser.add_argument("--branch")
+    evidence_parser.set_defaults(func=cmd_add_evidence)
+
+    add_branch_parser = subparsers.add_parser(
+        "add-branch", help="Create a new active branch entry."
+    )
+    add_branch_parser.add_argument("--id", required=True)
+    add_branch_parser.add_argument("--hypothesis", required=True)
+    add_branch_parser.add_argument("--artifact", action="append")
+    add_branch_parser.add_argument("--note")
+    add_branch_parser.set_defaults(func=cmd_add_branch)
+
+    close_branch_parser = subparsers.add_parser(
+        "close-branch", help="Close an existing branch."
+    )
+    close_branch_parser.add_argument("--id", required=True)
+    close_branch_parser.add_argument(
+        "--status", required=True, choices=["accepted", "rejected", "archived"]
+    )
+    close_branch_parser.add_argument("--note")
+    close_branch_parser.set_defaults(func=cmd_close_branch)
+
+    return parser
+
+
+def main() -> int:
+    parser = build_parser()
+    args = parser.parse_args()
+    return args.func(args)
+
+
+if __name__ == "__main__":
+    raise SystemExit(main())

vendor/codex

@@ -0,0 +1 @@
+Subproject commit e8ba912fcc9971d899808870821b83614ced7508