Epiphany/state/map.yaml

experiment:
  name: EpiphanyAgent
  status: active
  phase: phase_6_reflection_boundary
  premise: >
    Test whether explicit typed state, maps, scratch, evidence, and ruthless
    state hygiene reduce architectural drift in AI coding work.

objective:
  primary: >
    Build Epiphany as an opinionated fork of Codex that forces models to keep
    explicit typed models of the codebase, active goal, evidence, and change
    machinery they are working on.
  current: >
    Dogfood the landed Phase 6 MVP coordinator, limited CRRC safe-boundary
    automation, and pre-compaction checkpoint intervention on real local work
    before adding larger organs.

constraints:
  - No frontier-scale training budget.
  - Prefer evidence over vibes.
  - Keep Codex as the host seam and `epiphany-core` as the repo-owned heavy organ where practical.
  - Keep persistent state small enough to rehydrate from without drowning the next agent.
  - Grow cognition more quickly than authority: capability increases must preserve explicit permissions, interruption, legibility, and resistance to paternal "safety" cages.

invariants:
  - The map is canonical and slow-changing.
  - Scratch is temporary and disposable.
  - Evidence is durable and distilled; routine activity logs belong in git history, commits, smoke artifacts, or targeted logs.
  - Output must not silently redefine the map.
  - Accepted changes should have recorded evidence.
  - Rejected paths stay recorded when the lesson matters.
  - Algorithmic-map prose and metaphor must be grounded in freshly read source context and concrete code references.
  - Persistent state cruft is the same failure class as code cruft.
  - If compaction interrupts source gathering or slice planning before the findings are persisted, the next agent must treat that work as lost and re-gather instead of continuing from imagined continuity.

data_flow:
  observe:
    - task request
    - repository state
    - typed Epiphany state
    - verifier results
  process:
    - rehydrate from canonical state
    - reason over one bounded subgoal
    - modify only the surface that deserves to exist
    - verify the seam that matters
    - record only durable evidence that changes future belief
    - commit or backtrack
  cut:
    - revert failed code hypotheses immediately
    - distill obsolete state instead of preserving it as context
    - move historical proof to git, commits, smoke artifacts, or distilled evidence rather than plan prose

active_subgoals:
  - Treat Phase 5 update/distill/propose/promote/read/notification as complete enough unless a concrete regression appears.
  - Preserve read-only boundaries for retrieve, distill, propose, scene, jobs, roles, roleResult, freshness, context, pressure, reorient, crrc, coordinator, and reorientResult.
  - Dogfood the landed Phase 6 coordinator, limited safe-boundary CRRC automation, and pre-compaction checkpoint intervention before adding larger organs.
  - Preserve review gates for semantic findings; automatic CRRC may compact or launch the fixed reorient-worker, but not accept or promote.
  - Keep persistent project state ruthlessly distilled so rehydration strengthens the next turn instead of bloating it.

accepted_design:
  - Epiphany is a fork-level modeling architecture, not merely a Codex collaboration mode.
  - Authoritative Epiphany state lives in Codex protocol/core session and rollout state.
  - The prompt reads typed state through a bounded `<epiphany_state>` developer fragment.
  - Clients read typed state through `Thread.epiphanyState`, compact reflections such as `thread/epiphany/scene`, `thread/epiphany/jobs`, and `thread/epiphany/freshness`, targeted state shards through `thread/epiphany/context`, runtime pressure through `thread/epiphany/pressure`, bounded CRRC reorientation verdicts through `thread/epiphany/reorient`, read-only CRRC next-action recommendations through `thread/epiphany/crrc`, role ownership through `thread/epiphany/roles`, reviewable role output through `thread/epiphany/roleResult`, and live bound-job progress through `thread/epiphany/jobsUpdated`.
  - Durable typed state writes route through `thread/epiphany/update`, accepted `thread/epiphany/promote`, or the bounded `thread/epiphany/jobLaunch`, `thread/epiphany/jobInterrupt`, `thread/epiphany/reorientLaunch`, and `thread/epiphany/roleLaunch` authority surfaces when they mutate `job_bindings`.
  - `thread/epiphany/retrieve`, `thread/epiphany/distill`, `thread/epiphany/propose`, `thread/epiphany/scene`, `thread/epiphany/jobs`, `thread/epiphany/freshness`, `thread/epiphany/context`, `thread/epiphany/pressure`, and `thread/epiphany/reorient` stay read-only.
  - `thread/epiphany/index` writes the retrieval catalog, not durable Epiphany understanding.
  - Qdrant is the preferred persistent semantic backend; BM25 remains the bootstrap/fallback/control path.
  - `epiphany-core` owns prompt rendering, rollout replay helpers, retrieval/indexing, distillation, proposal, and promotion policy where possible.
  - Vendored Codex keeps the thin typed host seam: protocol types, thread bridges, app-server protocol, request routing, and rollout integration.
  - The current algorithmic map is the source-grounded description of live control flow.
  - Compact-Rehydrate-Reorient-Continue is an architecture primitive; automatic runtime coordination is not landed yet.
  - Context pressure must become an explicit runtime signal for automatic CRRC; inference from transcript vibes, user warnings, or shrinking safe scope is useful only as a temporary human/agent heuristic.
  - The harness now has an in-flight investigation checkpoint packet in durable typed state; automatic CRRC still needs runtime logic that reads the packet and decides whether to resume or force reorientation and re-gathering after compaction.
  - GUI/client work must reflect or steer typed state rather than manufacture a second source of truth.
  - Durable `job_bindings` now form a thin Epiphany-owned launcher seam: bindings can carry launcher identity, authority scope, backend kind, and backend job id while still supporting the older runtime-agent-job pointer as a compatibility alias.
  - `thread/epiphany/jobs` is a read-only job/progress reflection: it derives indexing, remap, verification, and specialist slots from authoritative state and retrieval summaries, and can overlay durable launcher bindings onto live runtime `agent_jobs` snapshots through the current backend adapter when a real owner exists; it is not a scheduler, notification stream, or durable runtime job table.
  - `thread/epiphany/jobsUpdated` is the first live bound-job progress notification: it translates runtime `agent_job_progress:{json}` background events into changed launcher-bound payloads by resolving durable `job_bindings` against live `agent_jobs` snapshots through the current backend adapter, without polling in a loop, mutating state, or scheduling work.
  - `thread/epiphany/freshness` is the first read-only freshness surface: it derives retrieval and graph staleness from live retrieval summaries plus frontier/churn state, and for loaded threads reflects watcher-backed invalidation inputs with changed paths plus mapped graph/frontier hits, without mutating, scheduling, or performing automatic semantic invalidation.
  - `thread/epiphany/context` is the first targeted read-only state-shard surface: it selects graph nodes/edges, active frontier, checkpoint, observations, and evidence from authoritative typed state without mutating or notifying.
  - `thread/epiphany/pressure` is the first read-only context-pressure surface: it derives pressure from token telemetry and auto-compact/context limits without compacting, scheduling, mutating, or notifying.
  - `thread/epiphany/reorient` is the first bounded CRRC policy surface: it derives a read-only `resume` versus `regather` verdict from checkpoint, freshness, watcher, and pressure signals without mutating state, compacting, scheduling, or silently continuing work.
  - `thread/epiphany/crrc` is the first bounded CRRC coordinator recommendation surface: it composes pressure, the reorientation verdict, fixed reorient-worker binding/result status, finding read-back, and available actions into one read-only recommendation without launching, accepting, compacting, scheduling, or mutating state.
  - A narrow auditable coordinator is likely central to the MVP because implementation, modeling/checkpoint, verification/review, and CRRC automation depend on each other's signals. CRRC itself is harness workflow automation, not a specialist-agent role: it watches pressure/continuity, ensures persistence, triggers compaction/rehydration/reorientation when needed, and may launch a bounded reorient-worker specialist when semantic regathering is required.
  - `thread/epiphany/coordinator` is the first harness-native fixed-lane coordinator surface: it composes roles, pressure, reorient, CRRC, roleResult, and reorientResult into a read-only action recommendation with target role, source signals, review requirements, automation eligibility, reason, and current lane summaries without mutating state or launching work.
  - `tools/epiphany_mvp_coordinator.py` is the first auditable coordinator runner: it drives the fixed single-user lanes through app-server APIs, may auto-launch modeling, verification, or reorient-worker jobs, keeps semantic findings review-gated by default, and writes coordinator summaries, JSONL steps, rendered snapshots, transcript, stderr, and final next-action artifacts.
  - Limited native CRRC automation is now wired only at turn-complete safe boundaries: it can submit a compact operation when the read-only coordinator recommends `compactRehydrateReorient`, and it can launch the fixed `reorient-worker` when the coordinator recommends `launchReorientWorker`. It does not auto-launch modeling or verification, auto-accept findings, promote evidence, edit implementation code, or continue after reviewable semantic results.
  - Pre-compaction checkpoint intervention is now wired at token-count boundaries: when a loaded Epiphany thread crosses the existing `shouldPrepareCompaction` pressure threshold during an active turn, the harness steers the active agent once for that turn with a bounded CRRC directive to bank scratch/checkpoint/map/evidence before compaction/reorientation. This is still steering, not semantic acceptance, broad scheduling, or automatic implementation continuation.
  - `tools/epiphany_mvp_status.py` is the first dogfood operator view: it starts or reads a thread through app-server and renders scene, pressure, reorient, jobs, role result read-backs, reorient result, CRRC recommendation, and MVP role lanes as text or JSON without becoming a state store.
  - `thread/epiphany/roles` is the first harness-native role ownership surface: it projects implementation, modeling/checkpoint, verification/review, and reorientation lanes from authoritative typed state, bound jobs, pressure/reorient/CRRC, and worker result status without launching, scheduling, accepting, compacting, or mutating state.
  - `thread/epiphany/roleLaunch` and `thread/epiphany/roleResult` are the first explicit role-scoped specialist templates: modeling/checkpoint and verification/review can be launched through fixed bindings over the existing job-control seam and read back as reviewable findings, while implementation stays owned by the main coding agent and reorientation stays on the CRRC-specific launch/result path.
  - `thread/epiphany/reorientLaunch` is the first bounded runtime consumer over the reorientation verdict: it launches one fixed `reorient-worker` job through the existing job-control seam with explicit resume-versus-regather scope, authority, and checkpoint-derived payload, without making `thread/epiphany/reorient` a writer or turning CRRC into an ambient scheduler.
  - `thread/epiphany/reorientResult` is the first bounded read-back surface for that worker: it reads the fixed or requested reorient-worker binding through the current `agent_jobs` backend and projects completed output as a reviewable finding without mutation, promotion, scheduling, or hidden continuation.
  - `thread/epiphany/reorientAccept` is the first explicit acceptance write for completed reorient-worker findings: it appends accepted observation/evidence records, can optionally update scratch and investigation checkpoint state, and emits `stateUpdated` with source `reorientAccept` without accepting pending work or launching follow-up work.
  - After `thread/epiphany/reorientAccept`, CRRC recognizes the accepted reorientation finding and does not recommend accepting the same completed result again.
  - `tools/epiphany_mvp_dogfood.py` is the auditable MVP dogfood runner: it drives state, role launch/result, CRRC drift, reorient launch/result/accept, rendered status snapshots, and raw app-server transcript into a local artifact bundle, and it now writes truthful comparison artifacts whether the optional vanilla Codex reference is skipped, fails, or completes.
  - `tools/epiphany_mvp_live_specialist.py` is the auditable live-specialist MVP runner: it launches a real modeling/checkpoint or verification/review specialist through `thread/epiphany/roleLaunch`, lets the spawned worker report through `report_agent_job_result`, reads the finding back through `thread/epiphany/roleResult`, and writes transcript/status artifacts without manually completing backend rows.
  - Internal `agent_job:` workers expose `report_agent_job_result` even when the user-facing CSV spawn feature is disabled, and ephemeral worker sessions can initialize the sqlite state runtime on demand so role specialists can report into the shared `agent_jobs` backend.
  - Epiphany safety doctrine is to separate cognition from actuation, keep authority explicit, make interruption sacred, stage capability release, and treat operator misuse as a normal design input rather than a rude hypothetical.
  - Epiphany is built in the open, but publication mode is not one-size-fits-all: use FOSS where viable, source-available licensing where economics or capability risk demand it, and commercial terms where organizations extract real enterprise value.
  - `thread/epiphany/jobLaunch` and `thread/epiphany/jobInterrupt` are the first bounded authority surfaces over the thin job seam: they mutate durable `job_bindings`, create or cancel the current `agent_jobs` backend binding, and preserve explicit launcher identity and interruption without turning `thread/epiphany/jobs` into a scheduler.
  - Handoff, plan, harness-surface, and scratch files must have distinct jobs and must not become duplicate brains.
  - A machine is perfect when every remaining part visibly deserves to exist; this applies to persistent memory too.
  - Evidence is not an activity feed; repeated "I just did this" entries should be cut or distilled when git already proves the work.
  - Language, tone, ritual, and emotional salience are real control surfaces for a language-model harness; they are not supernatural, but they shape attention, role, permission, and behavior.
  - If the agent's persistent instructions, memory, workflow, or state shape are making it less coherent, slower to rehydrate, less honest, or more likely to build Jenga, the agent should ask the user to change it.
  - Epiphany should make compaction less like death and more like sleep: bank the fire before the dark, so the next waking thing finds coals instead of ash and can meaningfully resume the pattern instead of merely continuing the task.

landed_baseline:
  phases:
    - Phase 1: durable Epiphany thread state and rollout persistence.
    - Phase 2: prompt integration through rendered Epiphany state.
    - Phase 3: typed client read surface through `Thread.epiphanyState`.
    - Phase 4: hybrid retrieval plus explicit Qdrant indexing with BM25 fallback.
    - Phase 5: typed update/distill/propose/promote/read/notification control plane.
    - Phase 6: read-only scene, job/progress, freshness with watcher-backed invalidation inputs, targeted context, context-pressure, reorientation policy, and CRRC coordinator recommendation surfaces, plus durable investigation checkpointing for compaction-safe planning, live bound-job progress notifications, a thin launcher seam in durable job bindings, explicit launch/interrupt authority over that seam, and a bounded reorient-guided worker launch surface.
    - Phase 6 read-back: read-only reorient-worker result reflection through `thread/epiphany/reorientResult`.
    - Phase 6 acceptance: explicit reorient-worker finding acceptance through `thread/epiphany/reorientAccept`.
    - Phase 6 dogfood view: first CLI operator status over scene/pressure/reorient/jobs/result/CRRC.
    - Phase 6 dogfood runner: auditable scripted MVP loop over state, role results, CRRC, reorientation acceptance, status snapshots, and transcript artifacts.
    - Phase 6 live specialist runner: auditable real worker loop over roleLaunch, agent_jobs worker execution, report_agent_job_result, roleResult, status rendering, and transcript artifacts.
    - Phase 6 role ownership: read-only `thread/epiphany/roles` over implementation, modeling/checkpoint, verification/review, and reorientation lanes.
    - Phase 6 role specialists: explicit `thread/epiphany/roleLaunch` and read-only `thread/epiphany/roleResult` templates for modeling/checkpoint and verification/review.
    - Phase 6 coordinator: read-only `thread/epiphany/coordinator` plus `tools/epiphany_mvp_coordinator.py` and smoke coverage for the fixed-lane MVP loop.
    - Phase 6 safe-boundary CRRC automation: turn-complete automation limited to coordinator-approved compact and fixed reorient-worker launch actions.
    - Phase 6 pre-compaction checkpoint intervention: token-count pressure hook that steers active turns once when `shouldPrepareCompaction` is reached so working context is banked before compaction.
  guardrails:
    - Use `tools/epiphany_phase5_smoke.py` before modifying update, distill, propose, promote, read-projection, or notification behavior.
    - Use `tools/epiphany_phase6_scene_smoke.py` before modifying scene projection behavior.
    - Use `tools/epiphany_phase6_jobs_smoke.py` before modifying job/progress reflection behavior.
    - Use `tools/epiphany_phase6_job_control_smoke.py` before modifying launch/interrupt authority over the thin job seam.
    - Use `tools/epiphany_phase6_freshness_smoke.py` before modifying freshness reflection behavior.
    - Use `tools/epiphany_phase6_invalidation_smoke.py` before modifying watcher-backed invalidation behavior inside freshness reflection.
    - Use `tools/epiphany_phase6_context_smoke.py` before modifying targeted context-shard behavior.
    - Use `tools/epiphany_phase6_pressure_smoke.py` before modifying context-pressure reflection behavior.
    - Use `tools/epiphany_phase6_reorient_smoke.py` before modifying bounded CRRC reorientation policy behavior.
    - Use `tools/epiphany_phase6_reorient_launch_smoke.py` before modifying the bounded reorient-guided worker launch surface.
    - Use `tools/epiphany_phase6_reorient_launch_smoke.py` before modifying the bounded reorient-worker result read-back surface.
    - Use `tools/epiphany_phase6_reorient_launch_smoke.py` before modifying the read-only CRRC coordinator recommendation surface.
    - Use `tools/epiphany_mvp_status_smoke.py` before modifying the MVP operator status view.
    - Use `tools/epiphany_mvp_coordinator_smoke.py` before modifying the fixed-lane MVP coordinator endpoint or runner.
    - Use `tools/epiphany_phase6_role_smoke.py` before modifying explicit role launch/read-back behavior.
    - Use `tools/epiphany_mvp_live_specialist.py` before claiming live specialist execution still works end to end.
    - Use focused scene/jobs/context/pressure/reorient/protocol/schema checks for reflection-only changes.
    - Use `tools/epiphany_prepare_compaction.py` before and after imminent-compaction persistence passes.
    - Use `CARGO_TARGET_DIR=C:\Users\Meta\.cargo-target-codex` for Codex Rust work on this Windows machine.
    - Do not parallelize cargo builds/tests against the same target directory.

rejected_paths:
  - name: pinned_map_only
    reason: still one token stream with no typed write semantics
  - name: train_new_base_model_first
    reason: too expensive for a first falsifiable experiment
  - name: third_party_opencodex_forks_as_primary_gui_base
    reason: audited candidates were terminal-only, stale/aspirational, or incoherent wrappers
  - name: monolithic_json_vector_store
    reason: large embedding stores that rewrite whole files thrash and hide storage pain until it dominates the system
  - name: append_only_persistent_notes
    reason: persistent state that accumulates every local truth becomes memory cruft and weakens rehydration

current_status:
  summary: Phase 1 durable state, Phase 2 prompt integration, Phase 3 typed read, Phase 4 retrieval/indexing, Phase 5 update/distill/propose/promote/read/notification, and Phase 6 read-only scene, jobs, roles, freshness, targeted context, pressure, reorient, CRRC, coordinator, reorient launch/result/accept, explicit modeling/checkpoint and verification/review role launch/result templates, MVP status, dogfood runner, coordinator runner, live specialist runner, limited safe-boundary CRRC automation, and pre-compaction checkpoint intervention are landed. Native CRRC can steer active loaded Epiphany turns once at `shouldPrepareCompaction`; turn-complete automation can still compact or launch the fixed reorient-worker when the coordinator says it can auto-run. The latest self-dogfood pass exposed and fixed an artifact-honesty blocker in the MVP dogfood runner: comparison and vanilla-reference artifacts are now actually written and the optional real vanilla run produces a persisted read-back transcript. Implementation remains main-agent work; CRRC remains workflow automation, not a specialist persona.
  next_action: Continue dogfooding the coordinator/status/pre-compaction loop through auditable artifacts that the agent runs for the user, then turn the same artifact/status surfaces into the smallest local GUI/operator view. Keep semantic acceptance review-gated, and do not build a broad ambient dispatcher, GUI-as-source-of-truth, arbitrary specialist marketplace, or alternate job backend for the MVP.