From f05686c137c6fe02160d049664ca2307eca818a8 Mon Sep 17 00:00:00 2001 From: Meta Vens Date: Tue, 15 Sep 2026 19:52:32 +0200 Subject: [PATCH] Map the Eureka pipeline state cuts Imagination's cut map, built from probes: 1. Re-pin CultLib: 12 load_envelope sites; core 154/154 at the new pin. 2. One profile-parameterized commit owner. 3. The pipeline document set, lease, admission, queries, and a merge command. 4. The eureka-state stdio MCP package. 5. Skill wiring. 6. A proof campaign on the CultRecordRefFormatter follow-ups. 7. voidbot projection. Five operator questions are open. The target now drops the claim that the store merges in git, and records that voidbot indexes only default branches. Co-Authored-By: Claude Opus 5 --- notes/eureka-pipeline-state-cut.md | 1265 +++++++++++++++++++++++++ notes/eureka-pipeline-state-target.md | 14 +- 2 files changed, 1273 insertions(+), 6 deletions(-) create mode 100644 notes/eureka-pipeline-state-cut.md diff --git a/notes/eureka-pipeline-state-cut.md b/notes/eureka-pipeline-state-cut.md new file mode 100644 index 00000000..dbd9f078 --- /dev/null +++ b/notes/eureka-pipeline-state-cut.md @@ -0,0 +1,1265 @@ +# Eureka pipeline state: cut map + +Date: 2026-09-15 (first Imagination pass). + +Status: cut map. The ends are owned by `notes/eureka-pipeline-state-target.md`; +this document owns the means. Self updates this header in every landing commit. +No cut has landed. + +Pins. Code anchors are `file:line` against Epiphany `81be7a2f`. HEAD has since +moved to `0636176a`, but only the target and `state/map.yaml` changed, so every +code anchor still holds. The other pins: + +- CultLib `main` `a0813c6`. +- VoidBot `main` `46d891b`. +- gamecult-ops `main` `647e57e`. +- The Eureka skill at `~/.claude/skills/eureka`, which is not a git repository. + +## Rulings in force (operator, 2026-09-15) + +1. Epiphany owns the schemas and admission. +2. The store serves agents first. Exact and filtered queries live in Epiphany; + voidbot indexes a projection for semantic search. The typed store is the truth. +3. There are two campaigns. This one covers schemas, store, admission, the MCP + server, the voidbot projection, the skill wiring, and a proof. Epiphany's own + organs consuming the documents belong to the second. +4. The MCP server is Rust, stdio, launched per Claude Code session, built as an + Epiphany package, and adds no daemon. It writes through a public admission + path under organ provenance `eureka`. +5. **The store lives in the repo where the task runs.** It is committed on the + campaign's working branch and is separate from the runtime and Mind stores, + with its own epoch. Knowledge crosses repos only through explicit sharing. The + operator's words: "have the pipeline store sit in the repo where the task + takes place. Each Epiphany learns for itself, with affordances for sharing + knowledge, same when running Eureka. Then the Eureka working branch can sync + with Yggdrasil". + (This supersedes the earlier same-day ruling of one user-level store. Nothing + below describes that design.) +6. **One runner per repo, plus a merge tool.** The operator's words: "There + should only ever be one Epiphany/Eureka running in any one repo. Nonetheless, + we should have a merge tool in case it ever happens." +7. Re-pin CultLib from `e171eca3` to `a0813c6` first. + +## Open operator questions + +Each question lists what depends on it. Hands can start Cuts 1 and 2 before any +answer. Cut 3a needs Q1 and Q5, Cut 3c needs Q3, Cut 3a's epoch text needs Q2, +and Cut 7 needs Q4. + +- **Q1. Store layout.** Candidates: + - **A.** One single-file `.cc` per repo, merged by an explicit admission-replay + command. + - **B.** Port C#'s v4 directory store to cultcache-rs. + - **C.** One single-file `.cc` per record, with receipts acting as batch + manifests. + + **Recommended: A.** Git never merges the bytes of a binary file, so every + divergent merge is forced through the merge command, which is Epiphany's + admission. B and C both let git silently union two branches' records around + admission. + - B also adds a CultLib foundation store with C# wire parity, and its manifest + is one hot file that conflicts on every divergent branch anyway + (`DirectoryMessagePackBackingStore.cs:15-18`, `WriteManifest` at `:287-295`). + - C loses batch atomicity: a multi-file batch cannot meet cultcache-rs's + atomic `push_all` contract (`lib.rs:307-336`). + - Probes, 300 documents of about 1 KB each, 30 commits, after + `git gc --aggressive`: + + | Layout | Pack size | + |---|---| + | single file | 20 KiB | + | per record | 46 KiB | + + The payload was highly compressible, so these are lower bounds. The per-record + layout merged disjoint branches cleanly, and a same-subject resolution raised + an add/add conflict. Single-writer (ruling 6) removes the case per-record was + built for. + - Depends on it: Cut 3a's store module, Cut 3c, and Cut 7's discovery path. +- **Q2. An older repo store meets a newer schema.** Candidates: + - **A.** Additive changes (new named fields carrying `#[serde(default)]`) keep + the epoch. A breaking change bumps it, and the new binary refuses the old + store for reads and writes with `ForeignEpoch`. The old store stays in git + history and remains readable by the binary tagged at its epoch. The operator + decides per repo whether to re-author live state. + - **B.** A breaking bump ships a one-release `pipeline-migrate` that + re-admits old documents through admission. + - **C.** A compatibility reader. + + **Recommended: A.** It keeps "no compatibility reader" (target, Store) and + Epiphany's no-migrator precedent (`state/map.yaml:118`, "No compatibility + reader, bootstrap aggregate, schema migrator, or dual reader survives"). + Nested values are named MessagePack maps, which tolerate added fields + (`cultcache-rs` `prepare_entry_named` docs, `lib.rs:2176-2180`). B is the + fallback if a live campaign ever straddles a breaking bump. + - Depends on it: Cut 3a's opener refusal text and the README's evolution rule. +- **Q3. Settling a merge conflict.** Candidates: + - **A.** The merge is all-or-nothing. It reports every conflict and writes + nothing. The operator settles by admitting, on ours, an explicit typed + `merge_exclusion` naming the theirs receipt and the reason, then re-runs the + merge. The merge skips excluded batches and reports their dependents as + conflicts too. The excluded documents are re-authored by hand. + - **B.** The merge only refuses and reports. Settlement happens outside the + tool. + + **Recommended: A.** Under B, a conflicted binary store cannot be merged at + all, because neither side's history can be rewritten. A keeps the settlement + typed, dated and queryable, and never lets the last writer win. + - Depends on it: whether Cut 3c adds the eleventh kind, `merge_exclusion`. +- **Q4. voidbot index scope.** The target says pushing the working branch is the + sync. The Body says otherwise: the mirror catalogs only public repos and fetches + only each repo's `default_branch` (gamecult-ops + `scripts/refresh-voidbot-sources.sh:84-89`, + `scripts/sync-gamecult-org-repos.sh:24-57`). Candidates: + - **A.** Index the default branch only, so learning becomes searchable across + repos when the campaign branch merges. + - **B.** Also mirror and index branches that carry a pipeline store. + + **Recommended: A.** Inside the running repo, Self reads the branch store + directly through `eureka-state`. Cross-repo discovery should surface settled, + merged learning, not abandoned or rebased campaign state. Private repos such as + gamecult-ops are never indexed under either option. + - Depends on it: Cut 7's scope. +- **Q5. `schemars` in `epiphany-core`.** Commit `07529fb5` deleted unused + `schemars` from Epiphany (`state/map.yaml:96-108`). Candidates: + - **A.** Make it an unconditional core dependency, derived on the pipeline + value types. Published JSON schemas and MCP tool schemas then come from one + Rust authority. + - **B.** Put it behind a core feature enabled only by the MCP package. + - **C.** Hand-write the JSON schemas and give MCP its own DTOs. + + **Recommended: A.** It now has two live consumers: the schema publication and + rmcp tool schemas, since rmcp's `server` feature already pulls `schemars` 1.x. + - B compiles `epiphany-core` twice in the shared target, because a + dependency's features change core's fingerprint. + - C creates a second owner of every field. + - Depends on it: Cut 3a's dependency line and Cut 4's tool types. + +## Probes and what they established + +All probes ran under the scratchpad at +`...\scratchpad\eureka-imagination\`. Cargo used +`CARGO_TARGET_DIR=C:\Users\Meta\.cargo-target-codex`, one invocation at a time. +Afterwards 3,796 probe outputs (2.1 GiB) were deleted. The shared target is back +to 5.6 GiB, with `debug\epiphany-state.exe` (1,776,640 bytes, 2026-08-24) +preserved. The scratch worktree was removed. + +| # | Probe | Result | +|---|---|---| +| P1 | Detached worktree at `81be7a2f`, pins sed-replaced to `a0813c6`, then `cargo check -p epiphany-core --lib` and `--lib --tests` | 12 × E0599 `load_envelope` not found; nothing else. `Cargo.lock` changes 5+/5−. | +| P2 | P1 plus `load_envelope` → `put_envelope` at the 12 sites | Lib builds clean except 3 `unused Result` warnings; lib tests add 5 more (8 total). | +| P3 | P2: `cargo test -p epiphany-core --lib`; `cargo check --lib` for `epiphany-model-adapter`, `epiphany-tool-adapter`, `epiphany-openai-runtime`, `epiphany-tool-mcp-runtime` | 154/154 tests pass; all four checks pass. Test children read before running: `git`, a PowerShell sleep, `cargo metadata`. None re-launches itself. | +| P4 | `rmcpprobe`: rmcp 2.2.0 with `default-features=false, features=["server","macros","transport-io"]`; one `#[tool]` returning `Json`; driven over stdin with JSON-RPC | Handshake at `protocolVersion 2025-06-18` works. `tools/list` carries both `inputSchema` and `outputSchema` from schemars. The call returns `structuredContent` plus a text copy. `ErrorData::invalid_params` becomes JSON-RPC `-32602`. The binary spawns nothing. | +| P5 | `lockprobe`: `SingleFileMessagePackBackingStore` at `a0813c6`; 4 writer processes × 200 (a disjoint insert plus a read-modify-CAS counter), 2 reader processes × 500 | 800/800 rows, counter 800/800, 0 read errors. Each writer lost about 500 CAS races, which it retried. The store lock is held per operation (`lib.rs:705-722`), so it serializes commits but excludes nothing at session scope. | +| P6 | `layoutprobe`: per-record files, a store-wide fs2 lock, receipt written last; 4 writers × 60 batches with 2 overlapping unlocked auditors | 240/240 receipts, 720/720 records, about 2.6M audited reads, 0 violations. On Windows, atomic replace is safe under concurrent unlocked readers. Kept as evidence for Q1-C, which is not recommended. | +| P7 | cultcache-ts `inspectCultCacheBytes` (CultLib dist 0.14.0) on a Rust-written record (a `DatabaseEntry` wrapper around a named nested struct holding an enum and an `Option`) | Format is `cultcache.store.v1`. The payload decodes as `[ {id, text, confidence: "Plausible", locations: [...], cut: null} ]`. Rust writes an empty member catalog (`members: []`), so the TS decode is generic, not typed. | +| P8 | Git with `* text=auto` (Epiphany's `.gitattributes`); system `core.autocrlf=true` on this machine | A 434-byte record holds no NUL, so git classified it `text: auto` (numstat `1 0`). With `binary` set, numstat is `- -` and `text/diff/merge` are unset. | +| P9 | Git merge rehearsal on per-record files | Disjoint branches merge cleanly; two branches writing the same resolution key conflict add/add. | +| P10 | Session lease: process A takes fs2 `try_lock_exclusive` and writes a sibling holder file; process B tries; A is killed with `-9`; B tries again | B is refused with `os error 33`, and the holder file is readable while the lock is held. After the kill, B acquires. The OS releases on death, so no repair step is needed. | +| P11 | Git history growth (Q1) | See Q1. | + +Source reads that mechanism claims rely on: + +- **Every `load_envelope` site uses a storeless cache.** Each builds + `CultCache::new()` and feeds it envelopes it pulled separately: + `persona_conversation.rs:959-961`, `persona_social_state.rs:231-251` then + `:846-866`, and `resident_self.rs:764-815`. `put_envelope` with zero stores + inserts into memory only (`home_index` returns `None`, `lib.rs:2425-2432`), + which is exactly the deleted `load_envelope` (CultLib `4ed9871`: "it had no + callers and admitted records into the view without a store"). +- **The old pin already refused unregistered types at pull** + (`e171eca3` `lib.rs:1965-1986`). The re-pin adds only the home-store check, + and every Epiphany cache has a single generic store. +- **Cargo.lock has one CultLib source** (`Cargo.lock:367,385,395,406`). No + second revision arrives through codex-connector or ghostlight. +- **VoidBot's vendored cultcache-ts is 0.1.0** and decodes only the legacy + envelope array (`vendor/cultcache-ts/dist/single-file-messagepack-backing-store.js:31`). + It cannot read a `cultcache.store.v1` store. + +Not probed: the working directory Claude Code gives a user-scope stdio server. +The design does not depend on it, because every tool takes an explicit +`repo_root`. + +## Shared design (cuts cite these sections) + +### D1. Documents + +Every pipeline document follows the Mind value-wrapper pattern +(`mind_documents.rs:129-137`). A `DatabaseEntry` has one slot: + +```rust +#[derive(Clone, Debug, PartialEq, Eq, DatabaseEntry)] +#[cultcache(type = "epiphany.pipeline.ruling.v1", schema = "EpiphanyPipelineRulingDocument")] +pub struct EpiphanyPipelineRulingDocument { #[cultcache(key = 0)] pub value: PipelineRuling } +``` + +Encoding and schema follow from that shape: + +- The value is a plain `serde` + `schemars::JsonSchema` struct. +- It is always prepared with `prepare_entry_named` (`lib.rs:2181-2200`). +- On the wire the payload is a one-element MessagePack array whose element is a + named map (P7, derive `lib.rs:216-245`). +- There are no per-field slot numbers: the named map is the evolution surface, + and slot 0 is the only slot. +- The published JSON Schema describes the value, which is exactly what MCP + clients send and receive and what cultcache-ts decodes as + `payloadPreview[0]`. + +Type ids are `epiphany.pipeline..v1`. Schema names are +`EpiphanyPipelineDocument`. The store epoch is `epiphany.pipeline.epoch.v1`. + +**Bounds, with no blob fields.** No `Vec`, no `serde_json::Value`, and no +free-form maps. Text fields use three bounded aliases, checked in UTF-8 bytes +following the `persona_feedback_admission.rs:994` precedent: + +| Alias | Bound | +|---|---| +| `Short` | ≤ 200 bytes | +| `Line` | ≤ 1,000 bytes | +| `Para` | ≤ 4,000 bytes | + +Every list has a maximum count, given below. Long narrative stays in repo docs and +is cited by `DocRef { path: Short, start_line: u32, end_line: u32, commit: Sha }`. +A breach is refused with `FieldBound { field, limit, actual }`. + +**Shared value types.** + +- `Sha`: 7–40 lowercase hex characters. +- `Date`: `YYYY-MM-DD`. +- `PipelineKind` is a closed enum: `Campaign`, `Target`, `Question`, `Ruling`, + `CutSpec`, `CutReport`, `Verdict`, `Finding`, `FollowUp`, `Resolution`, plus + `MergeExclusion` only if Q3 is A. +- `PipelineRef { kind, id: Short }`. +- `CodeLocation { path: Short, line: u32, end_line: Option }`. +- `CommitRange { base: Sha, head: Sha }`. +- `Evidence { kind: EvidenceKind, locator: Line, result: Line }`, where + `EvidenceKind` is one of `Command`, `Test`, `Mutation`, `Probe`, `SourceRead`, + `Capture`. +- `ForeignRef { repo: Short /* Org/Repo */, commit: Sha(40), kind, id: Short, payload_sha256: 64 hex }`: + the sharing citation (D6). +- `Faculty`: `SelfFaculty`, `Imagination`, `Hands`, `Soul`, `MindSteward`, + `Eyes`, `Operator`. + +**Keys: identity, not convenience.** Single-writer means a key only has to be +unique within one repo's history. It must also be stable across a merge, so that +the same document on two copies is the same key and a different document under +the same key surfaces as a real conflict. Keys are semantic and derived from +fields; admission recomputes each key and refuses a mismatch with +`InvalidIdentity { kind, key, expected }`, as `validate_mind_write_envelope` does +for Mind (`mind_documents.rs:357-382`). + +| Kind | Key | +|---|---| +| campaign | `` | +| everything else | `::` | + +`` and `` match `[A-Za-z0-9._-]{1,64}`. `` is the human label +agents already cite: + +| Kind | Local label | Example | +|---|---|---| +| target | `r` | `r2` | +| question | `Q-` | `Q10-1` | +| ruling | label | `R6` | +| cut spec | `cut-