diff --git a/docs/context-pack-map.md b/docs/context-pack-map.md new file mode 100644 index 0000000..8321815 --- /dev/null +++ b/docs/context-pack-map.md @@ -0,0 +1,363 @@ +# Context pack: map (eureka-substrate, 2026-10-08) + +Imagination pass `imagination-context-pack`, session `self-2026-10-08`, for +ruling `eureka-substrate:ruling:context-pack-resolving-anchors`. Evidence: +`F:\Projects\eureka-toolcall-patterns-2026-10-08.md` (Eyes facts) and +`F:\Projects\eureka-context-pack-prior-art-2026-10-08.md` (prior art). +The cut_specs and the question are the typed record; this page holds body +facts, the model page and rationale only. + +Pinned heads: Eureka main `0bb414d92728db61d2d587d749c94970ed2e5582` (also the +head of `hands/toolcall-relief`, whose changes are uncommitted in +`C:\eureka-toolcalls`); CultLib `05e67014`; Huginn `6b496eba`; Epiphany +`4006b14b`; Aetheria `e75083fe`. + +## Body facts + +Scratch: `C:\Users\Meta\AppData\Local\Temp\claude\F--Projects-CultLib\8ee3cab1-a20c-498d-a46b-e503a25ac030\scratchpad\cp\` +(`anchor-map.sh` prototype, `mindview.py`, `mindq.py`, `drift.py`, `all.json`). +This scratchpad is volatile; Hands re-derives from the specs, not from it. + +B1. Diff-hunk line mapping resolves a (base, path, start, end) anchor onto any +later tree with git alone. Prototype `anchor-map.sh` parses `git diff -U0 + -- ` hunk headers: hunks wholly before the range shift it +(MOVED), a hunk overlapping it marks it TOUCHED and widens the range by its net +lines, no hunk is SAME; `git diff -M --name-status` gives rename (RENAMED) and +delete (GONE). Target may be a commit or the working tree (`git diff ` +with no second commit). Dependencies: git and awk; no ctags, rg or tree-sitter. + +B2. Real spec, real drift: `cultmath-tapes:cut_spec:cut-affine-shaders.r2` +(base `a7966142`) mapped onto `05150acd` (the merge the affine-shaders Hands +started from). Result: `CultMath.hlsl:509-510` MOVED to 510-511 (the same two +`#include` lines, checked by `git show`); `GlslMirrorTests.cs:317-319` TOUCHED, +mapped to 336-340, which is the end of `Families` with snoise-tie's two new +families inside it (checked); five anchors SAME. The spec's own prose carried +the workaround: "(GlslMirrorTests.cs:317-319 at a7966142; the lines move with +the merge)". `cut-huginn-persona-state.r2` carries another: "Hands re-takes the +anchor at the merged 13a/13b tip". + +B3. Anchor accuracy at base: the four affine-shaders anchors read at +`a7966142` show exactly the code each `change` names +(`cultmath_iv_snoise_ball(float3, float)` at Interval.hlsl:83-89, the +`InlineData` list at HlslMirrorTests.cs:64-68, the presence asserts at +HlslSourceCompatibilityTests.cs:34-35, the Interval.hlsl lines in +build-unity-package.ps1:86-87). Imagination's line numbers are right at base; +what goes wrong is drift. + +B4. Drift across the whole mind (`drift.py` over `all.json`: 203 cut_specs, +143 cut_reports, 1,260 file_changes). For every spec with a report and a local +repo, each file_change was mapped from `spec.base` to the first report's +`range.base`, the tree Hands started on. 557 anchors: SAME 444 (79.7%), +MOVED 64 (11.5%), TOUCHED 17 (3.1%), path absent at base 32 (5.7%). All 32 +absent paths exist at the report head: they anchor files the cut creates +(none were typos). No base was unreachable. By repo: gamecult-site 55 of 125 +drifted (44%), CultLib 17 of 245, Aetheria 5 of 127, Idunn 3 of 24, Bifrost 1 +of 19, Epiphany, gamecult-ops and Huginn none. So about one anchor in seven was +stale when Hands first read it, concentrated in campaigns whose cuts land in +parallel. + +B5. Worktree, rename, delete and failure probes (scratch clone of Eureka at +`0bb414d`): two lines inserted at 100 and line 185 edited, uncommitted: +`briefs.md:50-60` SAME, `180-190` TOUCHED to 182-192, `300-310` MOVED to +302-312. A staged `git mv` reads SAME+RENAMED; a staged `git rm` reads GONE. +Unreachable base prints `UNRESOLVED base not in this clone: git fetch +origin `; path absent at base prints `UNRESOLVED absent at base`. +Limit: an unstaged delete plus an untracked new file is not detected as a +rename (git's rename detection needs both sides in the index or a commit). +Windows Git Bash prints an LF/CRLF warning per diff on stderr; the tool must +discard stderr of these calls. Line numbers are unaffected by CRLF. + +B6. Cost: rename detection over a whole tree takes about 0.5-1 s per call on +Aetheria (`git diff -M --name-status`), so the drift script, which ran it per +anchor, took over 10 minutes for 557 anchors. The tool runs it once per +invocation and one `-U0` diff per distinct path. + +B7. Symbol by name alone is not an identity. `git grep -W` with git's csharp +driver (supplied without touching the repo through `git -c +core.attributesFile=`) prints the whole enclosing function, but the +probe name `Dispose` matched several members in one file +(`CultMeshContentSessions.cs`), and a bare name has no namespace. A symbol is +usable as a check that the resolved window contains it, and as the key for a +caller list, not as the anchor itself. Git's built-in funcname drivers cover +csharp, rust, python, kotlin and cpp (usable for HLSL/GLSL), and none covers +TypeScript. + +B8. Tool availability. Starfire: Exuberant Ctags 5.8 (Strawberry Perl; no +TypeScript, Kotlin or HLSL), rg (Codex's copy), no tree-sitter, Python 3.13 +via `py -3` (no `tree_sitter` module), git 2.53, gawk, perl. Yggdrasil host: +git 2.47, python3, awk, perl; no ctags, no rg. So git plus the Python standard +library is the only toolset common to both. + +B9. Caller lists: `git grep -n -w -F ` takes about 0.4-0.6 s on CultLib +and Aetheria. `cultmath_iv_snoise_ball` has 20 hits in 8 files, including docs +and fixtures. `Dispose` has 386 hits, so a caller list needs a cap, and above +the cap it needs per-file counts rather than lines (SWE-agent: more than 50 +results returns none and asks for a narrower query). + +B10. Unity YAML: across the three most-modified Aetheria prefabs, fileIDs +(`--- !u! &`) are unique within each file and survive edits +(Djinni.prefab at 122e9292: 440 of 446 ids shared across a 91-hunk commit; +Inventory 116 of 116; Longinus 107 of 107). Diff-hunk mapping works on YAML +as on code. `&` is a good symbol for a YAML anchor. + +B11. The spec can be read without the model copying it. `mindview.py` spawns +`~/.eureka/bin/eureka-state.exe` (the binding in `~/.claude.json` +`mcpServers.eureka-state`: command plus `EUREKA_INSTANCE` and +`HUGINN_ENDPOINT`), speaks MCP JSON-RPC over stdio (initialize, initialized, +tools/call view) and prints the document, in 1.3 s. eureka-state has no CLI +mode of its own (`crates/eureka-state/src/main.rs:298`, stdio MCP only). + +B12. Schema facts for a typed home. `CodeLocation { path: Short, line: u32, +end_line: Option }` (Epiphany `epiphany-pipeline/src/lib.rs:463`) is +shared by cut_spec file_changes, finding and follow_up locations, and mutation +records. The epoch doc (`lib.rs:879-886`) says evolution is additive at +`epiphany.pipeline.epoch.v2`: "a new named field with a serde default". No +`#[serde(default)]` exists in the leaf yet, so the `value_types!` macro has +never carried one. Huginn pins the leaf by git rev +(`crates/huginn-mind/Cargo.toml:22`, `ef956865`), so a field reaches the mind +only after a leaf change, a Huginn re-pin and a daemon plus eureka-state +redeploy. + +B13. The Hands brief already asks for a hand-made pack: "Orient in one script. +One Bash call prints git status, the base and the spec's anchor ranges (`sed +-n`)" (briefs.md:276-278 at 0bb414d). That reads the frozen line numbers, so it +reads the wrong lines for the one anchor in seven that has drifted (B4). + +B14. Prose `path:line` references in spec text (first, adds, keeps_moves, +change): 224 across 99 of 203 specs. Imagination already writes read anchors +inline, untyped (for example "as iv_frustum_ball's FrustumBall check +(GlslMirrorTests.cs:191) does"). + +## Model page + +| Kind | Identity | Lifecycle | Authority | +|---|---|---|---| +| Edit anchor | `(cut_spec id, file_changes index)`; content `(spec.repo, spec.base, path, line, end_line)`. Namespace: the spec's repo at its base commit, which is injective: one path and range at one commit. A short `base` is resolved with `git rev-parse --verify ^{commit}`, and ambiguity is a loud failure. | Created with a spec revision and immutable. Changed only by revision N+1 with a superseding resolution. Its position at a later tree is derived at read time, never stored. A path absent at base is a file the cut creates (B4); it belongs in `adds`, and the tool reports it as NEW. | Imagination writes it (admits the spec). Hands and the tool never rewrite it. A wrong anchor is a deviation in Hands' report and a revision by Imagination. | +| Read anchor (code to see, not to change) | **Empty cell.** It has no typed home. Today it is prose `path:line` (B14) with no namespace (basenames), no base and no check. | None: it is not a document. | None. This is question `context-pack-read-anchors`. | +| Symbol (a check and a caller key) | **Empty cell.** A name is not injective (B7). Its only sound role is a needle that the resolved window must contain, plus a caller-list key. | Would be the anchor's. | Same question. | +| Resolved anchor and window (the expanded pack) | `(spec id, worktree HEAD, working-tree state)`. Derived and never persisted. | Printed into Hands' context. Stale once Hands edits inside it, and re-expanded for edited anchors only with `--since `. No cache, no pack file. | `tools/context-pack.py` derives it. Nothing stores it. A stored pack would be a cache pretending to be truth. | +| The reader tool | `tools/context-pack.py` in GameCult/Eureka. | Lands by Forgejo PR after a Soul pass. Pruned when no brief names it (tools/INDEX.md rule). | Eureka repo. Owns "what Hands sees of a spec's anchors at its tree". It does not own anchors and does not write the mind. | +| Measurement baseline | The Eyes report above and its scripts (`cls.py`, `an2/an3/an4.py`) under this session's scratchpad `eyes-toolcalls\`. | **Volatile.** The scratchpad is session-scoped, so the scripts can vanish before the re-measure. | Self (follow_up `context-pack-measurement`). | + +## Rationale + +**Anchor grammar: base-pinned line range, resolved by diff-hunk mapping.** The +candidates were: symbol paths (`File::Type.Member`), git funcname (`git grep +-W`, `git log -L :fn:file`), ctags, tree-sitter queries, LSP document symbols, +and a text needle plus line hint. Every one of them except the line range needs +a per-language parser, and most need a dependency that is missing on Yggdrasil +or on both hosts (B8). Unity YAML has no symbol grammar in any of them, and a +bare symbol name is not injective (B7). The spec already pins a full +coordinate, `spec.base` plus `path:line-end_line`, and git can carry that +coordinate onto any later tree, the working tree included (B1, B5). This is +the way Gerrit ports review comments across patch sets and GitHub marks a +comment outdated: the anchor is stored where it was made, and diffs carry it +forward. It is language-agnostic, needs only git, and fails loudly on every +path that cannot be carried (base unreachable, path absent, range past EOF, +file deleted). One anchor in seven needed carrying in practice (B4), and the +`sed -n` orient step reads those wrong today (B13). + +**Why the edit anchors are the v1 pack.** file_changes already hold the +planner's localization at base, accurate when mapped (B3), which is +Agentless's and ORACLE-SWE's edit-location handoff. The tool turns them into +SWE-agent-shaped output: one status table first (lost-in-the-middle: the +summary at the top), then windows of at most 100 lines (the SWE-agent ablation +optimum; full-file views lost 5.3 points), then caller lists capped at 50 +lines with per-file counts beyond that (the iterative per-hit search lost 6 +points). An output cap keeps the pack from becoming the 20%-more-steps +AGENTS.md overview (Gloaguen et al.). Re-expansion with `--since` answers the +re-read finding (54%, 1,198 repeats with no edit in between): the edited +windows come back fresh and nothing else is reprinted, as in SWE-agent's +viewer auto-refresh and CORVUS's current-contents registry. + +**Why read anchors and symbols are a question rather than a cut.** Three homes +exist: a typed field (an additive `CodeLocation.symbol` and a `reads` list on +cut_spec, three passes across Epiphany, Huginn plus a deploy, and Eureka, B12); +a `Pack:` line convention in `first` (free, but structure carried in prose); +or a sidecar pack file. The sidecar was rejected outright: it is a second store +with its own lifecycle that goes stale the day a spec is revised. Which of the +first two, and whether now or after measuring v1, decides schema work in the +mind's own leaf. The question records options, evidence and a recommendation. +Nothing in cuts context-pack-tool and context-pack-briefs depends on the +answer. + +**Base for the Eureka cuts.** Both cuts edit `tools/test-tools.sh` and +`references/briefs.md`, which `hands/toolcall-relief` is changing (uncommitted +in `C:\eureka-toolcalls`; its branch head equals main `0bb414d`). The specs pin +`base 0bb414d` and anchor at it. Hands starts from main after that PR merges, +and the first thing context-pack-briefs does is resolve its own briefs.md +anchors through the tool that context-pack-tool lands (the +39 lines shift +them). If the PR has not merged, Hands stops. + +**Subtraction.** Removed: the Hands orient step's `sed -n` of frozen ranges +(briefs.md:276-278) and the "reading a file twice is a smell" advice that the +tool replaces. Imagination's prose drift notes ("the lines move with the +merge", "re-take the anchor") are no longer needed and the map brief says so. +Not added: a parser per language, a cache, a pack file, or a mind field. + +## Read anchors and symbols in the schema (2026-10-08, imagination-read-anchors) + +Imagination pass `imagination-read-anchors`, session `self-2026-10-08`, for +operator ruling `eureka-substrate:ruling:context-pack-read-anchors` +(typed-fields-now). Cuts: `leaf-read-anchors` (Epiphany), `huginn-read-anchors` +(Huginn), `ops-read-anchors-upgrade` (gamecult-ops), `context-pack-reads` +(Eureka). Pinned heads: Epiphany origin/main `ef9568655e75bb6cb362a038e2739a5ac5e777ee` +(the rev Huginn pins; the local `F:\Projects\Epiphany` main is behind it at +`2bf39eef`), Huginn main `6b496eba8c8cfde7b93bb8bc46b1961abca52fd4`, +gamecult-ops main `7e233877dfb21f75d5bbe994641781f718098eac`, Eureka main +`0bb414d92728db61d2d587d749c94970ed2e5582`. Probes ran in detached worktrees +under this session's scratchpad `ra\`, removed afterwards. + +### Body facts + +B15. Stored payload encoding. `PipelineDocument::prepare` calls CultLib +`prepare_entry_named` (cultcache-rs at `8fc74c7`, `lib.rs:2244-2259`), which is +`rmp_serde::to_vec_named`: every value struct is a MessagePack map keyed by +field name. `decode` is `rmp_serde::from_slice` (leaf `lib.rs:745-751`). No +`deny_unknown_fields` exists in the leaf. Consequences: (a) an old document +lacking a new key decodes when the field is `Option` or carries +`#[serde(default)]`; (b) an old daemon reading a new document ignores the +unknown key, so after a rollback it serves such documents without the field +while the stored bytes keep it. + +B16. Re-encoding is load-bearing. Admission prepares every write afresh +(`admission.rs:143-157`) and the receipt digest is over those payloads +(`receipt.rs:123-126`); A9 compares it with stored receipts to answer +`AlreadyAdmitted` (`admission.rs:164-166`). So if a document without the new +fields encoded differently after the change (for example `symbol: null` or +`reads: []` written out), replaying any pre-upgrade batch would stop answering +`AlreadyAdmitted`, and target invariant `stored-documents-valid` ("No change +re-keys, re-encodes or invalidates a document already in the mind") would +break. Hence `skip_serializing_if` on both fields, not only `serde(default)`. + +B17. The macro. `value_types!` (`lib.rs:415-436`) emits a struct whose fields +carry only an optional `#[schemars(extend("maxItems"))]`; it has no way to put +a serde attribute on a field. `Bounded for Option` exists (`lib.rs:79-83`); +`Vec` is validated only through the `[max]` list form (`list`, `lib.rs:110`). +`=> extra` is a struct-level validation hook (`campaign_repos_are_distinct`, +`lib.rs:452-468`). Refusal variants are pinned at five by the schema test +(`lib.rs:2717`), so new bounds must reuse `FieldBound` and `InvalidFormat`. + +B18. Who constructs the changed structs. Epiphany: `CodeLocation {` 3 sites and +`PipelineCutSpec {` 1 test sample, all in `lib.rs`; no other Epiphany crate +builds them. Huginn: `CodeLocation {` in `huginn-mind/src/fixtures.rs:88`, +`huginn-daemon/src/daemon.rs:360`, `eureka-state/tests/client.rs:59`; +`PipelineCutSpec {` in `fixtures.rs:192`, `daemon.rs:361`, `client.rs:44` +(`admission.rs:432` uses `..spec.clone()` and needs no edit). Published schemas +that embed `CodeLocation`: Epiphany `schemas/cultnet/epiphany.pipeline.{cut_spec,cut_report,finding,follow_up}.v2.schema.json`; +Huginn `schemas/cultnet/huginn.mind_{request,response}.v1.schema.json`, pinned +byte for byte by `wire.rs:276-280`. The leaf's own check is +`pipeline_published_schemas_match_derivation` (`lib.rs:2723`), which writes the +derived copies to a temp dir on failure. + +B19. Size calibration. `daemon.rs:349-356`'s `cut_spec` claims "every list the +leaf bounds filled to its maximum"; `client.rs:220-236` calibrates `fits` (180 +file changes) under `MAX_RESPONSE_BYTES` and `wide` (256) over it. Filling the +new fields in the daemon builder keeps its claim true; filling them in the +client builder would move the calibration. + +B20. Admission has no tree. Huginn holds no repository checkout; admission +checks bounds, formats, keys and cross-document rules only. "The symbol is in +the window at base" cannot be an admission rule. Its owner is the reader tool, +run by Imagination at base after admitting (cut context-pack-briefs) and by +Hands at its tree. + +B21. Index text. `huginn-mind/src/index.rs:42-70` indexes a cut_spec's title, +first, adds and keeps_moves only, not file_changes. Leaving `reads` out keeps +the index text unchanged, so no reindex follows the upgrade. + +B22. Deploy owner. Huginn is not an Idunn target: `runbooks/huginn-yggdrasil.md` +(gamecult-ops) calls it "an interim deployment: a hand-installed systemd unit", +retired by Idunn only in Huginn's 14-I, and says "No brake gates deployment of +this unit". Target r2 names the same runbook as the deploy's canonical +implementation. The live release is `releases/107552dc4b27b8856b043616ae74c2ebe7908b66` +(read-only `readlink` on Yggdrasil, unit active); Huginn main since then changes +docs only (`git diff --stat 107552dc..6b496eba -- crates Cargo.toml Cargo.lock` +is empty). The client is `C:\Users\Meta\.eureka\bin\eureka-state.exe`, whose +MCP `admit` input schema is derived at run time from the leaf types through +schemars (`eureka-state/src/main.rs:73`), so a rebuilt client publishes the new +fields with no hand edit; an agent session sees them only after it starts +anew. The runbook's rollback paragraph (`:109-113`) is specific to the 64-byte +key widening. + +B23. The harvest. Target r2's canonical implementations put the leaf at Huginn +`crates/eureka-pipeline`, and Huginn `docs/eureka-self-cut.md:435` (harvest H1) +copies it there "from Epiphany head `4006b14b`", which is 5 commits behind +`ef956865`. H1 has no cut_spec yet (eureka-body has only questions and +rulings). If H1 copied from `4006b14b` after this change merges, it would drop +both leaf-key-bound and the new fields. + +### Rationale + +**Where the fields land.** The leaf still lives in Epiphany and Huginn pins it +by git rev, so the field goes into Epiphany's leaf, then Huginn re-pins, as the +ruling's option text names. Waiting for H1 would make this one Huginn cut +instead of two, but H1 is unscheduled. The operator accepted the three-pass +cost in her words. Follow-up `leaf-harvest-copies-head` records the constraint +on H1. If H1 merges before `leaf-read-anchors` starts, Self stops the cut and +Imagination re-maps it onto `crates/eureka-pipeline` (one Huginn cut, no +re-pin). + +**Shape.** `CodeLocation.symbol: Option` everywhere a location is (file +changes, findings, follow-ups, mutations). It means the same thing in every +one: a name the location's lines must contain. `PipelineCutSpec.reads: +Vec[16]` with `ReadAnchor { location: CodeLocation, why: Short }`. +It is a distinct type, not `FileChange`, because a read changes nothing and a +field named `change` would lie. Reads are in the spec's repo at its base, like +file_changes. A read in another repo stays prose in `first`, because a read +cannot pin another repo's commit. `Symbol` is a new bounded text: 1-128 bytes, +no whitespace or control character. It is new, so no stored document can fail +it. A read's window is bounded by an `=> extra` hook on `ReadAnchor`: `line >= +1`, `end_line >= line` (`InvalidFormat`), and span at most 100 lines +(`FieldBound`, limit 100). 100 is the window cap the tool already uses. No new +bound touches `CodeLocation`'s existing fields, because stored findings would +then stop validating. + +**The macro owns "additive".** `value_types!` gains a per-field marker written +after the type and any list maximum, for example `symbol: Option = +absent` and `reads: Vec[16] = absent`. It emits +`#[serde(default, skip_serializing_if = "Absent::absent")]`. Trait `Absent` is +implemented for `Option` and `Vec` only, and the validate expansion +names `<$ty as Absent>::$marker`, so a marker word other than `absent`, or the +marker on any other type, is a compile error. The epoch doc and the schemas +README then say this marker is how a field is added at v2. This replaces the +hope that every future author remembers both attributes (B16). + +**No admission rule in Huginn.** Bounds are the leaf's, and Huginn applies them +through `validate` on every write. The symbol-in-window check needs a tree +(B20) and belongs to `context-pack.py`, which exits nonzero on +`SYMBOL_MISSING`. + +**Deploy.** The owner is the gamecult-ops runbook's Upgrade section. It is not +Idunn: Idunn does not own this unit, and no brake applies (B22). The ops cut +generalises the rollback paragraph for additive fields and then executes the +Upgrade steps. One addition: the read-back test also asserts that every stored +document re-prepares to its stored payload bytes. That checks B16 on the live +mind, not only in a unit test. + +**Eureka.** One new cut after both v1 cuts and the deploy. The v1 specs stay as +they are, because v1 must not wait on the deploy. + +**Subtraction.** Removed: prose `path:line` read references in specs for the +spec's own repo (224 today, B14), replaced by checked reads; Imagination's +"re-take the anchor" notes; Hands' first-pass grep for callers of named +symbols. Added: one trait, one text type, one struct, one macro marker, two +fields, one tool section. Not added: an admission rule, a Huginn type, an +index change, or a second store. + +B24. The admit door drops unknown keys silently. The batch that admitted these +specs had one file_change location with a stray `"symbol": null`. It was +admitted as Committed (receipt `mind-commit-c528cc24...`), and a view of +`cut-context-pack-reads.r1` shows that location without the key. This is B15(b) +at the JSON door: a session that started before the client swap will drop +`reads` and `symbol` and still report success. Follow-up +`gap-admit-drops-unknown-keys` (owner: Huginn eureka-state). Meanwhile the ops +cut makes agents restart, and context-pack-reads stops if its admit schema +lacks `reads`. + +Admitted (receipts `mind-commit-c528cc24...`, `mind-commit-5d97a6ad...`): +`eureka-substrate:cut_spec:cut-leaf-read-anchors.r1`, +`cut-huginn-read-anchors.r1`, `cut-ops-read-anchors-upgrade.r1`, +`cut-context-pack-reads.r1`; follow-ups `leaf-harvest-copies-head` and +`gap-admit-drops-unknown-keys`. Order: leaf, then Huginn, then ops (deploy), +then Eureka. The Eureka cut also waits for both v1 cuts.