Map: Ghostlight verse phase 1, body facts and model page
Why the 2026-09-30 deploy died (Odin's route), what main's build and tests say, the play-session findings against main, the live refresh loop, the host's retired v1 body, and where phase 2's verb sets plug in. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
parent
1e5761d9e0
commit
46df4070a8
1 changed files with 258 additions and 0 deletions
258
notes/ghostlight-verse-map.md
Normal file
258
notes/ghostlight-verse-map.md
Normal file
|
|
@ -0,0 +1,258 @@
|
|||
# Ghostlight Verse: Map
|
||||
|
||||
Campaign `ghostlight-verse`, target `ghostlight-verse:target:r2`
|
||||
(`notes/ghostlight-verse-target.md`). This page holds body facts, the model
|
||||
page and rationale only. Cut specs, questions, rulings and follow-ups live in
|
||||
the Eureka mind; query them there, never copy them here.
|
||||
|
||||
Phase 1 map: Imagination, session `self-2026-10-09-morning`, 2026-10-09.
|
||||
Base for every phase-1 anchor: Ghostlight `origin/main` `4fee97f`.
|
||||
Times are UTC unless marked local (UTC+2).
|
||||
|
||||
## Body facts (probed 2026-10-09)
|
||||
|
||||
### Does main run on Yggdrasil?
|
||||
|
||||
- **The code on `main` is running.** The live unit
|
||||
`idunn-ghostlight-dda9de02…` (PID 818485, up since 2026-09-23 20:02:48) runs
|
||||
release `sha256-51b014be…`, whose binary embeds commit `454fc33` (grep of
|
||||
the binary for the stamped commit: `454fc33` 5 hits, every later commit 0).
|
||||
`git diff --stat 454fc33 origin/main` touches only
|
||||
`docs/architecture/ghostlight-play-agent-cut.md`,
|
||||
`notes/fresh-workspace-handoff.md` and `state/map.yaml`. So `main`'s code is
|
||||
live; only its provenance stamp is a week old.
|
||||
- **Why the 2026-09-30 deploy died.** Unit `idunn-ghostlight-0e4f20f5…`
|
||||
(release `sha256-ec17cae9…`, binary stamped `4fee97f`) logged
|
||||
`publishing initial Warming runtime presence failed; retrying error=timed
|
||||
out connecting runtime-presence publisher to 10.77.0.1:17871` every ~2.5 s
|
||||
from 10:58:35, then at 11:00:36 `Error: publishing initial Warming runtime
|
||||
presence timed out` and exit 1 (`journalctl -u` on the unit). The bound is
|
||||
`WRITE_LEASE_WAIT_TIMEOUT` = 120 s
|
||||
(`crates/ghostlight-dungeon/src/runtime.rs:1939`, used at `:1985-1992`).
|
||||
`10.77.0.1:17871` is nginx's UDP stream route to Odin's catalog
|
||||
(`/etc/nginx/idunn-stream-routes/odin.conf`: listen `10.77.0.1:17871`,
|
||||
upstream `127.0.0.1:17973`, the `odin-daemon` socket). Idunn itself could not
|
||||
reach it in the same minutes: `Idunn rejected admitted odin route
|
||||
continuity: timed out connecting CultMesh RUDP client idunn-route-observer
|
||||
to 10.77.0.1:17871` (idunn-yggdrasil journal, 10:57-11:08). Idunn recorded
|
||||
the transaction `up-855ddf0c…` as failed and kept the 09-23 generation:
|
||||
daemon survival worked as designed.
|
||||
- **The Odin route outage window.** Idunn's "odin route continuity"
|
||||
rejections per hour run from 2026-09-28 01:19 to 2026-09-30 12:xx, peaking
|
||||
at 1,326 in 2026-09-29 23:00, and stop after that. The live Ghostlight's
|
||||
failed presence republications per day: 3,186 (09-28), 3,793 (09-29),
|
||||
2,512 (09-30), then 1-7 per day from 10-01. The peak matches the scar in
|
||||
Eureka's `tools/stopgap/ygg-verify.sh` header: at 8 verify slots on
|
||||
2026-09-30 01:50 local, `/proc/pressure/io` full avg60 was 41% and Odin's
|
||||
fsyncs stalled long enough for Idunn's route challenges to time out. That
|
||||
explains the peak; the start on 09-28 is not proven to share the cause.
|
||||
Odin's unit failures in the window include `RUDP packet connection id …
|
||||
does not match` and `timed out waiting for reliable acknowledgement from
|
||||
CultMesh RUDP catalog 127.0.0.1:17973` (journal of `idunn-odin-*`).
|
||||
- **Consequence for deploys.** Ghostlight cannot start without Odin: the
|
||||
Warming presence and the process write lease both go through Odin
|
||||
(`runtime.rs:1974-2003`). That coupling is the lease design, not a defect
|
||||
in this campaign's scope. A deploy is preflighted on Odin's route health
|
||||
and on Yggdrasil's IO pressure.
|
||||
|
||||
### Does main build and pass?
|
||||
|
||||
Two jobs through `ygg-verify.sh` at `4fee97f`, with `vendor/eve` initialised:
|
||||
|
||||
- Rust (image `eureka-verify-rust`), the recipe's own steps:
|
||||
`ghostlight-persona-projection` 35 passed; `ghostlight` lib 621 passed,
|
||||
1 ignored (`real_local_model_cognition_modes_commit_speech`, needs a model);
|
||||
`tests/external_admission.rs` 13 passed; doc-tests 10 passed;
|
||||
`ghostlight-dungeon --bin` 198 passed, 4 ignored; release build green in
|
||||
3 m 08 s. Total 877 passed, 0 failed. Warnings: unused imports
|
||||
(`RECORD_GAP_PATCH_TOOL`, `WorldPhase`, `DEFAULT_LOCAL_MODEL_PREFIX`,
|
||||
`DateTime`).
|
||||
- Node (image `eureka-verify-kotlin`, Node 24.21): `sidecar/claude-sdk`
|
||||
27 passed, 3 skipped ("run `npm run build` first"); `web` 3 passed;
|
||||
`web` build green.
|
||||
- **The four ignored Dungeon tests are the Eve client-bridge tests**
|
||||
(`runtime.rs:3054`, `:3677`, `:3798`, `:3904`), the only tests that drive
|
||||
the real vendored browser lowering. They need Node beside cargo. The rust
|
||||
verify image has no `node` (probe: `command -v node` exit 127), and the
|
||||
recipe runs them in no step, so the release gate never runs them. PA.f214
|
||||
is the class of defect they exist to catch.
|
||||
|
||||
### The play-session findings against main today
|
||||
|
||||
No commit after `454fc33` touches code, so every PA.f211-PA.f216 is open on
|
||||
`main`. Mechanisms, probed:
|
||||
|
||||
- **PA.f211 and the tunnel.** Bonsai's endpoint on Yggdrasil is the reverse
|
||||
tunnel `127.0.0.1:18080`. With Bonsai down (as now), `curl` to it returns an
|
||||
empty reply (exit 52): sshd accepts, then closes. `local_inference.rs:166-181`
|
||||
classifies only `error.is_connect()` as retryable, so a Bonsai restart reaches
|
||||
the play loop as `InferenceFault::new` (not retryable) and closes the turn on
|
||||
the first call. Even a retryable fault gets
|
||||
`backoff_delay` (`play.rs:2970-2973`, base 100 ms, cap 5 s) times
|
||||
`ROUND_RETRY_BUDGET` 12 (`play.rs:89`), about 36 s in all, against a reload
|
||||
the Raven runbook puts at about 30 s plus load. Raven's
|
||||
`/opt/gamecult/bonsai/rss-watchdog.sh` (systemd `bonsai-rss.timer`, every
|
||||
2 min) restarts past 4.5 GiB RSS unless `/slots` shows `is_processing`;
|
||||
between the calls of a turn nothing is processing. Raven's
|
||||
`idle-reaper.sh` (`bonsai-idle.timer`) stops Bonsai and SillyTavern after
|
||||
30 minutes with no `launch_slot_` in the container log. Neither script is in
|
||||
version control: gamecult-ops has only the runbook describing them.
|
||||
- **PA.f212.** A closed turn records its fault (`PlayTurn.fault`,
|
||||
`play.rs:298`; `close_with_fault`, `play.rs:1753-1765`), but
|
||||
`PlayTurnView` (`play.rs:1224-1246`) has no fault field, the card
|
||||
(`eve.rs:455-543`) renders narration, question, refusal, running and
|
||||
resolving rows and nothing for a fault, and `close_with_fault` logs nothing.
|
||||
A local call can also wait `RESPONSE_TIMEOUT` = 900 s
|
||||
(`controllers.rs:68`, used by `local_inference.rs:157`) with "Resolving your
|
||||
turn" on the card the whole time.
|
||||
- **PA.f213.** The owner's Draft card always offers "Seed the world"
|
||||
(`eve.rs:366-399`); the handler refuses when `GHOSTLIGHT_SEED_VAULT_ROOT` is
|
||||
unset (`runtime.rs:1701`, `:1739-1745`). Yggdrasil sets none.
|
||||
- **PA.f214.** `world.play.text` (`eve.rs:588-594`) carries no authored
|
||||
`value`. The vendored lowering captures only `value`, never `placeholder`
|
||||
(the PA.f164 comment at `eve.rs:288-296`), so Play clicked on an untouched
|
||||
box sends no `text`, and `PlayPayload.text` (`runtime.rs:288-292`) is
|
||||
required. The "submit an empty message to continue" hint (`eve.rs:514-520`)
|
||||
asks the player for exactly that payload.
|
||||
- **PA.f215.** The acceptance step runs
|
||||
`controllers::tests::real_local_model_cognition_modes_commit_speech`
|
||||
(`controllers.rs:11231`): NarrativePersona and OperationalAgent each must
|
||||
commit speech in one shot against Bonsai. It exercises the library's
|
||||
controller path, not the Dungeon play loop. Failing runs' prose is written to
|
||||
`GHOSTLIGHT_ACCEPTANCE_PERSONA_PROSE_LOG`; Idunn's staging dirs on Yggdrasil
|
||||
hold `.runner-rust-acceptance` workspaces per transaction.
|
||||
- **PA.f216.** Creation is one form of eight fields, three of them raw JSON
|
||||
(`targets` `{}`, `jurisdictions` `[]`, lens weights; `eve.rs:244-331`), then
|
||||
a Draft card with Approve, Seed and Activate as separate buttons
|
||||
(`eve.rs:343-415`).
|
||||
|
||||
### A live defect not yet in the findings
|
||||
|
||||
The live process has logged `Heimdall refresh transport unavailable
|
||||
error=Heimdall denied the private command: Idempotency key was reused with
|
||||
different command content.` once a minute since 2026-09-23 21:04 (1,440 a
|
||||
day from 10-01). Sequence: 21:02:03 discovery timed out; 21:03:00 `local app
|
||||
session rejected Heimdall refresh error=local session changed during refresh`;
|
||||
from 21:04 the denial forever. Mechanism:
|
||||
|
||||
- Ghostlight's refresh key is `refresh:{heimdall_session_id}:{access_revision}`
|
||||
(`runtime.rs:1877-1880`), stable until the local session advances.
|
||||
- Every attempt is resealed with a fresh nonce, IV and issue time
|
||||
(`heimdall.rs:880-895`).
|
||||
- Heimdall fingerprints the sealed request bytes, not the command
|
||||
(`Heimdall/src/private-command-plane.ts:88`,
|
||||
`createHash("sha256").update(request.payload)`), and refuses a known key with
|
||||
a different fingerprint (`:93`).
|
||||
- So once Heimdall has executed one refresh (it rotated the token at 21:03)
|
||||
and the local commit refused it (`app_session.rs:293-300`), no retry under
|
||||
that key can ever succeed, and `sessions_due_for_refresh`
|
||||
(`app_session.rs:247-268`) keeps offering the same candidate while
|
||||
`refresh_expires_at > now`. Which clause of `app_session.rs:293-299` fired at
|
||||
21:03 is not logged.
|
||||
|
||||
### The host
|
||||
|
||||
- `ghostlight-dungeon.service` (the v1 body) is **enabled**, inactive since
|
||||
2026-09-23 14:31. It binds `127.0.0.1:8831`, runs `/srv/ghostlight/current`,
|
||||
targets CodexConnector, and would start at the next boot beside Idunn's
|
||||
generation. `/srv/ghostlight` holds 46 GiB (226 releases, acceptance
|
||||
build scripts), `/var/lib/gamecult/ghostlight-dungeon` 17 GiB of v1 world
|
||||
state (the v2 kernel refuses earlier schemas; nothing migrates it),
|
||||
`ghostlight-dungeon-quarantine` 44 KiB, `ghostlight-import-20260821T1858Z`
|
||||
8 KiB. Disk `/` is 20% of 2 TB used.
|
||||
- The v1 actuator and its tests live in gamecult-ops:
|
||||
`scripts/deploy-ghostlight-yggdrasil.sh`,
|
||||
`scripts/test-ghostlight-yggdrasil-wiring.sh`; the runbook's "One-time state
|
||||
cut" section (`runbooks/ghostlight-dungeon-yggdrasil.md:456-499`) describes
|
||||
that path.
|
||||
- Idunn's leftovers: 12 failed `idunn-ghostlight-*` transient units (11 from
|
||||
the 09-23 bring-up, 1 from 09-30), 15 release dirs under
|
||||
`/srv/ghostlight-idunn/releases` (403 MiB), per-transaction staging dirs
|
||||
under `/var/lib/gamecult/idunn/staging`. These are Idunn's: it launches the
|
||||
units and owns release retention. Phase 1 does not hand-clean them.
|
||||
- Hostname is `yggdrasil-candidate`. Heimdall still runs as the legacy
|
||||
`heimdall.service`; every Idunn v2 deploy of it has failed (`idunn status`).
|
||||
|
||||
### The repo
|
||||
|
||||
- The pre-rebuild research machine is still on `main`: `tools/` (17 files,
|
||||
4,146 lines, including the Python state store), `schemas/` (14, 4,057),
|
||||
`examples/` (194, 37,421), `experiments/` (172, 12,325), `acceptance/`,
|
||||
`prompts/`, `scripts/`, `output/`, `assets/newspaper/`, `state/` (map.yaml,
|
||||
evidence ledger and archive, a CultCache JSONL, corpus coverage, branches),
|
||||
`docs/aetheria/`, root `package.json` scripts pointing at a Codex Python
|
||||
runtime, and about 30 pre-rebuild architecture and note pages. No file
|
||||
outside Ghostlight references `Ghostlight/schemas`,
|
||||
`ghostlight_state_store` or `ghostlight.agent_state` (grep over
|
||||
`F:\Projects`, excluding build output).
|
||||
- `AGENTS.md` (234 lines) instructs agents to treat `state/` and the Python
|
||||
store as canonical state and the implementation plan as current.
|
||||
- Live docs that cite the old `world/*.rs` paths:
|
||||
`ghostlight-world-consumer-api.md:19,31,33,38,70`;
|
||||
`ghostlight-world-ontology.md:53,74,167,202,203,215,232,277`;
|
||||
`notes/fresh-workspace-handoff.md:253`; `notes/local-live-smoke.md:56`. The
|
||||
code is flat under `crates/ghostlight/src/`. Dated cut documents and
|
||||
postmortems keep their paths: they are history.
|
||||
|
||||
## Model page
|
||||
|
||||
One row per persistent kind phase 1 touches. No cell may be empty.
|
||||
|
||||
| Kind | Named by | Life over time | Who decides |
|
||||
|---|---|---|---|
|
||||
| World state and commit journal (`.cc`, redb) | `WorldId`; `revision`; digest chain | created Draft, approved, activated; one commit row per admitted command; earlier schemas refused, never migrated | the kernel, through `WorldMailbox`'s closed operation set; untouched in phase 1 |
|
||||
| Play turn row (play-turn store `.cc`) | `turn_id` (UUID); per-turn `question_token` | `Running` → `AwaitingPlayer` ↔ `Running` → `Closed` (narration or `fault`); an unreadable row is retired to a sidecar and play starts fresh | `PlayTable` (`play.rs`); the card is a projection of `PlayTurnView`, which phase 1 extends with the fault |
|
||||
| App session (`app-sessions-v2.cc`) | cookie hash; `heimdall_session_id` + `access_revision` | issued at sign-in; refreshed when access is near expiry; revoked on logout, custody change, invalid receipt, and (phase 1) on a refresh that can no longer succeed | `AppSessionOwner`; Heimdall decides the upstream session and rotates refresh tokens |
|
||||
| Heimdall private command receipt (Heimdall Postgres) | `(app_slug, idempotency_key)` + request fingerprint | written once per executed command; replayed for an identical request; a different request under the key is refused | Heimdall; its fingerprint rule is a Heimdall follow-up, not a phase-1 cut |
|
||||
| Idunn generation (transient unit, release dir) | `sha256-<release>` | staged → started → Warming → lease → Active → superseded or failed; the previous Active stays until a new one is Ready | Idunn; retention and failed-unit collection are Idunn's follow-up |
|
||||
| v1 body (`ghostlight-dungeon.service`, `/srv/ghostlight`, v1 state) | unit name; paths | retired in fact 2026-09-23; unit still enabled | nobody today; phase 1 assigns its removal to a gamecult-ops cut, data by operator ruling |
|
||||
| Bonsai lifecycle (Raven containers) | container `bonsai`; scripts under `/opt/gamecult/bonsai` | started by hand ("Start AI Chat"); restarted by the RSS watchdog; stopped by the idle reaper | gamecult-ops/Raven; Dungeon only reads the endpoint; availability policy is an operator question |
|
||||
| Repository docs (README, AGENTS, live architecture pages, handoff) | path on `main` | rewritten to the live machine; pre-rebuild material leaves `main` by operator ruling | the Ghostlight repo; history in git, dated cut documents and postmortems |
|
||||
|
||||
### Where phase 2's verb sets plug in
|
||||
|
||||
Ruling `verse-verbs-two-render-paths` and invariant `every-event-renderable`
|
||||
bound what the kernel may admit to two render paths. Today:
|
||||
|
||||
- The thirty operations (`ghostlight-world-ontology.md:451-478`) are
|
||||
state transitions, not verbs. They stay the closed substrate.
|
||||
- What an actor *does* is an affordance: a world-authored catalog entry whose
|
||||
kind is `AffordanceKindName` (`patch.rs:247-254`), free canonical text that
|
||||
"the kernel carries and branches on nowhere", plus the kernel-built `speak`
|
||||
(`controllers.rs:94`). Admitted acts surface as `DecisionEvents` on the
|
||||
commit.
|
||||
|
||||
So phase 2's two verb sets plug in at the affordance kind: a closed
|
||||
vocabulary of kinds, each tagged with its one render path, with effects that
|
||||
still compile to the thirty operations. Today the Play authority can declare
|
||||
any affordance kind, so nothing on `main` enforces the invariant yet. Phase 1
|
||||
neither widens nor narrows that, and its docs must not describe open-ended
|
||||
verbs as the target.
|
||||
|
||||
## Rationale
|
||||
|
||||
- **"main dies" was a misreading.** The deploy that died carried no code
|
||||
change, and it died on Odin's route, not on Ghostlight. So phase 1's deploy
|
||||
work is mostly preflight and proof: redeploy `main` after the play fixes,
|
||||
with Odin's route and IO pressure checked first. The Odin outage is
|
||||
recorded against its owner.
|
||||
- **Subtraction goes first and alone.** The pre-rebuild tree is about 60,000
|
||||
lines that the live machine never reads. It is cut before the docs are
|
||||
rewritten, so the rewrite describes only what is left. Parking it at a tag
|
||||
rather than deleting it outright is the operator's call (question).
|
||||
- **Faults belong to the card, not the log alone.** PA.f212 is an authority
|
||||
gap: the turn store knew the fault and the projection dropped it. The fix
|
||||
extends the projection; it adds no second record.
|
||||
- **Retrying inference is safe.** A model call is a proposal the kernel
|
||||
admits or refuses, so a transport fault before any reply is retryable. That
|
||||
is the Ghostlight half of PA.f211. The Raven half (the watchdog's quiet
|
||||
window) belongs to gamecult-ops, which also takes the scripts into version
|
||||
control.
|
||||
- **The refresh loop is a contract split.** Ghostlight assumes a stable key
|
||||
makes a retry safe; Heimdall's fingerprint makes every retry new. Ghostlight
|
||||
stops looping (a refresh that can no longer succeed revokes the local
|
||||
session). Heimdall's fingerprint rule goes to Heimdall as a follow-up.
|
||||
- **The bridge tests are the gate's blind spot.** They are the only
|
||||
Dungeon tests that drive the real client, and nothing runs them. Fixing the
|
||||
gate means a runner with both Rust and Node, which is Idunn's recipe and
|
||||
runner surface: a follow-up, with each phase-1 Hands run doing them by hand.
|
||||
Loading…
Reference in a new issue