Why the 2026-09-30 deploy died (Odin's route), what main's build and tests say, the play-session findings against main, the live refresh loop, the host's retired v1 body, and where phase 2's verb sets plug in. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
16 KiB
Ghostlight Verse: Map
Campaign ghostlight-verse, target ghostlight-verse:target:r2
(notes/ghostlight-verse-target.md). This page holds body facts, the model
page and rationale only. Cut specs, questions, rulings and follow-ups live in
the Eureka mind; query them there, never copy them here.
Phase 1 map: Imagination, session self-2026-10-09-morning, 2026-10-09.
Base for every phase-1 anchor: Ghostlight origin/main 4fee97f.
Times are UTC unless marked local (UTC+2).
Body facts (probed 2026-10-09)
Does main run on Yggdrasil?
- The code on
mainis running. The live unitidunn-ghostlight-dda9de02…(PID 818485, up since 2026-09-23 20:02:48) runs releasesha256-51b014be…, whose binary embeds commit454fc33(grep of the binary for the stamped commit:454fc335 hits, every later commit 0).git diff --stat 454fc33 origin/maintouches onlydocs/architecture/ghostlight-play-agent-cut.md,notes/fresh-workspace-handoff.mdandstate/map.yaml. Somain's code is live; only its provenance stamp is a week old. - Why the 2026-09-30 deploy died. Unit
idunn-ghostlight-0e4f20f5…(releasesha256-ec17cae9…, binary stamped4fee97f) loggedpublishing initial Warming runtime presence failed; retrying error=timed out connecting runtime-presence publisher to 10.77.0.1:17871every ~2.5 s from 10:58:35, then at 11:00:36Error: publishing initial Warming runtime presence timed outand exit 1 (journalctl -uon the unit). The bound isWRITE_LEASE_WAIT_TIMEOUT= 120 s (crates/ghostlight-dungeon/src/runtime.rs:1939, used at:1985-1992).10.77.0.1:17871is nginx's UDP stream route to Odin's catalog (/etc/nginx/idunn-stream-routes/odin.conf: listen10.77.0.1:17871, upstream127.0.0.1:17973, theodin-daemonsocket). Idunn itself could not reach it in the same minutes:Idunn rejected admitted odin route continuity: timed out connecting CultMesh RUDP client idunn-route-observer to 10.77.0.1:17871(idunn-yggdrasil journal, 10:57-11:08). Idunn recorded the transactionup-855ddf0c…as failed and kept the 09-23 generation: daemon survival worked as designed. - The Odin route outage window. Idunn's "odin route continuity"
rejections per hour run from 2026-09-28 01:19 to 2026-09-30 12:xx, peaking
at 1,326 in 2026-09-29 23:00, and stop after that. The live Ghostlight's
failed presence republications per day: 3,186 (09-28), 3,793 (09-29),
2,512 (09-30), then 1-7 per day from 10-01. The peak matches the scar in
Eureka's
tools/stopgap/ygg-verify.shheader: at 8 verify slots on 2026-09-30 01:50 local,/proc/pressure/iofull avg60 was 41% and Odin's fsyncs stalled long enough for Idunn's route challenges to time out. That explains the peak; the start on 09-28 is not proven to share the cause. Odin's unit failures in the window includeRUDP packet connection id … does not matchandtimed out waiting for reliable acknowledgement from CultMesh RUDP catalog 127.0.0.1:17973(journal ofidunn-odin-*). - Consequence for deploys. Ghostlight cannot start without Odin: the
Warming presence and the process write lease both go through Odin
(
runtime.rs:1974-2003). That coupling is the lease design, not a defect in this campaign's scope. A deploy is preflighted on Odin's route health and on Yggdrasil's IO pressure.
Does main build and pass?
Two jobs through ygg-verify.sh at 4fee97f, with vendor/eve initialised:
- Rust (image
eureka-verify-rust), the recipe's own steps:ghostlight-persona-projection35 passed;ghostlightlib 621 passed, 1 ignored (real_local_model_cognition_modes_commit_speech, needs a model);tests/external_admission.rs13 passed; doc-tests 10 passed;ghostlight-dungeon --bin198 passed, 4 ignored; release build green in 3 m 08 s. Total 877 passed, 0 failed. Warnings: unused imports (RECORD_GAP_PATCH_TOOL,WorldPhase,DEFAULT_LOCAL_MODEL_PREFIX,DateTime). - Node (image
eureka-verify-kotlin, Node 24.21):sidecar/claude-sdk27 passed, 3 skipped ("runnpm run buildfirst");web3 passed;webbuild green. - The four ignored Dungeon tests are the Eve client-bridge tests
(
runtime.rs:3054,:3677,:3798,:3904), the only tests that drive the real vendored browser lowering. They need Node beside cargo. The rust verify image has nonode(probe:command -v nodeexit 127), and the recipe runs them in no step, so the release gate never runs them. PA.f214 is the class of defect they exist to catch.
The play-session findings against main today
No commit after 454fc33 touches code, so every PA.f211-PA.f216 is open on
main. Mechanisms, probed:
- PA.f211 and the tunnel. Bonsai's endpoint on Yggdrasil is the reverse
tunnel
127.0.0.1:18080. With Bonsai down (as now),curlto it returns an empty reply (exit 52): sshd accepts, then closes.local_inference.rs:166-181classifies onlyerror.is_connect()as retryable, so a Bonsai restart reaches the play loop asInferenceFault::new(not retryable) and closes the turn on the first call. Even a retryable fault getsbackoff_delay(play.rs:2970-2973, base 100 ms, cap 5 s) timesROUND_RETRY_BUDGET12 (play.rs:89), about 36 s in all, against a reload the Raven runbook puts at about 30 s plus load. Raven's/opt/gamecult/bonsai/rss-watchdog.sh(systemdbonsai-rss.timer, every 2 min) restarts past 4.5 GiB RSS unless/slotsshowsis_processing; between the calls of a turn nothing is processing. Raven'sidle-reaper.sh(bonsai-idle.timer) stops Bonsai and SillyTavern after 30 minutes with nolaunch_slot_in the container log. Neither script is in version control: gamecult-ops has only the runbook describing them. - PA.f212. A closed turn records its fault (
PlayTurn.fault,play.rs:298;close_with_fault,play.rs:1753-1765), butPlayTurnView(play.rs:1224-1246) has no fault field, the card (eve.rs:455-543) renders narration, question, refusal, running and resolving rows and nothing for a fault, andclose_with_faultlogs nothing. A local call can also waitRESPONSE_TIMEOUT= 900 s (controllers.rs:68, used bylocal_inference.rs:157) with "Resolving your turn" on the card the whole time. - PA.f213. The owner's Draft card always offers "Seed the world"
(
eve.rs:366-399); the handler refuses whenGHOSTLIGHT_SEED_VAULT_ROOTis unset (runtime.rs:1701,:1739-1745). Yggdrasil sets none. - PA.f214.
world.play.text(eve.rs:588-594) carries no authoredvalue. The vendored lowering captures onlyvalue, neverplaceholder(the PA.f164 comment ateve.rs:288-296), so Play clicked on an untouched box sends notext, andPlayPayload.text(runtime.rs:288-292) is required. The "submit an empty message to continue" hint (eve.rs:514-520) asks the player for exactly that payload. - PA.f215. The acceptance step runs
controllers::tests::real_local_model_cognition_modes_commit_speech(controllers.rs:11231): NarrativePersona and OperationalAgent each must commit speech in one shot against Bonsai. It exercises the library's controller path, not the Dungeon play loop. Failing runs' prose is written toGHOSTLIGHT_ACCEPTANCE_PERSONA_PROSE_LOG; Idunn's staging dirs on Yggdrasil hold.runner-rust-acceptanceworkspaces per transaction. - PA.f216. Creation is one form of eight fields, three of them raw JSON
(
targets{},jurisdictions[], lens weights;eve.rs:244-331), then a Draft card with Approve, Seed and Activate as separate buttons (eve.rs:343-415).
A live defect not yet in the findings
The live process has logged Heimdall refresh transport unavailable error=Heimdall denied the private command: Idempotency key was reused with different command content. once a minute since 2026-09-23 21:04 (1,440 a
day from 10-01). Sequence: 21:02:03 discovery timed out; 21:03:00 local app session rejected Heimdall refresh error=local session changed during refresh;
from 21:04 the denial forever. Mechanism:
- Ghostlight's refresh key is
refresh:{heimdall_session_id}:{access_revision}(runtime.rs:1877-1880), stable until the local session advances. - Every attempt is resealed with a fresh nonce, IV and issue time
(
heimdall.rs:880-895). - Heimdall fingerprints the sealed request bytes, not the command
(
Heimdall/src/private-command-plane.ts:88,createHash("sha256").update(request.payload)), and refuses a known key with a different fingerprint (:93). - So once Heimdall has executed one refresh (it rotated the token at 21:03)
and the local commit refused it (
app_session.rs:293-300), no retry under that key can ever succeed, andsessions_due_for_refresh(app_session.rs:247-268) keeps offering the same candidate whilerefresh_expires_at > now. Which clause ofapp_session.rs:293-299fired at 21:03 is not logged.
The host
ghostlight-dungeon.service(the v1 body) is enabled, inactive since 2026-09-23 14:31. It binds127.0.0.1:8831, runs/srv/ghostlight/current, targets CodexConnector, and would start at the next boot beside Idunn's generation./srv/ghostlightholds 46 GiB (226 releases, acceptance build scripts),/var/lib/gamecult/ghostlight-dungeon17 GiB of v1 world state (the v2 kernel refuses earlier schemas; nothing migrates it),ghostlight-dungeon-quarantine44 KiB,ghostlight-import-20260821T1858Z8 KiB. Disk/is 20% of 2 TB used.- The v1 actuator and its tests live in gamecult-ops:
scripts/deploy-ghostlight-yggdrasil.sh,scripts/test-ghostlight-yggdrasil-wiring.sh; the runbook's "One-time state cut" section (runbooks/ghostlight-dungeon-yggdrasil.md:456-499) describes that path. - Idunn's leftovers: 12 failed
idunn-ghostlight-*transient units (11 from the 09-23 bring-up, 1 from 09-30), 15 release dirs under/srv/ghostlight-idunn/releases(403 MiB), per-transaction staging dirs under/var/lib/gamecult/idunn/staging. These are Idunn's: it launches the units and owns release retention. Phase 1 does not hand-clean them. - Hostname is
yggdrasil-candidate. Heimdall still runs as the legacyheimdall.service; every Idunn v2 deploy of it has failed (idunn status).
The repo
- The pre-rebuild research machine is still on
main:tools/(17 files, 4,146 lines, including the Python state store),schemas/(14, 4,057),examples/(194, 37,421),experiments/(172, 12,325),acceptance/,prompts/,scripts/,output/,assets/newspaper/,state/(map.yaml, evidence ledger and archive, a CultCache JSONL, corpus coverage, branches),docs/aetheria/, rootpackage.jsonscripts pointing at a Codex Python runtime, and about 30 pre-rebuild architecture and note pages. No file outside Ghostlight referencesGhostlight/schemas,ghostlight_state_storeorghostlight.agent_state(grep overF:\Projects, excluding build output). AGENTS.md(234 lines) instructs agents to treatstate/and the Python store as canonical state and the implementation plan as current.- Live docs that cite the old
world/*.rspaths:ghostlight-world-consumer-api.md:19,31,33,38,70;ghostlight-world-ontology.md:53,74,167,202,203,215,232,277;notes/fresh-workspace-handoff.md:253;notes/local-live-smoke.md:56. The code is flat undercrates/ghostlight/src/. Dated cut documents and postmortems keep their paths: they are history.
Model page
One row per persistent kind phase 1 touches. No cell may be empty.
| Kind | Named by | Life over time | Who decides |
|---|---|---|---|
World state and commit journal (.cc, redb) |
WorldId; revision; digest chain |
created Draft, approved, activated; one commit row per admitted command; earlier schemas refused, never migrated | the kernel, through WorldMailbox's closed operation set; untouched in phase 1 |
Play turn row (play-turn store .cc) |
turn_id (UUID); per-turn question_token |
Running → AwaitingPlayer ↔ Running → Closed (narration or fault); an unreadable row is retired to a sidecar and play starts fresh |
PlayTable (play.rs); the card is a projection of PlayTurnView, which phase 1 extends with the fault |
App session (app-sessions-v2.cc) |
cookie hash; heimdall_session_id + access_revision |
issued at sign-in; refreshed when access is near expiry; revoked on logout, custody change, invalid receipt, and (phase 1) on a refresh that can no longer succeed | AppSessionOwner; Heimdall decides the upstream session and rotates refresh tokens |
| Heimdall private command receipt (Heimdall Postgres) | (app_slug, idempotency_key) + request fingerprint |
written once per executed command; replayed for an identical request; a different request under the key is refused | Heimdall; its fingerprint rule is a Heimdall follow-up, not a phase-1 cut |
| Idunn generation (transient unit, release dir) | sha256-<release> |
staged → started → Warming → lease → Active → superseded or failed; the previous Active stays until a new one is Ready | Idunn; retention and failed-unit collection are Idunn's follow-up |
v1 body (ghostlight-dungeon.service, /srv/ghostlight, v1 state) |
unit name; paths | retired in fact 2026-09-23; unit still enabled | nobody today; phase 1 assigns its removal to a gamecult-ops cut, data by operator ruling |
| Bonsai lifecycle (Raven containers) | container bonsai; scripts under /opt/gamecult/bonsai |
started by hand ("Start AI Chat"); restarted by the RSS watchdog; stopped by the idle reaper | gamecult-ops/Raven; Dungeon only reads the endpoint; availability policy is an operator question |
| Repository docs (README, AGENTS, live architecture pages, handoff) | path on main |
rewritten to the live machine; pre-rebuild material leaves main by operator ruling |
the Ghostlight repo; history in git, dated cut documents and postmortems |
Where phase 2's verb sets plug in
Ruling verse-verbs-two-render-paths and invariant every-event-renderable
bound what the kernel may admit to two render paths. Today:
- The thirty operations (
ghostlight-world-ontology.md:451-478) are state transitions, not verbs. They stay the closed substrate. - What an actor does is an affordance: a world-authored catalog entry whose
kind is
AffordanceKindName(patch.rs:247-254), free canonical text that "the kernel carries and branches on nowhere", plus the kernel-builtspeak(controllers.rs:94). Admitted acts surface asDecisionEventson the commit.
So phase 2's two verb sets plug in at the affordance kind: a closed
vocabulary of kinds, each tagged with its one render path, with effects that
still compile to the thirty operations. Today the Play authority can declare
any affordance kind, so nothing on main enforces the invariant yet. Phase 1
neither widens nor narrows that, and its docs must not describe open-ended
verbs as the target.
Rationale
- "main dies" was a misreading. The deploy that died carried no code
change, and it died on Odin's route, not on Ghostlight. So phase 1's deploy
work is mostly preflight and proof: redeploy
mainafter the play fixes, with Odin's route and IO pressure checked first. The Odin outage is recorded against its owner. - Subtraction goes first and alone. The pre-rebuild tree is about 60,000 lines that the live machine never reads. It is cut before the docs are rewritten, so the rewrite describes only what is left. Parking it at a tag rather than deleting it outright is the operator's call (question).
- Faults belong to the card, not the log alone. PA.f212 is an authority gap: the turn store knew the fault and the projection dropped it. The fix extends the projection; it adds no second record.
- Retrying inference is safe. A model call is a proposal the kernel admits or refuses, so a transport fault before any reply is retryable. That is the Ghostlight half of PA.f211. The Raven half (the watchdog's quiet window) belongs to gamecult-ops, which also takes the scripts into version control.
- The refresh loop is a contract split. Ghostlight assumes a stable key makes a retry safe; Heimdall's fingerprint makes every retry new. Ghostlight stops looping (a refresh that can no longer succeed revokes the local session). Heimdall's fingerprint rule goes to Heimdall as a follow-up.
- The bridge tests are the gate's blind spot. They are the only Dungeon tests that drive the real client, and nothing runs them. Fixing the gate means a runner with both Rust and Node, which is Idunn's recipe and runner surface: a follow-up, with each phase-1 Hands run doing them by hand.