exchange_behind_private_lock (drivers.rs) is the single write primitive for
the projection directory: one nonblocking compare-exchange behind a lock only
Idunn can hold. An absent lock is created O_EXCL 0600; a lock that is not a
regular file, ours, mode 0600 is replaced by a fresh 0600 file renamed over it
(a chmod does not revoke a descriptor another uid already holds); a private
lock is opened as is. The topology driver's mutate and the incident store both
go through it; the blocking compare_exchange_snapshot call, publish_projection_mode
(which published the lock 0644 on a false premise that readers open it) and
incident.rs secure_lock are deleted.
mutate now retries only a lost race; a held lock is Err "the topology
projection is locked by another holder" and a no-op, so no caller records a
digest, evidence or phase for a publish that did not land.
Test helpers: the lock holder is a separate flock process, and run_while_held
counts opens of the lock (inotify), so ANY wait on a held lock fails, not only
one past the 30 s ceiling. Tests reproduce the live default ACL and use
setpriv for the other-uid half.
This commit has not been built; the verify run follows.
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
cd662cd claimed no target is Idunn-managed and that every idunn-odin-*
unit was failed. Both wrong. idunn-odin-25f32a8d is active and running
release 928da3ef, and it is the live Odin -- the odin.service unit still
in /etc/systemd/system is active with MainPID 0 and owns no process.
That claim came from reading the first 20 lines of an alphabetically
sorted unit list. 25f32a8d sorts below the cut. Truncated enumeration,
stated as a whole-system fact, in the same commit that corrected someone
else for a whole-system fact taken from the wrong artifact.
Heimdall genuinely is not migrated; that half stands.
Also records why odin's transaction is stuck in Committing: the gate
requires the latest Odin observation to equal the Ready receipt, which
no target with a working observer can satisfy.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The header claimed "no target has been migrated" and that the installed
binary was the previous generation. Half of that was stale: yggdrasil
runs the declarative binary and idunn status answers.
The other half was right, and a proposed "correction" to say odin and
heimdall are now Idunn-managed would have been false. Verified on the
host: heimdall.service and odin.service are hand-written units under
/etc/systemd/system, both active, and every idunn-heimdall-* and
idunn-odin-* unit is failed.
The "Correction: Odin is migrated first, not last" section reads like a
completion record and is not one -- it changed the ordering, it did not
migrate a target. Said so in the header, because that misreading has now
happened twice.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Five more preprovisioning requirements, each of which cost a candidate that
started, ran, and then died on a file it could not open: bundle traversal,
the setgid runtime root the write-lease hardening assumes, the lease record's
group, a default ACL on the published projection, and the route directory's
ownership left over from unprivileged Idunn.
Also states the two shapes that made every one of those faults arrive in
disguise: a store's .lock is opened whenever the store is, and CultLib reports
an unreadable file as an empty store rather than an error.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Five things that had never been exercised failed on the way through Odin, and
each is now a test: the unprojectable RUDP route, step environment not being
ambient, build containers having no machine identity, the published projection
sitting inside Idunn's private root, and a signing identity living in the
state root the workload can write.
Also documents that `idunn validate` proves nothing about deployability
without a binding -- admit is where every cross-document rule lives -- and
repairs apostrophes an earlier pass mangled into ".s".
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Deploying Heimdall proved the migration order backwards. A candidate's
warming presence -- the evidence Idunn requires before fencing an incumbent
-- is observed through Odin's authenticated runtime topology correlation.
advance_warming has exactly one exception: target == "odin" while no Odin
is admitted may warm on directly observed presence. Every other target
waits for Odin to publish.
So with the previous generation's Odin still running, a Heimdall candidate
starts, serves on its assigned bind, publishes health, and warms forever.
It never fails and never promotes. That is not a defect; it is the
observation chain being rooted somewhere I had not looked. It also explains
why Odin has the only recipe in the estate that was ever kept current.
The document said "migrate the authority's own dependencies last". Odin is
not a dependency to defer, it is the root, and the entry now says so.
The wait also recorded no reason, so a candidate could sit in Warming
indefinitely with nothing to read -- the fourth coarse-signal case today.
It now records what it is waiting for and why the wait cannot resolve
without Odin.
Noted alongside: health published over RUDP to the previous generation's
--rudp-health-bind goes nowhere, since the current serve has no such flag.
That produces a steady stream of timeouts in the candidate's journal and is
not the reason promotion stalls.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The first launch with a full environment failed EACCES on
/srv/heimdall/keys/signing-key.pem: the workload runs under DynamicUser and
cannot open a heimdall:heimdall 0600 file. The wrong fix is a supplementary
group -- widening access to resolve a permissions error.
Idunn already has the right one. A [workload.secret_files] entry is loaded
as a systemd credential, and the driver sets an env var named after the
entry whose value is the credential path. Naming the entry after the
variable the service already reads wires it up with no code change, and
setting that name in [workload.environment] too is refused as a collision.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The runtime presence identity is handed to the workload as an inherited
file descriptor, so Idunn requires the source file to be root-owned, mode
0400, and singly linked. Enrolment leaves it 0600, which fails.
The nlink check is the interesting one: a second hard link would be a
second path to the signing key, and a descriptor that cannot be reopened
is the whole point of passing it this way.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
ProtectSystem=full makes /etc read-only, so Idunn could not create Heimdall's
runtime bundle under /etc/gamecult/heimdall/runtime. A target whose roots are
not in ReadWritePaths is a target Idunn cannot deploy. Every migration adds
exactly three paths -- runtime root, state root, release root -- which keeps
the write set enumerated and is the honest blast radius of running as root.
Also records the per-target preprovisioning Idunn requires rather than
creates, each discovered by a failed deployment: the state root must be
root:<state_group> mode 2770 so the workload's dynamic user inherits group
access through setgid; the runner cache must be 0700 owned by the runner's
container uid, not the source identity, with a root-owned parent (/srv/build
is idunn:idunn and fails); the frozen source stage must be root-owned,
because source is frozen as idunn and then copied into a root-owned
actuation stage; and the binding's docker network must exist.
None of these are Idunn being fussy. Each is a check that a binding cannot
widen access by naming a path someone else can write.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The CultLib forks do not disagree. cultcache-ts reads Idunn.s live control
store -- 36 records -- and ordinary CultCache stores share one framing
across runtimes. Only service identity private stores use the bare
single-envelope container, deliberately, because they hold a key rather
than a record set.
The section now says which shape is which and names the failure it nearly
caused: a consumer that hand-decodes the private-store framing and applies
it to expected.cc and the write lease fails on first deployment while its
tests pass, because hand-built fixtures agree with the hand-built reader.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Enrolling Heimdall's signer surfaced that cultcache-ts cannot read a
single-file store written by cultcache-rs at all: pullAll() throws
invalid_type, expected object received array. Verified against the real
idunn-provision enrolment on yggdrasil.
Idunn writes every store a target must read -- presence identity, process
write lease, expected incarnation -- with the Rust client. Rust targets
consume them natively; TypeScript targets cannot, and must decode the
layout by hand. That makes this a per-runtime blocker rather than a
Heimdall quirk, so it belongs in the migration route.
Worth separating what broke from what held: the record schemas are
identical across runtimes, including the identity's protector binding. The
contracts are fine. The store framing drifted.
Which framing is canonical is a CultLib decision and is not taken here; the
C# implementation is the stated reference and is the tiebreaker.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Shared prerequisite for every routed target, done on yggdrasil. Debian
ships ngx_stream_module as a separate dynamic package and it was absent:
nginx -V showed --with-stream=dynamic with no .so. Installed
libnginx-mod-stream at 1.26.3-3+deb13u7, matching the running nginx
exactly so nothing else moved, added a top-level stream block including
/etc/nginx/idunn-stream-routes/*.conf, and created that directory
root:idunn 0775. nginx -t passed before the reload; gamecult.org and
heimdall.gamecult.org/healthz both returned 200 after it. Backup at
/root/nginx.conf.bak-20260906-stream.
Worth recording how the module check nearly went wrong: the first
`nginx -V` ran without sudo, hit "command not found" because the binary is
not on the unprivileged PATH, and reported NO STREAM MODULE. That is the
estate's own trap -- a failed read presented as a negative result -- caught
only because the answer looked too convenient.
The larger correction: the unit I wrote yesterday ran User=idunn, and this
generation cannot work that way. There is no sudo anywhere in Idunn's
source; it calls systemctl, systemd-run, nginx and docker directly, so a
confined user actuates nothing. deployment-authority.md states the intended
model plainly -- freeze source as idunn, copy into a root-owned actuation
stage, the privileged driver never opens Git -- and --source-uid/--source-gid
are validated as non-zero, which only makes sense for a privileged daemon
dropping down for source work.
So the unit now runs as root with --source-uid 986 --source-gid 978, keeps
ProtectSystem=full and re-opens each actuated path individually, and adds
RuntimeDirectory=idunn for the route driver's preflight staging. The
comment block explains the tradeoff rather than leaving the next reader to
infer it: privilege separation is internal, and what makes it tolerable is
that Idunn accepts no imperative input.
That is a genuine increase in blast radius over a confined process, and
migration.md says so instead of selling it. The counter-argument is that
the old shape was confined in name only -- a sandboxed process whose sudo
grant pointed at an unaudited root shell with eleven hardcoded targets.
systemd-analyze verify passes on the rewritten unit.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
StreamPixels is changing direction: 2D animation could not express what
the spec required, so it is moving to a 3D-rendered overlay, and whether
that runs in WebGL or needs dedicated rendering infrastructure is open.
The client funding it cannot absorb rendering overhead, which is what makes
the WebGL question load-bearing rather than a preference.
Its deployment shape is downstream of that answer -- a WebGL overlay keeps
roughly the current two-unit shape, rendering infrastructure is a new
target class with hardware requirements this host has never expressed. A
binding written now would pin the shape being replaced.
The survey of its current defects stays as a record but is marked
superseded, with an explicit instruction not to act on it. The tsx-at-
runtime path is exactly the kind of thing a passing agent would tidy, and
tidying it would be work spent on something being replaced.
heimdall leads the remaining three instead. Its prerequisite is deeper
than repixelizer's but far better understood: a vendor/CultLib gitlink,
following Ghostlight's existing vendor/eve pattern, replacing a
cultlib_commit hardcoded in a shell manifest.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
I called these three "the simple ones" and "compose-shaped". Both were
wrong. None of them is compose -- the legacy actuator restarts all three
with `docker compose -f /srv/compose/yggdrasil-apps.yaml`, and that file
does not exist; /srv/compose holds only odin, voidbot and voidbot-retrieval.
The path is guarded, so every restart of these three has been failing at
the guard. They are running only because nothing has asked them to restart,
which is the difference between a supervised service and one that happens
to be up.
Each needs a change in its own repository before a recipe means anything,
and the changes are different shapes, so this is not one piece of work:
streampixels has real build scripts (tsc, next build) and no external
dependencies, but its service unit runs TypeScript through tsx from a
checkout, and it is two units under one target name. A recipe declares one
[service]; the two have different environments, ports, and an ordering
dependency. It was one target only because a shell line restarted both.
repixelizer has the cleanest dependencies of the three -- no CultLib, no
submodules -- but runs a venv interpreter against a checkout, so there is
no artifact to seal and no revision observable from the process.
heimdall is the most release-shaped and needs the deepest change: it takes
CultLib as file:../CultLib, a sibling checkout outside the repository. The
current manifest works around that by cloning CultLib separately with
cultlib_commit hardcoded in shell -- exactly the untyped authority recipes
exist to replace. It needs vendor/CultLib as a gitlink, like Ghostlight's
vendor/eve.
Also corrects the advice to copy CodexConnector's recipe. That recipe is
stale: it declares required_gitlinks for vendor/cultcache-rs and
vendor/cultnet-rs, and the repo has no .gitmodules, no vendor/, and takes
CultLib as a pinned cargo git dep instead. Recipes drift from their repos
silently because nothing re-checks them; run `idunn validate` in each
target's CI so drift fails at the repo rather than on the host.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The migration procedure told operators to dry-run a target, and no dry-run
existed. `up` needs a running daemon and a state store, so the only way to
find out whether a binding was well-formed was to install it and see
whether `serve` refused to start -- on the host, with the authority down.
`idunn validate --recipe PATH [--binding PATH]` runs the same
TargetDeclaration::parse and OperatorBinding::admit the daemon runs at
load. It opens no store, contacts no host, and actuates nothing.
Owner: the existing control-plane CLI, no new module or target. Consumer:
whoever authors a binding, starting with this migration. Invariant: a
binding installed on the host admits against its target's recipe. It
replaces the install-and-see loop rather than adding a layer.
Verified against the real estate: Odin, CodexConnector and Ghostlight
recipes all parse. Negative paths checked too, because a validator that
only ever passes is worse than none -- an unknown recipe field (the
privilege-escalation shape), a missing --recipe, and an unreadable file
all fail with the reason.
A test asserts validate stays declarative and refuses --deploy-command,
matching the guarantee cli_exposes_only_declarative_commands makes for the
rest of the surface. 64 passed / 18 failed, against a 63/18 baseline: one
added pass, no new failures, and the 18 remain the known Windows
absolute-path cases. cargo check for x86_64-unknown-linux-gnu is clean.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Gjallar had a brief useful life on Nightwing's framebuffer, broke, and was
never fixed. Its Yggdrasil incarnation ran --headless --refresh-hz 2 for
eleven days: a compositor with no display recomposing twice a second, 55
minutes of CPU and 125M resident producing frames nothing rendered. The
only reference to gjallar.overview in the estate is one Odin test file.
Stopped and disabled; 420M under /srv/gjallar is reclaimable.
Removed the codex-connector and gjallar sudo grants from the host and
synced deploy/legacy/idunn-yggdrasil.sudoers to match. Four grants gone,
verified by diffing the grant set against a backup, nothing else changed.
That diff caught a defect I introduced: removing the last entry of a
Cmnd_Alias leaves a dangling continuation, and my repair silently dropped
the wildcard from `deploy streampixels`, which would have failed every
streampixels deployment. The procedure now says validate with visudo -c -f
before installing and diff the grant set afterwards.
The larger finding: the installed generation CANNOT express that a target
is retired. Its target list is compiled in, it offers restart and redeploy
and no way to drop a daemon from supervision, and its only brake is the
deployment brake -- which doctrine forbids using for this, since a
deployment brake must not suspend continuity. Stopping codex-connector
therefore produced eighty "decision: restart" lines in ten minutes against
a service we had just retired. Suspending continuity for one target is
precisely the current generation's lifecycle brake. The estate found the
argument for its own migration by trying to throw something away.
Gjallar was handled correctly by contrast: its failure classified as
dependency-unavailable, so Idunn raised an alarm instead of resurrecting
it. The old generation can tell failures it owns from failures it does
not; it cannot tell a broken target from a finished one.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Stopped and disabled on yggdrasil 2026-09-05; port 4103 closed, the
three-minute 401 loop ended. It had burned 1min 30s CPU and peaked at 1.2G
retrying expired credentials against a provider the estate has left.
Two things the stop surfaced, both worth having written down:
ghostlight-dungeon.service still carries GHOSTLIGHT_MODEL_CONNECTOR
pointed at 127.0.0.1:4103, which is now nothing. It had logged nothing for
24 hours beforehand so no working path broke, but that environment variable
is where the provider replacement lands first.
epiphany-model-connector.service binds the same port with --model gpt-5.4
and its own --codex-home. It was already inactive and disabled. Two Codex
connectors were bound to one endpoint, one of which has no repository in
the swarm -- duplicate authority of exactly the shape the census went
looking for, found by stopping one of them rather than by reading.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The operator is not returning to Codex, which removes the consumer for a
credential-isolated Codex subscription transport. The deployed daemon is
already dead in practice: codex-connector.service on yggdrasil is active but
has been failing 401 token_expired every three minutes against
chatgpt.com/backend-api/codex/models, with nothing connected to port 4103.
It was this document's first migration target; writing a binding for it
would have been work spent on a corpse. gjallar leads now.
Adds a Retiring Codex section with the blast radius in removal order --
the running unit, the sudoers grants, GameCult/CodexConnector,
Epiphany's openai-codex-spine and codex-bridge plus vendored app-server,
VoidBot's Codex model config, and the AGENTS.md doctrine twin maintained
for a second agent runtime.
It also says what NOT to do: Epiphany and Ghostlight still need an
inference transport, and CodexConnector was one implementation of that
seam rather than the seam itself. Deleting the boundary and letting each
mind grow its own HTTP client would re-fork exactly the wire law that
README exists to prevent. Replace the backend, keep the seam, and settle
that before writing epiphany bindings.
Also corrects a false claim I made in the first draft. Odin's recipe is
not the only worked example -- there are eight, including Ghostlight's and
four Ghostlight-worlds variants. Recipe authoring is not the bottleneck
for those targets; bindings and the daemon cutover are.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
I had Ghostlight first on the strength of "actively developed". That is
the reason to keep out, not to go in: a world-elaboration and ontology
rebuild is running in it now, with commits landed 2026-09-05 across 178
branches. Adding a recipe to that tree collides with live work.
codex-connector leads instead -- one service, small blast radius, and it
already consumes the core stack from CultLib rather than forking it.
gjallar second as the cheapest end-to-end proof of the systemd-transient
workload driver: quiet repo, one unit, no compose indirection.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The README and unit file I wrote at extraction documented the previous
generation's CLI. Running the deployed ExecStart against a build of this
repository returns `unknown Idunn command "--swarm-profile"`: commit 330e4cd
rebuilt Idunn around typed authority ports and a test now asserts that
--deploy-command, --restart-command and --swarm-profile are rejected. Anyone
following the README would have installed a unit that crashloops on exec
every 5 seconds under Restart=always.
Corrected against the binary, not against the old docs:
- deploy/idunn-yggdrasil.service now invokes `idunn serve` with the real
flags. systemd-analyze verify passes on yggdrasil. /var/lib/gamecult/odin
moved from ReadWritePaths to ReadOnlyPaths -- Idunn only reads Odin's
topology correlation (pull_all_read_only_snapshot, absent store yields
None), and it gates deployment promotion only: continuity never consults
it, per stale_odin_provider_receipt_cannot_gate_continuity_restart.
- README documents the three declarative commands, and adds a section on the
recipe/binding split, which is the security model: capability lives in the
target's repo, privilege lives in the host binding, and a binding that
omits an affordance the recipe derives fails admit().
- Corrected two inherited claims. The distinct-brake check is per-binding
validation ("deployment and lifecycle brakes are identical"), not a
startup check. And the "inputs: nothing else" claim was false -- the Odin
correlation read is a real, deliberate exception and now says so.
The legacy actuator moves to deploy/legacy/ rather than being deleted: it is
what is installed and running, so it is the before-side of the migration.
Its README maps each shell mechanism to the contract that replaced it, and
preserves the eleven-target inventory read out of deployment_target_policy().
Seven of the eleven are pinned to a codex/ branch rather than main.
docs/guide.md is marked as the previous generation rather than silently
carrying dead commands, and docs/migration.md is new: per-target order,
what to fix before carrying a ref forward, and why the authority cuts over
last. Also fixes a broken cross-link in authority-map.md.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
