First typed Eureka campaign (idunn-watchdog): 8 cuts, 17 Soul verdicts,
55 findings; landed about 2.5x its estimate. Idunn pages the operator on
Discord through Bifrost; closure DM still owed.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Eureka idunn-watchdog cut idunn-topology-lock, attempts 1-2. Soul verdict
idunn-watchdog:verdict:cut-idunn-topology-lock.s2 holds. Topology and
incident writes share one private-lock exchange that replaces a non-private
lock by rename; a contended publish stays in phase and retries. Measured
tick with a held lock: 0.09 s (was 5.07 s).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
resume_candidate turned any failed pre-Fencing step into a pre-fencing
abort, so a topology publish that met a held lock (the nightly backup's
shared lock, an operator) ended the deploy or continuity restart on its
first attempt. LockContended is now a typed error (drivers::LockContended,
raised by mutate and IncidentStore::exchange with the same texts as before)
and resume_candidate's one classification keeps such a transaction in its
phase with last_error set; the backoff retries it and the phase deadline
bounds the wait. Other publish errors still abort.
Tests at the resume path: a contended Starting publish stays in Starting
across ticks and lands after release with one continuity restart; a
non-contention publish error still aborts. Both builds.
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Mutation of the first cut left make_lock_private's error classification and
lost-create-race branch unkilled. Two behavioural tests at the primitive:
an uninspectable lock (name one byte past NAME_MAX) reports an inspection
error, an uncreatable one (procfs) a creation error, and neither is retried
as a lost race; eight publishers released together to create the same lock
all find it private, so none fails for having lost.
Tests only; builds.
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
exchange_behind_private_lock (drivers.rs) is the single write primitive for
the projection directory: one nonblocking compare-exchange behind a lock only
Idunn can hold. An absent lock is created O_EXCL 0600; a lock that is not a
regular file, ours, mode 0600 is replaced by a fresh 0600 file renamed over it
(a chmod does not revoke a descriptor another uid already holds); a private
lock is opened as is. The topology driver's mutate and the incident store both
go through it; the blocking compare_exchange_snapshot call, publish_projection_mode
(which published the lock 0644 on a false premise that readers open it) and
incident.rs secure_lock are deleted.
mutate now retries only a lost race; a held lock is Err "the topology
projection is locked by another holder" and a no-op, so no caller records a
digest, evidence or phase for a publish that did not land.
Test helpers: the lock holder is a separate flock process, and run_while_held
counts opens of the lock (inotify), so ANY wait on a held lock fails, not only
one past the 30 s ceiling. Tests reproduce the live default ACL and use
setpriv for the other-uid half.
This commit has not been built; the verify run follows.
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Eureka idunn-watchdog cut idunn-incident, attempts 1-3. Soul verdict
idunn-watchdog:verdict:cut-idunn-incident.s3 holds. Incidents live in
their own incidents.cc (0644, lock private 0600), written with non-blocking
snapshot exchanges; the tick never waits on them. Close rule: running and
no longer exhausted (operator ruling exhaustion-close-rule-op).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Only the daemon writes topology.cc; Odin and Idunn read it lock-free; the
root backup's shared lock survives a private 0600 lock. A chmod does not
revoke an existing holder, so the shared primitive renames a fresh 0600
lock over a non-private one.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
lock-acl-world-openable: CultCache creates a lock with mode 0666, so under
Yggdrasil's default ACL (u::rw g::r o::r) the incident lock landed 0644 and
any uid could hold it and keep every incident from being recorded. Every
incident write (incidents.cc and incident-history.cc) now goes through
exchange(), which first creates the lock with O_CREAT mode 0600 (the create
mode is intersected with the ACL, so 0600 holds and no wider file ever
exists) and fchmods an earlier, wider lock to 0600 on the descriptor it just
opened. The test sets the live default ACL on its directory, asserts a
control file made the way CultCache does reads 0644 there, asserts both
locks are 0600 after open, close and retire, and, as root with setpriv,
that uid 65534 cannot open them.
close-fault-repeats: record_incident keys its fault by operation and subject
(open/close per target, * for the reconcile and retire passes), so one
site's Ok no longer clears another site's standing fault.
mutation-count-poisoned: a_held_incident_lock_changes_nothing_continuity_decides
failed under parallel load because run_while_held asserted the work took
under 2 s of wall clock, and the work is a whole continuity sequence. The
holder now lets go when the work returns, or after a 30 s stuck ceiling;
the test fails only if the work was freed by the ceiling. eventually() and
the control-plane retry use the same ceiling.
Builds.
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
route_connect_retries_connection_refused_until_listener_is_ready blocked in
accept() with no deadline. If the client connected to another listener on the
reused port, accept() waited forever and the whole verify job held its slot
for 39 minutes. The kernel completes a handshake from a listener's backlog,
so the connect succeeds without an accept: the listener now only outlives the
connect, waiting on a channel with a 30 s ceiling. A lost port race fails
the test instead of hanging it.
Builds.
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Tests only, from Soul's last B5 check:
- F2: a rollback or withdrawal back to an incumbent route deletes no
firewall rule (S-R2 survived).
- F3: expiry requests whose ids are a strict prefix of the transaction's
command id, or extend it, leave it untouched (S-ME1, S-ME2 survived).
- F4: the second and third failures wait two and four polls, as back_off
writes them (S-MJ1, S-MJ2 and a clamp survived).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Phase deadlines are resolved before every post-fence step; `idunn expire` requests an
expiry through the same resolver; lease adoption is recorded from Odin's Ready; an abort
withdraws any issued lease (Prepared or Granted); install/rollback/withdrawal inside a
transaction always write and reload; a first-deploy withdrawal closes the firewall rule
before reloading. The operator-required arm for an adopted-lease deploy stays until the
Q3-ii snapshot cuts (Q-S5: B5 first).
Soul passes 1-3. Merged tree verified on Yggdrasil: 360 passed, 2 ignored.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The route ran the gamecult-ops compose lane, which would start a container beside the native
unit. Operator: "delete the Bifrost lane". Grant, dispatch and the Bifrost-authority check that
served only it are removed; bash -n and visudo -cf pass.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The operator ruled "delete the Bifrost lane". This route was that lane's live
trigger: /srv/odin/deploy-manifests/bifrost-persona-feedback runs
`docker compose up persona-feedback`, and the provider actually running on
Yggdrasil is the native bifrost-persona-feedback.service. Firing the route
would start a container next to it.
Following deploy/legacy's own order, the grant goes first, then the dispatch:
- sudoers: drop `deploy bifrost-persona-feedback *`.
- idunn-yggdrasil: drop the target's policy case and its dispatch token, plus
the Bifrost release-authority check that only it used (requires_bifrost_authority
and the `idunn validate-release-authority` call). A deploy request that claims
a Bifrost authority is still refused. The manifest environment keeps
IDUNN_REQUIRES_BIFROST_AUTHORITY=false, because the installed Epiphany manifest
checks for it.
- README: record the retirement and the counts. Also record that on 2026-09-30
the installed actuator matched this file, but the sudoers grant was not
installed: the idunn user cannot sudo.
Rebinding persona-feedback to a current Idunn binding is a separate follow-up.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
`restore` with no prior membership now withdraws the stable endpoint's
firewall allow before it writes the fragment and reloads. A failing `ufw
delete` retries without touching nginx, which bounds the withdrawal reload
loop at zero until the Q3-ii cut gives the abort a deadline. A crash between
the two steps is safe: a rule already gone reads as withdrawn, and the retry
removes the fragment and reloads.
Pins from Soul pass 2:
- MB2: a redeploy over an existing incumbent route installs, rolls back and
withdraws with a reload each (`changes_route` as `is_none()` survived).
- ME2/ME3: the resolver's command filter is tested with an id of equal
length and the same leading bytes.
- MJ2: the backoff test asserts the wait `back_off` writes, one poll after
the failure.
- Soul's F2 probe returns as a test: a withdrawal whose `ufw delete` keeps
failing reloads nginx not at all over 20 retries.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A test that held incidents.cc.lock and then released it failed once in a full
parallel run: another test's forked child briefly shared the open file
description. The follow-up calls are exactly the "retried on a later pass" the
store promises, so the tests now retry for up to two seconds instead of
asserting the first call after release lands.
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
A refused delete after a successful archive leaves the record in both files by
design; the test asserted the history file was absent. It now asserts the repeat
archives nothing twice.
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Every incident write is one nonblocking try_compare_exchange_snapshot of the
whole file against the snapshot it was decided on (as drivers.rs does for the
write lease). A held lock or a lost race is an error and a no-op for the tick,
retried on a later pass; it is also what makes one open incident per
(condition, subject) real. The incident .lock is no longer published: readers
of incidents.cc take no lock, so only Idunn can hold it. History is written the
same way.
An exhaustion incident closes as recovered only when the workload is observed
running and restarts_exhausted is false (ruling exhaustion-close-rule), so a
release that survives one pass after each restart keeps one incident.
Reconcile closes only incidents whose condition ends with admission and whose
subject is not admitted; its dead accumulator is gone. IncidentRecord owns the
retention predicate that retirement and status both use. status writes to a
writer and serve's loop is serve_while, so tests reach both.
Not verified yet: this commit has not been built.
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
`a_failing_step_is_retried_after_a_backoff_not_every_tick` raced the
wall-clock wait against ten ticks and lost on a loaded host (9 of 15
mutant runs). The test now owns the wait: held open, ten ticks retry
nothing; once due, the next tick retries.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A stateful candidate Odin reports on is warming until it holds the lease;
the world's fixture now says so, and a post-fence warming word lets the
grant happen. Odin's Ready carrying the granted lease records the adoption,
sourced `OdinTopology` and naming the presence Odin observed. Kills Soul's
M6 (the source labelled `Direct`) and M7 (no adoption from Odin).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
With two live transactions, an expiry request for one leaves the other
running. Soul's mutant "any request expires every live transaction"
survived the suite.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The physical grant lands before `Granted` is durable. A continuity that
died in that window and passed its deadline was aborted from `Prepared`,
its abort withdrew only a `Granted` lease, and the lease left on disk made
every later fencing of the target refuse "unexpected process write lease".
The abort now withdraws any issued lease; revoking an absent one is a no-op.
Test: a granted lease on disk behind a `Prepared` record, deadline passed:
the abort removes it and the next fencing's revoke succeeds.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
`6ff7d5a` skipped the write and reload in `install` and `restore` when the
fragment on disk already equalled what they would write. On a transaction
path that hides a crash between a write and its reload: an abort that
retries after the crash reloads nothing and then stops the candidate, so
nginx keeps proxying to a dead port, and a fenced install skips every retry
(and a first deploy's `ufw allow`) until its deadline. The skip on equal
bytes goes back to supervision only.
What stays: a transaction whose candidate membership is the preflight's
baseline (a continuity over its admitted route) has no route change to make
or undo, so its install, rollback and withdrawal run nothing. No write of
its own can have been cut short. `RoutePreflightReceipt::changes_route`
names it.
`restore_admitted_membership` now puts back the bytes it replaced when its
reload fails, instead of deleting the fragment, so the disk still shows
what nginx serves.
Tests: a crash between a write and its reload is reloaded on retry (install
past the fence and withdrawal, driver level; the abort, engine level); a
transaction that does not change the route runs nothing. Soul's F2 probe
test is gone with the rule it pinned: a withdrawal whose `ufw delete` keeps
failing reloads on every retry again, until the Q3-ii cut gives the abort a
deadline.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Only cultcache-ts was renamed in CultLib; Bifrost's tests and three tools
still use the bare name. Production Persona delivery is unaffected: its
release bundles CultLib f67f5122.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A terminal record leaves the live set, so it could not tell the verb's own
command filter from a looser one; another target's deployment in flight can.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The retention constant and the usage line survived mutation untested. The
incident module doc named the delivery service, which the no-transport grep
over src/ must not find. Builds.
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Kills three cargo-mutants survivors: the --requested-by arm, run's
dispatch, and expire's own-live-transaction filter.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
One open and one closed record, written by IncidentStore on Yggdrasil under
IDUNN_WRITE_FIXTURES=1 and read back by the_incident_fixture_is_what_idunn_reads.
Bifrost's reader tests copy this file. Builds.
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Idunn's daemon, at the continuity decision site, writes one
idunn.operator_incident.v1 record per exhaustion into its own published
incidents.cc (--incident-store), never control.cc. The record opens once per
(target, condition), closes on an observed running workload (recovered) or on
the target leaving the admitted set, and retires to incident-history.cc seven
days after closing. A store failure is traced once and never returned into the
tick. idunn status prints each target's incidents. ReportOnce is demoted to a
stderr trace.
Does not build the fixture test until tests/fixtures/idunn.operator_incident.v1.cc
lands in the next commit.
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Imagination's cut-mapping probes and rationale. Self collapses the model
page's record and projection rows: control.cc refuses foreign types, so
incidents.cc is both the record and what Bifrost reads.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Imagination step 0b for campaign idunn-watchdog. Forks are admitted in the
mind as questions delivery-crossing, paging-conditions, closure-notice.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The live shape of Soul's F1: a continuity reuses the admitted Expected, so
its install and rollback find the admitted fragment already on disk and run
no host program, before or after its Routing deadline.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
