Compare commits
30 changed files with 787 additions and 16554 deletions
1
Cargo.lock
generated
1
Cargo.lock
generated
|
|
@ -482,7 +482,6 @@ dependencies = [
|
|||
"libc",
|
||||
"rmp-serde",
|
||||
"serde",
|
||||
"serde_json",
|
||||
"sha2",
|
||||
"tempfile",
|
||||
"toml",
|
||||
|
|
|
|||
|
|
@ -18,10 +18,6 @@ path = "src/bin/idunn-provision.rs"
|
|||
name = "idunn-host"
|
||||
path = "src/bin/idunn-host.rs"
|
||||
|
||||
[[bin]]
|
||||
name = "idunn-launch"
|
||||
path = "src/bin/idunn-launch.rs"
|
||||
|
||||
[lib]
|
||||
name = "idunn_daemon"
|
||||
path = "src/lib.rs"
|
||||
|
|
@ -38,7 +34,6 @@ serde = { version = "1", features = ["derive"] }
|
|||
ed25519-dalek = "2.2"
|
||||
toml = "0.8"
|
||||
uuid = { version = "1", features = ["v4"] }
|
||||
serde_json = "1"
|
||||
|
||||
[dev-dependencies]
|
||||
tempfile = "3"
|
||||
|
|
|
|||
131
README.md
131
README.md
|
|
@ -58,64 +58,6 @@ idunn status --command <id>
|
|||
the bindings, so `idunn up profile:full-gamecult` brings up everything bound
|
||||
into that profile.
|
||||
|
||||
### Steward orders
|
||||
|
||||
A second root unit from the same binary, `idunn-orders` (`idunn orders serve`),
|
||||
takes typed orders from one steward Mind over `/run/idunn-orders/steward.sock`.
|
||||
It has its own store, `orders.cc`, and its own binding, `steward.toml`; the
|
||||
survival daemon takes no order. There is still no command string:
|
||||
|
||||
```bash
|
||||
idunn order package-install <pkg>[=version]... --reason TEXT # apt-get update, install named packages, check them
|
||||
idunn order unit <start|stop|restart|reload> <unit> --reason TEXT
|
||||
idunn order deploy <service|profile:name> --reason TEXT # the same commit as `idunn up`
|
||||
idunn order show <id> # status and output tails
|
||||
idunn order list
|
||||
```
|
||||
|
||||
A unit order submits exactly one job, for the unit it names: every verb runs
|
||||
`systemctl <verb> --job-mode=ignore-requirements -- <unit>`, so systemd builds a
|
||||
transaction of one job. It does not start what the unit requires or restart what
|
||||
is part of it: order the dependency first, and order `postgresql@16-main.service`
|
||||
rather than the `postgresql.service` wrapper. Targets are not orderable. The job is one; what the unit does afterwards is
|
||||
not: systemd still runs full transactions of its own for `Restart=` after a
|
||||
failure, for a `StopWhenUnneeded=` dependency left without a user, and for a
|
||||
`Conflicts=` partner of a unit that fails, so an ordered stop of a unit
|
||||
that something restarts does not last, and a start of a `Restart=always`
|
||||
unit can report failure and be active seconds later. A unit
|
||||
order may act on any unit except Idunn's own (`idunn-*`) and the minds' own
|
||||
(`mind-self@*`, `mind-persona@*`), and the units that end or take the host
|
||||
(`systemd-suspend`, `systemd-hibernate`, `systemd-hybrid-sleep`,
|
||||
`systemd-suspend-then-hibernate`, `rescue`, `emergency`). That set is compiled
|
||||
into the binary and read from no configuration. An order is refused, recorded
|
||||
and posted when the ordered unit, or an alias it resolves from, is one of them;
|
||||
when it has an emergency action (`SuccessAction=`, `FailureAction=`,
|
||||
`StartLimitAction=`, `JobTimeoutAction=`) other than `none` or an `OnFailure=`
|
||||
or `OnSuccess=` unit (not for a reload); or, for a stop or restart, when another
|
||||
unit is `BoundBy=` it, because systemd stops that unit with it whatever the job
|
||||
mode; for a start or restart, when it has an `Upholds=` unit, which systemd
|
||||
starts in a full transaction whatever the job mode; or, for a stop, restart or
|
||||
reload, when it has a `ConsistsOf=` member, a wrapper such as
|
||||
`postgresql.service` whose verbs would exit 0 and touch nothing. A unit
|
||||
`systemctl show` cannot resolve fails the order. Slices, scopes,
|
||||
targets and globs are not units an order can name.
|
||||
|
||||
A package order installs the packages it names and nothing else: apt runs with
|
||||
`--no-remove`, each name is passed as an exact-name pattern so `.` and `+`
|
||||
match themselves, and a last step fails the order unless every named package
|
||||
is installed.
|
||||
|
||||
An order runs once, after the steward's uid is checked and the instance's
|
||||
brake check (`mind-control --instance <principal> check`) exits 0. The check
|
||||
that the order's principal and uid are the bound steward's runs again at every
|
||||
execution, so an order accepted before `steward.toml` changed is refused and
|
||||
recorded, not run. The socket belongs to the steward user alone (mode 0600);
|
||||
root may `show` and `list` orders but never issue one. An order that had
|
||||
started when the service died is recorded as lost and never run again. No order releases a brake. Acceptance and result are posted to
|
||||
`#operator` as a retried projection of `orders.cc`: a chat outage delays the
|
||||
posts and not the orders. A refusal names the field and a code, never the value
|
||||
it refused.
|
||||
|
||||
## Authority
|
||||
|
||||
**Owner.** Deployment admission, and the continuity of installed bodies.
|
||||
|
|
@ -208,79 +150,6 @@ omits one. Unknown fields are rejected on both sides. A repository that gets
|
|||
compromised can therefore change what is built, but not what it is allowed to
|
||||
touch on the host.
|
||||
|
||||
An upstream daemon that does not speak Idunn's runtime contract is deployed
|
||||
with `idunn-launch` as its executable artifact and the upstream tree as a
|
||||
required adjacent artifact; the launcher finds the child inside the release that
|
||||
holds it. The launcher is the unit's process: it binds the
|
||||
candidate endpoint and answers Idunn's challenge there, renders the child's
|
||||
config, starts a stateful child only once it holds the write lease, and signs
|
||||
presence. Members never pass through it: the binding's `[route.member]` makes
|
||||
Idunn route a second listener straight to the child, which binds the address
|
||||
Idunn gives it in `GAMECULT_IDUNN_MEMBER_BIND`.
|
||||
|
||||
```toml
|
||||
# recipe fragment — the upstream tree is built or fetched as an artifact
|
||||
[service]
|
||||
executable_artifact = "idunn-launch"
|
||||
arguments = [
|
||||
{ kind = "literal", value = "--child" },
|
||||
{ kind = "literal", value = "upstream/bin/daemon" },
|
||||
{ kind = "literal", value = "--template" },
|
||||
{ kind = "literal", value = "config.tmpl" },
|
||||
{ kind = "literal", value = "--health-path" },
|
||||
{ kind = "literal", value = "/health" },
|
||||
{ kind = "literal", value = "--health-contract" },
|
||||
{ kind = "literal", value = "example.presence-launcher.v1" },
|
||||
]
|
||||
required_environment = [
|
||||
"GAMECULT_IDUNN_RUNTIME_BUNDLE",
|
||||
"GAMECULT_IDUNN_CANDIDATE_BIND",
|
||||
"GAMECULT_IDUNN_MEMBER_BIND",
|
||||
]
|
||||
```
|
||||
|
||||
A directory artifact is sealed read-only: its files are 0444 and its
|
||||
directories 0555, whatever mode the build or git gave them. A file the tree
|
||||
must run, such as the launcher's `upstream/bin/daemon` child, is declared in
|
||||
`executable_members` and sealed 0555. The members are part of the sealed release
|
||||
address, so changing the list makes a new release. Idunn refuses the release if a
|
||||
member is absolute, climbs with `..`, is missing, is a symlink or sits behind
|
||||
one, or is not a regular file; `executable = true` on a directory output and
|
||||
`executable_members` on a file output are refused too.
|
||||
|
||||
A tree may hold relative symlinks whose `..` steps all come first and climb no
|
||||
higher than the artifact's top (`share/terminfo/4/4025ex -> ../t/tek4025ex` is
|
||||
kept). An absolute link, a `..` after a name, or a climb out of the artifact
|
||||
refuses the release when it is sealed. Host targets (Windows) record
|
||||
`executable` and `executable_members` in the receipt and do not apply them.
|
||||
|
||||
```toml
|
||||
# recipe fragment — the upstream artifact declares the child it launches
|
||||
[[artifacts]]
|
||||
id = "upstream"
|
||||
source_kind = "runner-output"
|
||||
runner = "build"
|
||||
source = "dist/upstream"
|
||||
destination = "upstream"
|
||||
executable_members = ["bin/daemon"]
|
||||
```
|
||||
|
||||
```toml
|
||||
# binding fragment — members connect to the member endpoint, which Idunn
|
||||
# forwards to the private port derived from the candidate: member
|
||||
# private_port_start + (candidate port - route private_port_start)
|
||||
[route]
|
||||
stable_endpoint = "tcp://10.77.0.1:8008"
|
||||
private_port_start = 18831
|
||||
private_port_end = 18839
|
||||
# ...
|
||||
|
||||
[route.member]
|
||||
stable_endpoint = "tcp://10.77.0.1:8448"
|
||||
private_port_start = 18841
|
||||
# max_connections = 128 # nginx limit_conn on the member listener (1-65535)
|
||||
```
|
||||
|
||||
## Contracts
|
||||
|
||||
Idunn's typed contracts are published by
|
||||
|
|
|
|||
|
|
@ -1,47 +0,0 @@
|
|||
[Unit]
|
||||
Description=Idunn steward orders: typed package, unit and deploy orders from the steward
|
||||
# Deliberately no After= or Requires= on any managed target, Synapse or
|
||||
# idunn-yggdrasil: orders are how an operator repairs those, so none of them
|
||||
# may be a condition of this unit starting. Posts are a retried projection and
|
||||
# wait for Synapse on their own.
|
||||
|
||||
[Service]
|
||||
Type=simple
|
||||
|
||||
# Root, from the same binary as idunn-yggdrasil, with its own store and socket.
|
||||
# idunn-yggdrasil keeps its no-imperative-input invariant and its write set:
|
||||
# no order code runs in `idunn serve`.
|
||||
#
|
||||
# The unit name matches idunn-*, which is in the always-refused set, so no
|
||||
# order can stop or restart this service.
|
||||
User=root
|
||||
# No Group= and no UMask=: the service gives its socket to the steward user
|
||||
# (mode 0600) itself and sets Idunn's own umask for the files it writes, so
|
||||
# nothing here reaches control.cc. The steward user is the only peer the
|
||||
# service answers: it checks SO_PEERCRED on every connection.
|
||||
ExecStart=/usr/local/bin/idunn orders serve \
|
||||
--store /var/lib/gamecult/idunn-orders \
|
||||
--state-store /var/lib/gamecult/idunn/control.cc \
|
||||
--steward-binding /etc/gamecult/idunn/steward.toml \
|
||||
--socket /run/idunn-orders/steward.sock
|
||||
Restart=always
|
||||
RestartSec=5
|
||||
TimeoutStopSec=20
|
||||
|
||||
RuntimeDirectory=idunn-orders
|
||||
RuntimeDirectoryMode=0755
|
||||
|
||||
# The Synapse access token of @idunn, copied from a root-only file, and read
|
||||
# through $CREDENTIALS_DIRECTORY; the service names it in steward.toml.
|
||||
LoadCredential=idunn-matrix-token:/etc/gamecult/idunn/matrix-token
|
||||
|
||||
PrivateTmp=yes
|
||||
ProtectHome=yes
|
||||
ProtectSystem=strict
|
||||
# Its own order store, and the control store a deploy order commits a command
|
||||
# to. Transient units started with systemd-run run in the manager's namespace,
|
||||
# so packages and units are not limited by this list.
|
||||
ReadWritePaths=/var/lib/gamecult/idunn-orders /var/lib/gamecult/idunn
|
||||
|
||||
[Install]
|
||||
WantedBy=multi-user.target
|
||||
|
|
@ -104,11 +104,6 @@ ReadWritePaths=/etc/gamecult/ghostlight/runtime /var/lib/gamecult/ghostlight-wor
|
|||
ReadWritePaths=/etc/gamecult/streampixels-service/runtime /var/lib/gamecult/streampixels-service /srv/streampixels-idunn/releases/service
|
||||
# StreamPixels web is stateless; it needs only its runtime bundle and releases.
|
||||
ReadWritePaths=/etc/gamecult/streampixels-web/runtime /srv/streampixels-idunn/releases/web
|
||||
# Vili (motion worker): runtime bundle, state, and releases. Prefixed "-" so a
|
||||
# target whose roots are not created yet cannot stop Idunn itself from starting
|
||||
# (Idunn startup never depends on a managed target). A root created after Idunn
|
||||
# starts is writable only from Idunn's next restart.
|
||||
ReadWritePaths=-/etc/gamecult/vili/runtime -/var/lib/gamecult/vili -/srv/vili-idunn/releases
|
||||
|
||||
[Install]
|
||||
WantedBy=multi-user.target
|
||||
|
|
|
|||
|
|
@ -55,10 +55,7 @@ A deployable repository publishes one strict declaration, normally
|
|||
- immutable artifact outputs, optionally pinned to an expected SHA-256;
|
||||
- a constrained launch contract;
|
||||
- one health-contract identity;
|
||||
- an optional typed state contract with state schema generation and slots
|
||||
(kinds `cultcache-file`, `external-file` and `external-directory`; the
|
||||
external kinds are process-owned state an external runtime imposes, which
|
||||
Idunn preserves and fences but never interprets);
|
||||
- an optional typed state contract with state schema generation and slots;
|
||||
- provided capabilities, dependency kinds, startup requirements, capacity, and
|
||||
conflicts.
|
||||
|
||||
|
|
@ -70,52 +67,6 @@ recipe fields.
|
|||
Stateless services declare no state block. Their operator binding therefore has
|
||||
no state root, state-transition policy, or process write-lease record.
|
||||
|
||||
## Foreign workloads: idunn-launch
|
||||
|
||||
An upstream daemon that cannot speak Idunn's runtime contract (Synapse, MAS)
|
||||
runs under `idunn-launch`, the unit's own process. Idunn's two-witness rule is
|
||||
unchanged: Idunn observes the unit, and the launcher, inside the unit and
|
||||
supervising the child, signs presence with the provider identity and the
|
||||
activation key.
|
||||
|
||||
- **Owner:** the launcher owns presence, config rendering and the child's
|
||||
lifetime inside the unit, and the route answers it signs for that child.
|
||||
Idunn owns admission, lease, route and continuity, as before.
|
||||
- **Inputs:** the runtime bundle, the activation credential and provider
|
||||
identity descriptors, the process write lease, the systemd credentials the
|
||||
binding names, and the child's health over TCP at the member bind.
|
||||
- **Outputs:** signed `gamecult.runtime_presence_health.v2` (Warming, Active
|
||||
and failed to Odin; route observations on the candidate endpoint), one child
|
||||
process, and a rendered config in the unit's private tmp that is never read
|
||||
back.
|
||||
- **Route:** Idunn owns both listeners of the route: the Expected route's
|
||||
stable endpoint (presence) and, when the binding has `[route.member]`, the
|
||||
member endpoint, rendered as a second server block in the one nginx fragment
|
||||
with a `limit_conn` on one per-route zone (`max_connections`, default 128), so
|
||||
members cannot exhaust the connections the presence block shares with them.
|
||||
The launcher binds the candidate endpoint and answers Idunn's challenge there,
|
||||
one connection at a time within a one-second deadline; any other first byte
|
||||
closes the connection. The answer never waits on the child: it reports the
|
||||
lifecycle's last probe, and a failed, missing or stale probe reads `degraded`.
|
||||
The child binds the member bind Idunn sets in
|
||||
`GAMECULT_IDUNN_MEMBER_BIND`, derived from the candidate port. Member bytes
|
||||
never touch the launcher.
|
||||
- **Lease:** with `write_lease_required` the launcher starts the child only
|
||||
after it holds the lease (bound to this Expected, this activation and one of
|
||||
its own Warming proofs) and keeps the lease store share-locked while the
|
||||
child lives. Losing the lease stops the child.
|
||||
- **Config:** one template from the release is rendered with the closed
|
||||
placeholder set `{{child.listen}}`, `{{child.listen_host}}`,
|
||||
`{{child.listen_port}}` (all three the member bind), `{{credential:NAME}}`,
|
||||
`{{credential_path:NAME}}` and `{{state_root}}`. Any other placeholder
|
||||
refuses to start.
|
||||
- **Forbidden:** member traffic through the launcher; the child binding the
|
||||
candidate endpoint; a child of a lease-requiring target running without a held lease; any presence not signed
|
||||
by both keys.
|
||||
- **State kind `external-directory`:** a process-owned directory imposed by an
|
||||
external runtime contract. Same writer, recovery and startup rules as
|
||||
`external-file`: it is lease-gated, preserved and fenced, never interpreted.
|
||||
|
||||
## Operator binding
|
||||
|
||||
The root-admitted operator binding supplies the physical policy a repository
|
||||
|
|
@ -222,40 +173,24 @@ health plus Ready correlation. It changes and observes route membership only
|
|||
afterward. Only after the proxy has adopted the candidate may Idunn clear the
|
||||
promotion fence and drain the incumbent.
|
||||
|
||||
A target has two slots: the incumbent (the admitted generation) and a
|
||||
transaction's candidate. Continuity owns the incumbent slot and keeps the
|
||||
admitted generation alive; it never stops. A live `DeploymentTransaction` owns
|
||||
its private candidate before Fencing and takes the target itself at Fencing (a
|
||||
stop-then-start rollout, at Starting) through one `take_target` CAS, so of two
|
||||
transactions entering from one view exactly one succeeds. A continuity
|
||||
restart is itself a transaction and takes the target the same way.
|
||||
|
||||
Phases from Warming through Committing carry deadlines, as do aborts and
|
||||
drains; an expired deadline never waits silently. A stateless deployment
|
||||
aborts to its incumbent. A deployment whose candidate was issued a write lease
|
||||
is left to the operator (`operator-required` incident, the second incident
|
||||
condition beside `continuity-exhausted`; its subject is the transaction id and
|
||||
it closes when the record ends or its step goes on). A continuity after
|
||||
Fencing retries its phase forward on a capped backoff and never stops its live
|
||||
candidate, since that is the only running copy of the admitted body; the
|
||||
deadline opens the same incident. The one exception is yielding: a continuity
|
||||
whose route proof has not succeeded gives way, through its own post-fencing
|
||||
abort, to a deployment the deployment brake has released and that awaits the
|
||||
handover; a continuity that has proven its route never yields. The
|
||||
`ContinuityExhausted` incident (six charged attempts in an hour,
|
||||
`restarts_page`) pages and gates nothing. When continuity next acts is derived
|
||||
only from its own record in `TargetSupervision` (schema v2): the attempt
|
||||
streak, the backoff anchor (written by the dead-and-free pass, the only
|
||||
writer), and the once-per-admission re-materialization bound. The workload
|
||||
driver alone answers whether a lost unit was a host reboot
|
||||
(`WorkloadPort::lost_to_host_reboot`). Operator-facing incidents and
|
||||
`idunn status` are projections that decide nothing. A boot lift returns a
|
||||
continuity abort left by an older Idunn, one that had not stopped its running
|
||||
candidate, to forward retry; if it cannot decide, every continuity abort is
|
||||
held until a lift succeeds. It is a migration with a deletion line (follow-up
|
||||
`c2c-delete-boot-lift`). Cut line: cuts 3 and 4 of `reincarnation-map.md`
|
||||
still owe dependency failures not growing the streak and the dead-generation
|
||||
route refresh.
|
||||
Ownership of a target is exclusive at each instant. A live
|
||||
`DeploymentTransaction` owns it from Fencing on (a stop-first rollout, from
|
||||
Starting); before Fencing a deployment owns only its private candidate and
|
||||
continuity owns the incumbent slot. Continuity owns the admitted generation's
|
||||
survival and never stops. There is no stopped state: six charged attempts in
|
||||
an hour (`restarts_page`) open the ContinuityExhausted incident and gate
|
||||
nothing, and no deployment path yields to continuity. When continuity next acts
|
||||
is derived only from its own record in `TargetSupervision` (schema v2, a v1
|
||||
record lifts at store open): the attempt streak, the backoff anchor (written
|
||||
by the dead-and-free pass, the only writer), and the once-per-admission
|
||||
re-materialization bound. The workload driver alone answers whether a lost unit
|
||||
was a host reboot (`WorkloadPort::lost_to_host_reboot`). Crash restart, restart
|
||||
after a sustained run and reboot re-materialization all mint through one CAS;
|
||||
continuity supervision never aborts a transaction. Operator-facing incidents
|
||||
and `idunn status` are projections that decide nothing. Cut line: cuts 2 to 4
|
||||
of `reincarnation-map.md` still owe continuity restarting alongside a
|
||||
pre-fencing build, dependency failures not growing the streak, and the
|
||||
dead-generation route refresh.
|
||||
|
||||
The deployment brake gates changes to source, release, configuration, schema,
|
||||
unit, or authority binding. The lifecycle brake separately gates restart of an
|
||||
|
|
|
|||
|
|
@ -1,394 +0,0 @@
|
|||
# Deployment provenance: map
|
||||
|
||||
Status: map, Imagination, 2026-10-23 (campaign `idunn-deploy-provenance`,
|
||||
session `self-2026-10-23-eureka-b`). The ends are in
|
||||
[`deploy-provenance-target.md`](deploy-provenance-target.md). Questions,
|
||||
rulings, cut specs and follow-ups live in Eureka's mind; this page holds what
|
||||
has no kind: body facts, rationale and sequencing. Line numbers in T1-T12 are
|
||||
at `d74ac8d` (origin/main on 2026-10-23); T4, T9 and T13-T16 are at `fd040f0`,
|
||||
the head of Hands attempt 1, which revision 2 of the spec builds on.
|
||||
|
||||
## Cut `tree-executable-members`
|
||||
|
||||
Spec `idunn-deploy-provenance:cut_spec:cut-tree-executable-members.r2`
|
||||
(revision 2, after Soul pass 1 found the real interpreter tree refused). It
|
||||
serves `motion-pipeline:follow_up:idunn-tree-executable-members` (Vili's
|
||||
admission waits on it) and will serve gamecult-chat C3 too: the presence
|
||||
launcher runs a child such as `upstream/bin/daemon` from inside a tree artifact
|
||||
(README, launcher fragment), and Synapse needs a Python interpreter there.
|
||||
|
||||
### Body facts
|
||||
|
||||
- T1. `ArtifactOutput` (`src/deployment.rs:106-117`) carries one `executable`
|
||||
bool. `ArtifactReceipt` (`src/deployment_plan.rs:731-737`) copies it and
|
||||
`validate` (`:740-760`) requires it to equal the declaration.
|
||||
- T2. `harden_installed_release` (`src/drivers.rs:6848-6855`) builds its
|
||||
executable set from `root.join(receipt.destination)` only. `harden_root_tree`
|
||||
(`:6858-6892`) seals every directory 0555 and every file 0444 unless that
|
||||
exact path is in the set. A directory artifact with `executable = true` is
|
||||
therefore silently meaningless today: its files are all 0444. Nothing refuses
|
||||
it. No live recipe declares one (CodexConnector, Ghostlight, Heimdall, Odin,
|
||||
StreamPixels service and web surveyed): `executable = true` is on files only.
|
||||
- T3. A tree output is copied by `copy_artifact`/`copy_tree`
|
||||
(`src/drivers.rs:8053-8126`), which recreate symlinks verbatim. The digest
|
||||
(`digest_tree`, `:8142-8175`) covers names, file bytes and link targets, not
|
||||
mode bits. So runner or git exec bits never reach the release: harden is the
|
||||
only mode authority.
|
||||
- T4. `validate_release_symlink` (`src/drivers.rs:6897-6916`) keeps a symlink
|
||||
only if its target is relative with no `..`. It runs only inside
|
||||
`harden_root_tree`, at install, so a tree with a `..` or absolute link copies,
|
||||
digests and seals, then fails at `install_release` (Soul s1: copying and
|
||||
digesting the real tree twice took 160 s before harden refused it). The rule
|
||||
is sound but too narrow: it refuses the real interpreter tree (T13). The
|
||||
frozen-source path has a physical resolver (`resolve_frozen_source_symlink`,
|
||||
`:6617`) that walks the filesystem with a 40-link budget and a memo. Its
|
||||
first version was exponential (Soul S4-1, recorded in `d562a12`), and it needs
|
||||
the tree on disk to answer. Revision 2 does not reuse it (T14).
|
||||
- T5. The sealed release id hashes `(schema, plan_id, artifacts,
|
||||
external_inputs)` (`src/deployment_plan.rs:1000-1011`), and `plan_id` hashes
|
||||
the whole plan including `recipe_blob`. A changed member list is therefore
|
||||
already a new plan and a new release. The receipt must still carry the list,
|
||||
because `harden_installed_release` reads receipts only; its two callers
|
||||
(`install_release` `:2788`, `validate_installed_release` `:2822`) re-harden
|
||||
every installed release on each observation.
|
||||
- T6. Encoding is `rmp_serde::to_vec`, positional. A trailing field with
|
||||
`#[serde(default, skip_serializing_if = "BTreeSet::is_empty")]` keeps every
|
||||
existing receipt's bytes and release id: the plan's `deadlines` field
|
||||
(`:574-577`) is the precedent.
|
||||
- T7. `digest_artifact` and `digest_tree` (`:8128-8175`) `fs::read` each file
|
||||
whole. A torch CPU tree's largest file (`libtorch_cpu.so`, hundreds of MB)
|
||||
is held in Idunn's memory three times per install (seal, install, every
|
||||
validate). `hash_frozen_source_tree` (`:6788-6840`) already streams in 64 KiB
|
||||
chunks (S9). The cut collapses both onto one streaming helper.
|
||||
- T8. `require_leaf_path` (`src/deployment.rs:1951-1958`) only keeps the
|
||||
artifact destination a single filename. Members are paths inside that
|
||||
destination, so it does not change. `require_relative_path` (`:1936-1949`)
|
||||
with `allow_current = false` already refuses absolute, `..`, root and prefix
|
||||
components.
|
||||
- T9. The host actuator re-spells the seal (`src/host.rs:1050-1086`): copy,
|
||||
mode check, digest, pinned-digest check and receipt, the same steps as
|
||||
`collect_artifact` (`src/drivers.rs:2305-2326`). Soul s1: deleting its mode
|
||||
check survives every test. Revision 2 collapses both onto one `seal_artifact`
|
||||
in `drivers.rs`. Each caller keeps only how it finds the source path.
|
||||
- T10. The transient unit runs with `ProtectSystem=strict` and the release in
|
||||
`ReadOnlyPaths` (`src/drivers.rs:3675-3698`), `/tmp` mounted `noexec`, and no
|
||||
`MemoryDenyWriteExecute`. Executing a 0555 file from the release works.
|
||||
Shared libraries need no exec bit.
|
||||
- T11. Live body on Yggdrasil: `/usr/local/bin/idunn` sha256 `44cf8a14...`,
|
||||
installed 2026-10-20 02:34 CEST, before `6e2a31c` (02:40 CEST). `src/` is
|
||||
identical between `8760a4f` and `6e2a31c`, so the live code is the
|
||||
reincarnation c2 merge. `origin/main` is 26 commits ahead (+4194/-112 in
|
||||
`src/`: the presence launcher and member route). Installing this cut ships all
|
||||
of them. Idunn's own body changes only through gamecult-ops
|
||||
`scripts/install-idunn-yggdrasil-release.sh` (staged binaries, unit,
|
||||
bindings). The stage at `/home/gamecultadmin/idunn-stage` is stale (2026-09-11).
|
||||
- T13. Link census of the real interpreter tree (Yggdrasil probe of
|
||||
`cpython-3.12.15+20261021-x86_64-unknown-linux-gnu-install_only.tar.gz`,
|
||||
sha256 `5ff9f060...77191124`; torch adds no links, as Soul s1 counted 1,049
|
||||
after pip): 1,049 links. 0 are absolute, 749 have no `..`, and 300 have all
|
||||
their `..` components before the first name (every one is
|
||||
`share/terminfo/<c>/<name> -> ../<c2>/<name2>`, one `..` each). 0 have a `..`
|
||||
after a name, 0 climb above the tree, 0 dangle and 0 resolve outside it.
|
||||
Blender 5.2.2 has 79 links, none absolute or with `..` (Soul s1).
|
||||
- T14. Lexically normalising `<link's directory>/<target>` does not check
|
||||
containment. The kernel resolves a target one component at a time, so a `..`
|
||||
after a symlinked name climbs from where that link points, not from where its
|
||||
name sits. Probe on Yggdrasil: `root/a/s -> ..` and `root/a/L -> s/../x`
|
||||
normalise to `root` and `root/a/x`, both inside, yet `L` reads the file `x`
|
||||
beside `root`. `root/a/d -> .` with `root/a/M -> d/../../x` escapes the same
|
||||
way. Idunn has met this before: verify-transaction-cut F5 (`D -> .`,
|
||||
`L -> D/D/../../../etc/passwd` passed a lexical check).
|
||||
The sound lexical rule constrains the target's shape: all its `..` components
|
||||
come before its first name, and they number no more than the link's depth
|
||||
below the artifact's top. The climb then starts from the link's own
|
||||
directory, which the walk reached through real directories only (copy and
|
||||
digest never descend through a link), so it lands on a real ancestor inside
|
||||
the artifact. After that the target only descends, and any link it passes
|
||||
through obeys the same rule. By induction on the kernel's hops (capped at 40,
|
||||
then ELOOP), every hop stays inside. The check is one pass over the target's
|
||||
components with no filesystem reads, so its cost is linear in the tree and no
|
||||
resolver is needed. The shapes it refuses (`s/../x`) occur in neither real
|
||||
tree (T13).
|
||||
Prior art, from documentation as known to Imagination (high confidence for Go
|
||||
and securejoin, medium for the tar details):
|
||||
- Go `path/filepath.IsLocal` (1.20) is lexical and says so: it does not account
|
||||
for symlinks. It would admit `s/../x`. Used alone, it is the T14 bug.
|
||||
- Go `os.Root` (1.24) and `cyphar/filepath-securejoin` (used by runc,
|
||||
containerd and umoci) resolve each component inside the root when a path is
|
||||
opened, refusing or clamping an escape. They guard one later access, need
|
||||
the tree on disk, and check again on every open. Idunn instead certifies a
|
||||
sealed tree once, so that the kernel's own resolution stays inside on every
|
||||
later access.
|
||||
- Zip-slip defences and archive extractors check entry names lexically and
|
||||
protect against link targets by refusing to write through links
|
||||
(libarchive's `ARCHIVE_EXTRACT_SECURE_SYMLINKS`, and `SECURE_NODOTDOT` for
|
||||
names; GNU tar creates such links last). They protect the extraction, not
|
||||
what a kept link points at afterwards.
|
||||
- OCI layer extraction keeps link targets verbatim, absolute ones included,
|
||||
because the runtime resolves them inside the container's root. Idunn runs a
|
||||
release in the host's mount namespace, so absolute targets stay refused.
|
||||
- Nix keeps link targets verbatim and tracks dependencies by scanning for
|
||||
references. The whole store is its root, so it has no per-artifact
|
||||
containment to compare with.
|
||||
None of these is a static per-link rule for a tree that runs unconfined. The
|
||||
T14 shape rule fits that case.
|
||||
- T15. Host targets: `idunn-host` observes, starts and stops processes on
|
||||
Windows only (the non-Windows `platform` module in `src/host.rs` bails), and
|
||||
`host.rs` sets no file modes. On Windows a file runs by its type, not by a
|
||||
mode bit, so `executable` and `executable_members` have nothing to set there.
|
||||
The file `executable` flag has always been carried and never applied on host
|
||||
targets. A recipe does not know which workload kind its binding will choose.
|
||||
- T16. The release root holds only artifact destinations. Runner workspaces,
|
||||
where external inputs land, are removed before `SealedRelease::new`
|
||||
(`src/drivers.rs:2421-2423`). `install_release` (`:2781-2788`) and
|
||||
`validate_installed_release` (`:2815-2822`) digest every receipt's
|
||||
destination before calling harden. So a rule in the digest walk runs at seal,
|
||||
at install and at every validate, over every link harden would see.
|
||||
- T12. Disk: `/` on Yggdrasil has 1.3 TB free. A tree release costs about three
|
||||
copies (runner workspace, staging, installed). Nothing collects old releases or
|
||||
staging (`ghostlight-verse:follow_up:idunn-generation-retention`), so every
|
||||
Vili deploy leaves about 7 GiB behind.
|
||||
|
||||
### Contract, corrected against the body
|
||||
|
||||
- Any directory artifact may declare `executable_members` (a set of relative
|
||||
paths). The follow-up limited it to runner-output; that limit protects no
|
||||
invariant, and the launcher's worktree or fetched trees need the same feature.
|
||||
- `executable = true` names a file artifact. On a directory output the release
|
||||
is refused, and `executable_members` on a file output is refused (T2).
|
||||
- A member is refused at recipe parse if it is empty, absolute, or holds `.`,
|
||||
`..`, a root or a prefix. At seal it is refused if it is missing, is a
|
||||
symlink, passes through a symlink or non-directory, or is not a regular file.
|
||||
Refusal happens before `SealedRelease::new`, so no sealed release can name a
|
||||
member it cannot seal.
|
||||
- Harden seals each member 0555 and fails if any declared executable path was
|
||||
not visited as a regular file. "Every declared executable is sealed 0555, or
|
||||
the install fails" holds structurally, not by a lookup that can miss.
|
||||
- Symlinks: a link in a tree artifact is kept when its target is relative, its
|
||||
`..` components all come before its first name, and they number no more than
|
||||
the link's depth below the artifact's top (T14). One function,
|
||||
`check_release_link`, decides this. Only the digest walk calls it, so a bad
|
||||
link refuses the release at seal, before `SealedRelease::new`, and again at
|
||||
install and validate (T16). Harden no longer judges links. The climb is
|
||||
bounded by the artifact's top, not the release root: no live release has a
|
||||
`..` link (the old rule refused them all), so nothing installed is newly
|
||||
refused, and each artifact's digest describes a closed tree.
|
||||
- Host targets carry `executable` and `executable_members` in the receipt,
|
||||
shape-checked at seal by the shared `seal_artifact`, and do not apply them
|
||||
(T15). Refusing members there would make a recipe's validity depend on its
|
||||
binding.
|
||||
- Vili's trees must
|
||||
not come from `python -m venv`: its `bin/python` is an absolute link and its
|
||||
`pyvenv.cfg` names the build workspace. Install packages into the relocatable
|
||||
interpreter tree instead (`motion-pipeline:follow_up:vili-relocatable-python-tree`).
|
||||
|
||||
### Authority
|
||||
|
||||
Owner: the seal (`seal_artifact`, called by `collect_artifact` for systemd
|
||||
targets and by `materialize` in `host.rs` for host targets) decides
|
||||
membership from the recipe and writes it into the receipt. Harden enforces the
|
||||
receipt and decides nothing. Runner exec bits and git `100755` modes are dead
|
||||
inputs for release modes. Link containment belongs to `check_release_link`,
|
||||
which only `digest_tree` calls. `validate_release_symlink` is deleted, and
|
||||
harden's symlink arm leaves links as they are.
|
||||
|
||||
### Acceptance sizing
|
||||
|
||||
The big-tree job (spec `verification.operator`) needs about 12 GiB of scratch
|
||||
on Yggdrasil: a 3.5 GiB tree, its staging copy, downloads and pip cache. It
|
||||
removes it at the end. Revision 2 runs it on the trees as shipped, with the 300
|
||||
terminfo links (T13). If a tree holds a link the T14 rule refuses, the job
|
||||
prints it and stops, and the link is reported to Self, not deleted.
|
||||
|
||||
## Cut `isolation-credential-evidence`
|
||||
|
||||
Spec `idunn-deploy-provenance:cut_spec:cut-cut-isolation-credential-evidence.r1`,
|
||||
at `112a7a5` (origin/main, live on Yggdrasil since 2026-10-24 11:29 UTC). It
|
||||
serves Vili's admission (`motion-pipeline:cut_report:cut-vili-admission.h4`,
|
||||
first deviation): every `idunn up vili` is refused before fencing.
|
||||
|
||||
### Body facts (probed read-only on Yggdrasil, 2026-10-24)
|
||||
|
||||
- I1. `idunn up vili` runs 6 and 7 failed with
|
||||
`systemd workload isolation properties differ from the Idunn contract`, the
|
||||
message of `observe_unit` (`src/drivers.rs:2975-3005`). Of its conjuncts,
|
||||
only one differs on the failed Vili unit: `LoadCredential`. Idunn expects
|
||||
`""` because Vili has no service credential besides the presence identity
|
||||
(`service_credential_names` empty); systemd shows `[unprintable]`. Every
|
||||
other conjunct matches, InaccessiblePaths included (one value across all
|
||||
five idunn-vili/odin units).
|
||||
- I2. systemd 257.13 `systemctl show -p LoadCredential` prints
|
||||
`LoadCredential=[unprintable]` for every unit, whatever the credentials:
|
||||
`ssh.service` and `systemd-journald.service` (not DynamicUser, zero
|
||||
LoadCredential), the four failed `idunn-vili-*` units (zero), and the Odin
|
||||
unit (one). The display form carries no information, so the conjunct was
|
||||
vacuous whenever credentials existed and always false when none did. Odin
|
||||
and StreamPixels passed only because each carries a regular credential.
|
||||
- I3. The exact value is readable: D-Bus
|
||||
`org.freedesktop.systemd1.Service.LoadCredential` (`busctl get-property`)
|
||||
answers `a(ss) 0` for the Vili units and ssh, `a(ss) 1
|
||||
"ODIN_TOPOLOGY_IDENTITY" "<path>"` for Odin.
|
||||
- I4. What the workload can actually read is the delivered directory. On the
|
||||
host, `/run/credentials/` holds a directory only for units with credentials
|
||||
(getty, journald, Odin, StreamPixels service and web). Inside Odin's MainPID
|
||||
root, `/run/credentials/<unit>/` holds exactly `ODIN_TOPOLOGY_IDENTITY`
|
||||
(`-r--r-----+ root root`). The presence identity is a parent-only `OpenFile`
|
||||
descriptor, never a credential.
|
||||
- I5. The rule arrived whole in `1fc0dad` (2026-09-03, "Rebuild Idunn
|
||||
deployment transaction authority"), with its twin clause in
|
||||
`validate_launch_observation` (`:3468-3472`, `:3513-3516`); no later commit
|
||||
touched it. No ruling or finding in any campaign concerns workload isolation
|
||||
verification; gamecult-chat's foreign-workloads cut touched
|
||||
`CREDENTIALS_DIRECTORY` reservation, not this check.
|
||||
- I6. `SystemdWorkloadObservation` is persisted by previous Idunns in both
|
||||
MessagePack encodings, compact (positional) included
|
||||
(`persisted_systemd_observations_decode_as_the_systemd_variant`,
|
||||
`:9915-9930`), so deleting the `load_credential` field shifts every later
|
||||
slot of a live incumbent's record. The field is also in the drift list
|
||||
(`:2462`) that `stop` (`:4064-4070`) uses to refuse stopping a changed
|
||||
workload: changing what the field holds without leaving that list would make
|
||||
the next Idunn refuse to stop the live Odin and StreamPixels incumbents.
|
||||
- I7. Fixtures: the verbatim 257.13 `systemctl show` output of base's 27
|
||||
properties for a Vili unit (presence identity only) and the Odin unit
|
||||
(presence identity plus one service credential), captured 2026-10-24 into
|
||||
this Imagination's scratchpad `fixtures/`.
|
||||
- I8. The new rule holds for every live systemd incumbent: inside each
|
||||
MainPID root the delivered entries equal its binding's `secret_files` keys
|
||||
minus `GAMECULT_RUNTIME_PRESENCE_IDENTITY` (Odin 1, StreamPixels service 11,
|
||||
StreamPixels web 1; key names compared, no values read). An Idunn restart onto
|
||||
this cut re-observes them (`:3955-3970`) and passes.
|
||||
|
||||
### Decision
|
||||
|
||||
A property systemd cannot display does not belong in the comparison. The
|
||||
invariant it reached for, that the workload holds exactly the credentials Idunn
|
||||
delivered, is observed where it is decided: the entries of
|
||||
`/run/credentials/<unit>` in the MainPID's root equal Idunn's service
|
||||
credential names, absent directory equal to none. That covers
|
||||
`LoadCredential`, `SetCredential`, `ImportCredential` and drop-ins alike, and
|
||||
needs no new program. The D-Bus read (I3) is exact but sees only one of those
|
||||
properties and adds a second systemd client and parser; rejected.
|
||||
|
||||
### Authority
|
||||
|
||||
Owner: `observe_unit` decides the credential rule, from the delivered
|
||||
directory. `LoadCredential`'s display form is no longer read or compared; the
|
||||
`load_credential` field is a retired wire slot, written empty, out of the drift
|
||||
list, kept only so persisted records decode. Forbidden writers: the
|
||||
`observe_unit` conjunct and the `validate_launch_observation` clause. Dropping
|
||||
the slot waits on the next schema revision of the observation.
|
||||
|
||||
### Deploy
|
||||
|
||||
Idunn's body changes only through gamecult-ops
|
||||
`scripts/install-idunn-yggdrasil-release.sh` (stage `idunn` only; unit and
|
||||
bindings unchanged). After it, Vili's Hands reruns from step 4 of
|
||||
`motion-pipeline:cut_report:cut-vili-admission.h4`'s undone list.
|
||||
|
||||
## Cut `runner-failure-evidence`
|
||||
|
||||
Spec `idunn-deploy-provenance:cut_spec:cut-runner-failure-evidence.r1`, at
|
||||
`804ed75` (origin/main, live on Yggdrasil). It serves the StreamPixels-area
|
||||
Self: `idunn up vili` (command `up-4866467c-ee98-4a2b-b4a9-a008d7c44acd`,
|
||||
transaction `tx-a04b68c7-c742-4d7d-8220-8aa08b9febae`, 2026-10-25 00:50-00:55
|
||||
UTC) failed in Sealing with `Docker runner exited with exit status: 1: ` and
|
||||
nobody could tell which recipe step failed.
|
||||
|
||||
### Body facts (source read at `804ed75`; host read-only, 2026-10-25)
|
||||
|
||||
- R1. The runner is `docker run --rm` (`src/drivers.rs:2088-2090`), launched by
|
||||
`DockerRunnerDriver::docker` (`:2173-2195`) with stdin null and
|
||||
`Command::output()`: stdout and stderr are each buffered whole in the
|
||||
daemon's memory, unbounded. On a non-zero exit the error is
|
||||
`"Docker runner exited with {status}: {stderr}"`; stdout is dropped. The
|
||||
container is gone the moment it exits, so `docker ps -a` and `docker logs`
|
||||
hold nothing.
|
||||
- R2. The empty message: the failing process wrote nothing to stderr, and
|
||||
stdout, the only stream that carried its report, is discarded at
|
||||
`:2182-2193`. The same interpolation echoes whatever a step does write to
|
||||
stderr into `last_error`, `pre_fencing_abort.error` and
|
||||
`completion.error` (`bounded_detail`, 2048 bytes), and from there into
|
||||
`idunn up`'s `deployment failed:` line and `idunn status`.
|
||||
- R3. No step identity reaches the error. `materialize` (`:2368-2391`) loops
|
||||
over `declaration.steps` and calls `run_in_workspace` with argv only;
|
||||
`step.id` is used only in the environment precondition. External inputs
|
||||
(`curl` in the runner, `:2247-2258`) fail through the same helper with the
|
||||
same anonymous message.
|
||||
- R4. Which step failed on 2026-10-25, inferred from the failed transaction's
|
||||
staging root, which survives (926 MiB,
|
||||
`/var/lib/gamecult/idunn/staging/tx-a04b68c7-.../`): in `.runner-node-build`,
|
||||
`node_modules/.package-lock.json` was written at 00:51:24 (so
|
||||
`install-vili-dependencies`, `npm ci`, finished), nothing outside
|
||||
`node_modules` is newer, and `release/vili` (the next step's output) is
|
||||
absent; the python workspace holds only fetched inputs. The failing step was
|
||||
`test-daemon` (`node --test ... scripts/test-vili-daemon.mjs`), whose test
|
||||
reporter writes to stdout: exactly the stream R1 discards. Inference from
|
||||
residue, not from output.
|
||||
- R5. The failure path: `resume_one_transaction` (`src/control_plane.rs:6291-6320`)
|
||||
sends a pre-fence error to `begin_pre_fencing_abort` (`:9916-9935`), which
|
||||
persists `bounded_detail(format!("{error:#}"))`. Nothing reaches the journal
|
||||
on this path: the idunn-yggdrasil journal for 00:45-01:10 holds only
|
||||
provider-exclusion lines.
|
||||
- R6. Stores. `control.cc` (root 0640) holds commands, transactions
|
||||
(`idunn.deployment_transaction.v4`, keys 0-36), generations, supervision and
|
||||
expiry requests; `ControlSnapshot::read` refuses any other type
|
||||
(`:3420`). Records must re-encode byte-exactly (`decode_record`,
|
||||
`:3717-3727`), so a new transaction field is a schema bump and a v4 lift
|
||||
(`:4100-4110`). Terminal transactions move verbatim to `history.cc`
|
||||
(`:4366-4396`; 39.8 MB on the host), which status reads for `--command`.
|
||||
`incidents.cc` lives in the published projection directory and by its own
|
||||
doc (`src/incident.rs:1-11`) carries no free text and is never `control.cc`,
|
||||
because a type added there stops continuity for every target on a rollback.
|
||||
- R7. The daemon runs as root (`User=root`). Runner secrets reach the
|
||||
container as read-only file mounts named by env (`:2041-2047`), so any text
|
||||
a step derives from them can land in its output.
|
||||
- R8. Staging residue: 475 transaction roots, 55 GiB under
|
||||
`/var/lib/gamecult/idunn/staging`; a failed Sealing leaves its workspaces
|
||||
because `materialize` returns before `remove_tree_inside`.
|
||||
|
||||
### Decision
|
||||
|
||||
The failure becomes a typed value at the runner, `RunnerStepFailure`: the
|
||||
subject (recipe step or external input, by id), the runner id, the exit code or
|
||||
signal, and the last 8 KiB of stdout and of stderr with each stream's total
|
||||
byte count. Its `Display` carries the subject and status, never output.
|
||||
Capture streams both pipes through bounded tails, so a verbose build no longer
|
||||
buffers whole in the daemon.
|
||||
|
||||
The tail lives in a new private store, `runner-evidence.cc`, beside
|
||||
`control.cc` (mode 0600), one record per transaction
|
||||
(`idunn.runner_failure.v1`), newest 32 kept by the writer. Not in the
|
||||
transaction record: that is a v5 bump whose rollback makes every older Idunn
|
||||
refuse its control store (R6), the scar `incident.rs` already names. Not in
|
||||
`incidents.cc`: published, and free text is barred there. Not in the journal:
|
||||
journal readers are a wider set than root, so the journal line names the
|
||||
transaction, the subject, the status and the store, and no output.
|
||||
|
||||
No redaction. Matching secret bytes would make the daemon read every secret,
|
||||
and still misses any encoding, prefix or transform of one; a claimed redaction
|
||||
that leaks is worse than a stated rule. The rule instead is the reader set: the
|
||||
store's only readers (root) can already read every secret the runner was
|
||||
given.
|
||||
|
||||
Keeping the failed container (dropping `--rm`) is worse: its output then lives
|
||||
in Docker's json-file log, unbounded, untyped, outside Idunn's state and
|
||||
readable by the docker group; containers accumulate with no owner to reap them;
|
||||
`docker system prune` erases the evidence; and nothing ties a container to the
|
||||
transaction in `idunn status`.
|
||||
|
||||
### Authority
|
||||
|
||||
Owner: the runner decides what failed and keeps the bounded tail; the engine's
|
||||
pre-fence failure path is the one writer of `runner-evidence.cc`.
|
||||
`last_error`, `pre_fencing_abort.error` and `completion.error` keep only the
|
||||
failure's `Display`. Forbidden writers: the `stderr` interpolation in
|
||||
`DockerRunnerDriver::docker`. Retention is by count inside the writer, so the
|
||||
store needs no external owner; staging roots (R8) stay with
|
||||
`ghostlight-verse:follow_up:idunn-generation-retention`.
|
||||
|
||||
### Deploy
|
||||
|
||||
Idunn's body changes only through gamecult-ops
|
||||
`scripts/install-idunn-yggdrasil-release.sh` (stage `idunn` only; unit and
|
||||
bindings unchanged). The next failed Sealing then shows its step and tail on
|
||||
`sudo idunn status --command <id>`.
|
||||
|
|
@ -313,36 +313,10 @@ written when the pass first finds the admitted generation dead and the target
|
|||
free, and a capped doubling wait from that anchor (5 s base, 300 s cap). A run
|
||||
of 600 s or more by the admitted body resets the streak. A generation lost to a
|
||||
host reboot (the workload driver's evidence, `lost_to_host_reboot`) is
|
||||
re-materialized once per admission without charge or wait.
|
||||
|
||||
Each target has two slots, the incumbent and a transaction's candidate. The
|
||||
incumbent is continuity's until a transaction takes the target, at Fencing (at
|
||||
Starting for a stop-then-start rollout), through one `take_target` CAS; of two
|
||||
transactions entering from one view exactly one succeeds. Before that, a
|
||||
deployment owns only its private candidate and the target is free for it
|
||||
between continuity attempts.
|
||||
|
||||
Phases Warming through Committing, post-fencing aborts and drains all carry
|
||||
deadlines (the plan's frozen `PhaseDeadlines`; `WARMING_DEADLINE_SECONDS`, and
|
||||
`PRE_FENCING_ABORT_DEADLINE_SECONDS` for a pre-fencing abort). What an expired
|
||||
post-fencing deadline does depends on the owner. A deployment whose candidate
|
||||
was never issued a write lease aborts to the incumbent. One whose candidate was
|
||||
issued a lease may have written under a new state contract and is left to the
|
||||
operator. A continuity retries its phase forward on a capped backoff and never
|
||||
stops its live candidate, because that candidate is the only running copy of
|
||||
the admitted body. Both operator cases open an `operator-required` incident
|
||||
(subject: the transaction id) that closes when the record ends or its step goes
|
||||
on.
|
||||
|
||||
The one yield: a continuity whose route proof has not succeeded gives way to a
|
||||
deployment the deployment brake has released that awaits the handover of the
|
||||
target. It begins its own post-fencing abort and the deployment fences what is
|
||||
admitted. A continuity that has proven its route is serving never yields.
|
||||
|
||||
At boot, a lift returns to forward retry any continuity abort an older Idunn
|
||||
left over a running candidate. If the lift errs or cannot observe the
|
||||
candidate, the abort is held, not run. The lift is a migration with a deletion
|
||||
line (follow-up `c2c-delete-boot-lift`).
|
||||
re-materialized once per admission without charge or wait. The target is free
|
||||
for a deployment's candidate between attempts, and a deployment never yields to
|
||||
continuity before Fencing; from Fencing on (or from Starting for a stop-first
|
||||
rollout) the live transaction alone owns the target.
|
||||
|
||||
An admitted generation keeps its Ready and latest Odin receipts as durable
|
||||
proof of the exact activation at their recorded admission times. Dependency
|
||||
|
|
|
|||
|
|
@ -97,8 +97,7 @@ Each of these was found by running the pipe against Raven, not by reading.
|
|||
bind time. `rollout.strategy = "stop-then-start"` exists for exactly a
|
||||
route-less host-actuator workload: the admitted incumbent is stopped at
|
||||
Starting, before the candidate starts, and supervision leaves the target
|
||||
alone from Starting on: the transaction takes the target there (one
|
||||
`take_target` CAS) rather than at Fencing. A systemd-transient workload may not declare it.
|
||||
alone from Starting on. A systemd-transient workload may not declare it.
|
||||
- **An unobservable host is silence.** The hub returns a typed
|
||||
`HostUnobservable` for not-attached, detached, reconnected and unanswered.
|
||||
Supervision skips that tick and logs it. Before this, an Idunn restart
|
||||
|
|
|
|||
|
|
@ -575,26 +575,3 @@ its stores and `idunn.running` together, because the cb60965 binary cannot read
|
|||
"migrated 5 control record(s) … idunn.target_supervision.v2". streampixels-service
|
||||
kept running through the restart (unit 73ca40ba, PID 968752, NRestarts 0). The
|
||||
catalog returned 200, and 4443/udp was listening.
|
||||
|
||||
## Installed: cut 2
|
||||
|
||||
Cuts c2-phase-split, c2b-owner-deadlines, c2c-post-fence-forward and
|
||||
c2d-continuity-yields (branch tip `8760a4f`) were installed on Yggdrasil on
|
||||
2026-10-20 at 00:34:06 UTC, with the operator's OK.
|
||||
|
||||
- **The hotfix before it.** `ec45906` (`bounded_detail`) was installed on its
|
||||
own on 2026-10-19 at 17:32 UTC. It unwedged tx-f978ab86.
|
||||
- **Build.** `/srv/build/idunn-8760a4f`; `cargo test --lib` passed 471.
|
||||
- **Backup.** `/root/idunn-store-backup-*-pre-reincarnation-c2` holds the
|
||||
stores, `incidents.cc` and the binaries. A rollback must restore them
|
||||
together: older binaries refuse two live transactions on one target, and
|
||||
they refuse the `operator-required` incident condition.
|
||||
- **First boot.** The boot lift returned exactly tx-039b91e9 (ghostlight) and
|
||||
tx-4eaed669 (odin) to forward retry, as the Soul install gate predicted.
|
||||
Odin's continuity, wedged in Routing since the 2026-10-18 reboot, committed
|
||||
4 s later. Generation tx-4eaed669 is admitted, its route is healthy, and the
|
||||
same PID 2571 still runs. Ghostlight's continuity is retrying forward in
|
||||
AwaitingReady. Its daemon's readiness is a ghostlight-side fault
|
||||
(`Permission denied` reading its session-wrapping key).
|
||||
- **Live processes.** Every live MainPID was the same before and after the
|
||||
install. The streampixels catalog returned 200, and 4443/udp was listening.
|
||||
|
|
|
|||
|
|
@ -52,18 +52,12 @@ web app and service through Idunn v2. The live evidence is on Yggdrasil:
|
|||
the admitted incarnation. The proof answers an Idunn challenge and is signed
|
||||
with the credential Idunn issued to that launch. Invariant 3 changes where the
|
||||
proof travels, not what it proves.
|
||||
5. **Every post-fencing phase has a deadline and is never a silent wait.** A
|
||||
deployment's stateless phase that passes its deadline aborts to its
|
||||
incumbent. A continuity past its deadline is left to the operator
|
||||
(operator-required incident) and retries its phase forward on a capped
|
||||
backoff, because aborting it would stop the only running copy of the admitted
|
||||
body (ruling post-fence-forward-retry), until a deployment that the
|
||||
deployment brake has released awaits the handover of its target and the
|
||||
continuity's route proof has not succeeded: then it yields through its own
|
||||
post-fencing abort, and the deployment fences what is admitted (rulings
|
||||
continuity-yields-to-deploy, c2d-yield-only-unproven-to-movable). A
|
||||
continuity that has proven its route is serving and never yields. A
|
||||
stateful deployment is left to the operator.
|
||||
5. **Every post-fencing phase ends.** Each phase after fencing has an admitted
|
||||
deadline. When a deadline expires, Idunn resolves it: stateless targets abort
|
||||
to the incumbent. Stateful targets get an explicit terminal record with a
|
||||
named recovery. A granted lease that the process does not pick up within its
|
||||
window is a named failure with an Idunn-owned recovery. It is never a silent
|
||||
wait. Resumable-forever is gone.
|
||||
6. **The admitted receipt describes the running incarnation.** Continuity
|
||||
commits update the admitted generation atomically, so the rollback path never
|
||||
meets drift between the receipt and the projection. Rollback demotes the exact
|
||||
|
|
|
|||
|
|
@ -21,8 +21,8 @@ probes ran on Yggdrasil over `ssh ygg`, read-only, 2026-09-30 13:51-13:55 UTC
|
|||
| I2 | Continuity exhaustion (since `idunn-reincarnation` cut 1: `restarts_page`, a paging threshold only; nothing stops, and the line numbers below are pre-cut) | `:68-71` (6 restarts / 3,600,000 ms, 5 s doubling), `:1066-1072` (`restarts_used`, `restarts_exhausted`), decided `:5709-5722` under key `continuity:<target>` (`:5678`) | Exhaustion is **derived** from `TargetSupervision.continuity_restarts` (newest 6 times, sliding window). No write marks it. The fault clears (`:5723`) as soon as one entry leaves the window, so a permanently dead target leaves and re-enters exhaustion several times each hour. |
|
||||
| I3 | Route degradation | `RouteSupervisionState` `:890-942`; `record_failed_challenge` sets `degraded_since_unix_millis` on the **first** failed challenge, `record_proved_challenge` clears it; reported at `:5975` under `route:<target>` | Durable per incarnation (inside `admitted_generation.v4`); a new generation starts from `default()`. Backoff doubles from the topology max age to a cap. |
|
||||
| I4 | Other operator-facing faults | `:5645-5651` (`generation-down:<target>`, "holds a dead admitted generation down"), `:6016` (`route-record:<target>`), `note_fault` per transaction | Also stderr plus `last_error`. Not named by the target; listed so the paging-set question sees them. |
|
||||
| I5 | `OperatorRequired` on main (pre-cut-2 reading; since `idunn-reincarnation` cut 2 `resolve` reaches it in production through `PhaseEnd::OperatorRequired` and `PhaseEnd::RetryForward`, both calling `leave_to_operator`, which opens the `operator-required` incident `IncidentCondition::OperatorRequired`, subject the transaction id; `close_spent_operator_incidents` closes it when the record no longer `leaves_to_operator`) | `TerminalRecovery::OperatorRequired` `:878-885`, validated `:1954`; constructed only in tests (`:12525`, `:12535`) | No production path produces it on main. |
|
||||
| I6 | `OperatorRequired` on B5 (superseded by cut 2: B5's deadlines merged as `PhaseDeadline`, and a continuity past its deadline now also lands here) | `hands/b5` `resolve_phase_deadline` (diff hunk at `control_plane.rs` ~8560-8625 on the branch), `PhaseEnd::OperatorRequired` | Deadline passed on a stateful deploy with an issued lease: `report_once("deadline:<transaction_id>")`, returns `Ok(None)`, the transaction stays live. No terminal transition and no operator action that ends it were found on the branch (`idunn expire` writes `idunn.expiry_request.v1`, which re-enters the same arm). |
|
||||
| I5 | `OperatorRequired` on main | `TerminalRecovery::OperatorRequired` `:878-885`, validated `:1954`; constructed only in tests (`:12525`, `:12535`) | No production path produces it on main. |
|
||||
| I6 | `OperatorRequired` on B5 | `hands/b5` `resolve_phase_deadline` (diff hunk at `control_plane.rs` ~8560-8625 on the branch), `PhaseEnd::OperatorRequired` | Deadline passed on a stateful deploy with an issued lease: `report_once("deadline:<transaction_id>")`, returns `Ok(None)`, the transaction stays live. No terminal transition and no operator action that ends it were found on the branch (`idunn expire` writes `idunn.expiry_request.v1`, which re-enters the same arm). |
|
||||
| I7 | Store kinds | `:55-61`, `:936-966`; B5 adds `idunn.expiry_request.v1` | `control.cc`: `idunn.deployment_command`, `idunn.deployment_transaction`, `idunn.admitted_generation.v4`, `idunn.target_supervision.v1`. `history.cc` beside it holds terminal transactions (`:3653`), retired one per tick (`:5068-5091`). |
|
||||
| I8 | Single-writer precedent inside `control.cc` | B5 `ExpiryRequest` doc comment | "The CLI only writes this record. The daemon's deadline resolver reads it ... No CLI writes a transaction field." One writer per kind. |
|
||||
| I9 | Serve loop | `:5041-5058`, `run_scheduler_tick` `:5098-5110` | 500 ms poll. A tick error is logged, never propagated. |
|
||||
|
|
@ -166,10 +166,9 @@ cut spec fixes them.
|
|||
|
||||
Empty cells, stated plainly:
|
||||
|
||||
- `operator-required` closure: the incident closes when the transaction ends or
|
||||
its step goes on (`close_spent_operator_incidents`). A stateful deployment
|
||||
left to the operator still has no Idunn-owned terminal action; the operator's
|
||||
`idunn expire` re-enters the same arm.
|
||||
- `operator-required` closure: no path on `hands/b5` ends a transaction left to
|
||||
the operator (I6). The incident can open and never close until B5 names that
|
||||
action.
|
||||
- Delivery-failure visibility: no owner.
|
||||
- Discord `enforce_nonce` window: not verified against Discord's current docs
|
||||
in this pass.
|
||||
|
|
|
|||
|
|
@ -30,12 +30,9 @@ journald (`report_once`, `control_plane.rs:4599`):
|
|||
- An admitted route is degraded. It fails its stable-route challenge
|
||||
(`supervise_admitted_route`, ~5973). The degraded mark is durable, but no
|
||||
one is told.
|
||||
- A post-fencing phase deadline leaves a transaction to the operator: a
|
||||
stateful deployment whose candidate was issued a lease, or a continuity
|
||||
retrying forward past its deadline. Since `idunn-reincarnation` cut 2 it is
|
||||
the second incident condition, `operator-required` (subject: the transaction
|
||||
id), beside `continuity-exhausted`; it closes when the record ends or its
|
||||
step goes on.
|
||||
- A post-fencing phase deadline expires into `OperatorRequired`. This lives on
|
||||
B5, which is unmerged on `hands/b5`. The failure record is durable and the
|
||||
notice is a log line.
|
||||
|
||||
Odin's own store condition (`odin.store_condition.v1`, `contested`) is mapped
|
||||
but not built. Idunn reads nothing about Odin's store health today.
|
||||
|
|
@ -88,8 +85,8 @@ the machine has a detector with no record, and a deliverer that nothing feeds.
|
|||
- Odin's store-recovery cuts, O2 included. They produce `odin.store_condition`
|
||||
and are Odin's campaign, still in prose. This campaign consumes the record
|
||||
once it exists.
|
||||
- Idunn's phase deadlines. They landed in `idunn-reincarnation` cut 2, which
|
||||
also owns the `operator-required` incident condition.
|
||||
- Idunn's B5 phase deadlines. B5 is in prose on `hands/b5`. This campaign
|
||||
reports its `OperatorRequired` outcome once B5 merges.
|
||||
- VoidBot's `notify_owner` MCP tool and its separate DM code.
|
||||
- Bifrost's Persona delivery crossing.
|
||||
- Eve/TUI lowering of incidents.
|
||||
|
|
|
|||
|
|
@ -1,96 +0,0 @@
|
|||
//! The unit's process for an upstream daemon: parse the recipe's argv, read
|
||||
//! what Idunn set, run the launcher, and map the outcome to an exit code.
|
||||
#[cfg(unix)]
|
||||
fn main() -> std::process::ExitCode {
|
||||
use std::sync::atomic::{AtomicBool, Ordering};
|
||||
|
||||
use anyhow::Context;
|
||||
use idunn_daemon::deployment::{
|
||||
IDUNN_PROCESS_WRITE_LEASE_ENVIRONMENT, IDUNN_RUNTIME_BUNDLE_ENVIRONMENT,
|
||||
IDUNN_RUNTIME_CANDIDATE_BIND_ENVIRONMENT, IDUNN_RUNTIME_MEMBER_BIND_ENVIRONMENT,
|
||||
};
|
||||
use idunn_daemon::launcher::{self, *};
|
||||
|
||||
static STOP: AtomicBool = AtomicBool::new(false);
|
||||
extern "C" fn on_signal(_: libc::c_int) {
|
||||
STOP.store(true, Ordering::SeqCst);
|
||||
}
|
||||
|
||||
fn launch() -> anyhow::Result<Outcome> {
|
||||
let release_root = std::env::current_exe()?
|
||||
.parent()
|
||||
.context("the launcher is not inside a release")?
|
||||
.to_owned();
|
||||
let contract = LaunchContract::parse(release_root, std::env::args().skip(1))?;
|
||||
let bundle = std::env::var_os(IDUNN_RUNTIME_BUNDLE_ENVIRONMENT)
|
||||
.context("the Idunn runtime bundle is not set")?;
|
||||
let bind = std::env::var(IDUNN_RUNTIME_CANDIDATE_BIND_ENVIRONMENT)
|
||||
.context("the Idunn candidate bind is not set")?;
|
||||
let member = launcher::parse_member_bind(
|
||||
&std::env::var(IDUNN_RUNTIME_MEMBER_BIND_ENVIRONMENT)
|
||||
.context("the Idunn member bind is not set")?,
|
||||
&bind,
|
||||
)?;
|
||||
let (activation, provider) = launcher::inherited_authority_files()?;
|
||||
let authority = RuntimeAuthority::open(bundle.as_ref(), activation, provider)?;
|
||||
authority.require_candidate_bind(&bind)?;
|
||||
let lease_path = std::env::var_os(IDUNN_PROCESS_WRITE_LEASE_ENVIRONMENT);
|
||||
anyhow::ensure!(
|
||||
lease_path.is_some() == authority.expected().write_lease_required,
|
||||
"the process write lease binding differs from Expected"
|
||||
);
|
||||
let route = std::net::TcpListener::bind(&bind).context("binding the candidate endpoint")?;
|
||||
let sink = OdinRudpSink::from_expected(authority.expected())?;
|
||||
let lease = DriverLeaseReader::new(
|
||||
&authority.expected().target,
|
||||
lease_path.as_deref().unwrap_or_default().as_ref(),
|
||||
);
|
||||
let health_path = contract.health_path.clone();
|
||||
let ports = Ports {
|
||||
credentials: std::sync::Arc::new(CredentialsDirectory::from_environment()),
|
||||
clock: std::sync::Arc::new(SystemClock),
|
||||
sink: std::sync::Arc::new(sink),
|
||||
lease: std::sync::Arc::new(lease),
|
||||
spawner: Box::new(ProcessSpawner),
|
||||
health: Box::new(move |member| {
|
||||
std::sync::Arc::new(HttpHealthProbe {
|
||||
member,
|
||||
path: health_path.clone(),
|
||||
})
|
||||
}),
|
||||
};
|
||||
// SAFETY: the handler only stores to an atomic.
|
||||
unsafe {
|
||||
libc::signal(libc::SIGTERM, on_signal as *const () as libc::sighandler_t);
|
||||
libc::signal(libc::SIGINT, on_signal as *const () as libc::sighandler_t);
|
||||
}
|
||||
Ok(launcher::run(
|
||||
&contract,
|
||||
authority,
|
||||
route,
|
||||
member,
|
||||
&std::env::temp_dir(),
|
||||
ports,
|
||||
&STOP,
|
||||
))
|
||||
}
|
||||
|
||||
match launch() {
|
||||
Ok(outcome) => {
|
||||
if let Outcome::Failed(code) = &outcome {
|
||||
eprintln!("idunn-launch ended: {code}");
|
||||
}
|
||||
std::process::ExitCode::from(outcome.exit_code() as u8)
|
||||
}
|
||||
Err(error) => {
|
||||
eprintln!("idunn-launch refused: {error}");
|
||||
std::process::ExitCode::from(2)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
#[cfg(not(unix))]
|
||||
fn main() {
|
||||
eprintln!("idunn-launch runs only on Unix");
|
||||
std::process::exit(2);
|
||||
}
|
||||
5192
src/control_plane.rs
5192
src/control_plane.rs
File diff suppressed because it is too large
Load diff
|
|
@ -9,27 +9,12 @@ pub const TARGET_DECLARATION_SCHEMA: &str = "gamecult.idunn.target_declaration.v
|
|||
pub const OPERATOR_BINDING_SCHEMA: &str = "gamecult.idunn.operator_binding.v2";
|
||||
pub const IDUNN_RUNTIME_BUNDLE_ENVIRONMENT: &str = "GAMECULT_IDUNN_RUNTIME_BUNDLE";
|
||||
pub const IDUNN_RUNTIME_CANDIDATE_BIND_ENVIRONMENT: &str = "GAMECULT_IDUNN_CANDIDATE_BIND";
|
||||
pub const IDUNN_RUNTIME_MEMBER_BIND_ENVIRONMENT: &str = "GAMECULT_IDUNN_MEMBER_BIND";
|
||||
pub const IDUNN_PROCESS_WRITE_LEASE_ENVIRONMENT: &str = "GAMECULT_IDUNN_PROCESS_WRITE_LEASE";
|
||||
pub const RUNTIME_PRESENCE_IDENTITY_BINDING: &str = "GAMECULT_RUNTIME_PRESENCE_IDENTITY";
|
||||
/// A host actuator hands the workload its activation credential as a file
|
||||
/// named by this variable; on the Idunn host it arrives as a parent-only
|
||||
/// descriptor instead and this name is never set.
|
||||
pub const IDUNN_ACTIVATION_CREDENTIAL_ENVIRONMENT: &str = "GAMECULT_IDUNN_ACTIVATION_CREDENTIAL";
|
||||
/// systemd hands a unit its credential directory under this name; Idunn owns
|
||||
/// the directory, so no binding may set it.
|
||||
pub const SYSTEMD_CREDENTIALS_DIRECTORY_ENVIRONMENT: &str = "CREDENTIALS_DIRECTORY";
|
||||
/// Every launch-environment name Idunn sets or reserves. A binding may name
|
||||
/// none of them; this is the one list both the binding check and the
|
||||
/// launch-input check read.
|
||||
pub(crate) const IDUNN_OWNED_LAUNCH_ENVIRONMENT: [&str; 6] = [
|
||||
IDUNN_RUNTIME_BUNDLE_ENVIRONMENT,
|
||||
IDUNN_RUNTIME_CANDIDATE_BIND_ENVIRONMENT,
|
||||
IDUNN_RUNTIME_MEMBER_BIND_ENVIRONMENT,
|
||||
IDUNN_PROCESS_WRITE_LEASE_ENVIRONMENT,
|
||||
IDUNN_ACTIVATION_CREDENTIAL_ENVIRONMENT,
|
||||
SYSTEMD_CREDENTIALS_DIRECTORY_ENVIRONMENT,
|
||||
];
|
||||
pub const RUNTIME_PRESENCE_IDENTITY_FD_NAME: &str = "gamecult-runtime-presence-identity";
|
||||
|
||||
#[derive(Clone, Debug, PartialEq, Eq, Serialize, Deserialize)]
|
||||
|
|
@ -113,10 +98,6 @@ pub struct ArtifactOutput {
|
|||
pub expected_sha256: Option<String>,
|
||||
#[serde(default)]
|
||||
pub executable: bool,
|
||||
/// Regular files inside a directory artifact that are sealed executable
|
||||
/// (0555); every other file in the tree is sealed 0444.
|
||||
#[serde(default)]
|
||||
pub executable_members: BTreeSet<PathBuf>,
|
||||
}
|
||||
|
||||
#[derive(Clone, Copy, Debug, PartialEq, Eq, Serialize, Deserialize)]
|
||||
|
|
@ -203,11 +184,6 @@ pub enum StateKind {
|
|||
/// Process-owned state imposed by an external runtime contract. Idunn
|
||||
/// preserves and fences the file but never interprets its payload.
|
||||
ExternalFile,
|
||||
/// A process-owned directory imposed by an external runtime contract (for
|
||||
/// example Synapse's media store). Idunn preserves and fences it and never
|
||||
/// interprets its contents. Same writer, recovery and startup rules as
|
||||
/// `ExternalFile`.
|
||||
ExternalDirectory,
|
||||
}
|
||||
|
||||
#[derive(Clone, Copy, Debug, PartialEq, Eq, PartialOrd, Ord, Serialize, Deserialize)]
|
||||
|
|
@ -528,7 +504,8 @@ impl WorkloadBinding {
|
|||
{
|
||||
require_environment_name(name)?;
|
||||
ensure!(
|
||||
!IDUNN_OWNED_LAUNCH_ENVIRONMENT.contains(&name.as_str()),
|
||||
name != IDUNN_RUNTIME_BUNDLE_ENVIRONMENT
|
||||
&& name != IDUNN_RUNTIME_CANDIDATE_BIND_ENVIRONMENT,
|
||||
"operator binding attempts to replace Idunn-owned workload environment"
|
||||
);
|
||||
}
|
||||
|
|
@ -661,7 +638,9 @@ pub struct ProcessWriteLeaseBinding {
|
|||
|
||||
impl ProcessWriteLeaseBinding {
|
||||
pub fn lock_path(&self) -> PathBuf {
|
||||
crate::drivers::authority_lock_path(&self.record_path)
|
||||
let mut lock_path = self.record_path.as_os_str().to_os_string();
|
||||
lock_path.push(".lock");
|
||||
PathBuf::from(lock_path)
|
||||
}
|
||||
}
|
||||
|
||||
|
|
@ -683,101 +662,42 @@ pub struct RouteBinding {
|
|||
pub private_port_end: u16,
|
||||
pub config_path: PathBuf,
|
||||
pub reload_unit: String,
|
||||
/// A second listener for the route's members. When set, members reach the
|
||||
/// workload straight through a second nginx server block, and the
|
||||
/// workload's launcher answers presence only.
|
||||
#[serde(default, skip_serializing_if = "Option::is_none")]
|
||||
pub member: Option<MemberListenerBinding>,
|
||||
}
|
||||
|
||||
/// The member listener of a route: where members connect, and the first of the
|
||||
/// private ports the workload binds for them. The private range is as wide as
|
||||
/// the route's own.
|
||||
#[derive(Clone, Debug, PartialEq, Eq, Serialize, Deserialize)]
|
||||
#[serde(deny_unknown_fields)]
|
||||
pub struct MemberListenerBinding {
|
||||
pub stable_endpoint: String,
|
||||
pub private_port_start: u16,
|
||||
/// Most member connections nginx holds open at once for this route. Each
|
||||
/// costs nginx two connections (client and upstream) from the pool the
|
||||
/// route's presence block shares, so this limit is what keeps members from
|
||||
/// exhausting it.
|
||||
#[serde(default = "default_member_max_connections")]
|
||||
pub max_connections: u16,
|
||||
}
|
||||
|
||||
const DEFAULT_MEMBER_MAX_CONNECTIONS: u16 = 128;
|
||||
|
||||
fn default_member_max_connections() -> u16 {
|
||||
DEFAULT_MEMBER_MAX_CONNECTIONS
|
||||
}
|
||||
|
||||
impl RouteBinding {
|
||||
pub(crate) fn stable_socket(&self) -> Result<(IpAddr, u16)> {
|
||||
parse_stable_socket(self.driver, &self.stable_endpoint, "stable endpoint")
|
||||
}
|
||||
|
||||
/// The member listener's stable socket, when the route has one.
|
||||
pub(crate) fn member_stable_socket(&self) -> Result<Option<(IpAddr, u16)>> {
|
||||
self.member
|
||||
.as_ref()
|
||||
.map(|member| {
|
||||
parse_stable_socket(self.driver, &member.stable_endpoint, "member endpoint")
|
||||
})
|
||||
.transpose()
|
||||
}
|
||||
|
||||
/// The private port the workload binds for members when its presence
|
||||
/// candidate is `candidate_port`. Derived, never stored: the member range
|
||||
/// is the private range shifted to the member start.
|
||||
pub(crate) fn member_candidate_port(&self, candidate_port: u16) -> Result<Option<u16>> {
|
||||
let Some(member) = self.member.as_ref() else {
|
||||
return Ok(None);
|
||||
};
|
||||
let authority = match self.driver {
|
||||
RouteDriver::NginxStreamTcp => self
|
||||
.stable_endpoint
|
||||
.strip_prefix("http://")
|
||||
.or_else(|| self.stable_endpoint.strip_prefix("tcp://")),
|
||||
RouteDriver::NginxStreamUdp => self.stable_endpoint.strip_prefix("rudp://"),
|
||||
}
|
||||
.context("stable endpoint transport does not match its route driver")?;
|
||||
ensure!(
|
||||
(self.private_port_start..=self.private_port_end).contains(&candidate_port),
|
||||
"candidate port is outside the route binding"
|
||||
!authority
|
||||
.chars()
|
||||
.any(|character| matches!(character, '/' | '?' | '#' | '@' | '[' | ']')),
|
||||
"stable endpoint is not a plain IP address and port"
|
||||
);
|
||||
let offset = candidate_port - self.private_port_start;
|
||||
let port = member
|
||||
.private_port_start
|
||||
.checked_add(offset)
|
||||
.context("member private port overflows")?;
|
||||
Ok(Some(port))
|
||||
let (host, port) = authority
|
||||
.rsplit_once(':')
|
||||
.context("stable endpoint has no port")?;
|
||||
let host = host
|
||||
.parse::<IpAddr>()
|
||||
.context("stable endpoint host is not an IP address")?;
|
||||
ensure!(
|
||||
matches!(host, IpAddr::V4(_)),
|
||||
"route binding v2 stable endpoints use explicit IPv4 addresses"
|
||||
);
|
||||
let port = port
|
||||
.parse::<u16>()
|
||||
.context("stable endpoint port is not a u16")?;
|
||||
ensure!(port > 0, "stable endpoint port is zero");
|
||||
Ok((host, port))
|
||||
}
|
||||
}
|
||||
|
||||
fn parse_stable_socket(driver: RouteDriver, endpoint: &str, what: &str) -> Result<(IpAddr, u16)> {
|
||||
let authority = match driver {
|
||||
RouteDriver::NginxStreamTcp => endpoint
|
||||
.strip_prefix("http://")
|
||||
.or_else(|| endpoint.strip_prefix("tcp://")),
|
||||
RouteDriver::NginxStreamUdp => endpoint.strip_prefix("rudp://"),
|
||||
}
|
||||
.with_context(|| format!("{what} transport does not match its route driver"))?;
|
||||
ensure!(
|
||||
!authority
|
||||
.chars()
|
||||
.any(|character| matches!(character, '/' | '?' | '#' | '@' | '[' | ']')),
|
||||
"{what} is not a plain IP address and port"
|
||||
);
|
||||
let (host, port) = authority
|
||||
.rsplit_once(':')
|
||||
.with_context(|| format!("{what} has no port"))?;
|
||||
let host = host
|
||||
.parse::<IpAddr>()
|
||||
.with_context(|| format!("{what} host is not an IP address"))?;
|
||||
ensure!(
|
||||
matches!(host, IpAddr::V4(_)),
|
||||
"route binding v2 endpoints use explicit IPv4 addresses"
|
||||
);
|
||||
let port = port
|
||||
.parse::<u16>()
|
||||
.with_context(|| format!("{what} port is not a u16"))?;
|
||||
ensure!(port > 0, "{what} port is zero");
|
||||
Ok((host, port))
|
||||
}
|
||||
|
||||
#[derive(Clone, Copy, Debug, PartialEq, Eq, Serialize, Deserialize)]
|
||||
#[serde(rename_all = "kebab-case")]
|
||||
pub enum RouteDriver {
|
||||
|
|
@ -916,14 +836,6 @@ impl TargetDeclaration {
|
|||
if let Some(expected_sha256) = &artifact.expected_sha256 {
|
||||
require_sha256(expected_sha256, "expected artifact sha256")?;
|
||||
}
|
||||
ensure!(
|
||||
!(artifact.executable && !artifact.executable_members.is_empty()),
|
||||
"artifact {} declares both executable and executable_members",
|
||||
artifact.id
|
||||
);
|
||||
for member in &artifact.executable_members {
|
||||
require_relative_path(member, "executable member", false)?;
|
||||
}
|
||||
ensure!(
|
||||
destinations.insert(artifact.destination.as_path()),
|
||||
"artifact destination {} is declared twice",
|
||||
|
|
@ -1007,14 +919,6 @@ impl TargetDeclaration {
|
|||
== self.service.route_required,
|
||||
"routed services must require exactly one Idunn-owned candidate bind"
|
||||
);
|
||||
ensure!(
|
||||
!self
|
||||
.service
|
||||
.required_environment
|
||||
.contains(IDUNN_RUNTIME_MEMBER_BIND_ENVIRONMENT)
|
||||
|| self.service.route_required,
|
||||
"only routed services may require the Idunn-owned member bind"
|
||||
);
|
||||
ensure!(
|
||||
self.service
|
||||
.required_environment
|
||||
|
|
@ -1472,39 +1376,7 @@ impl OperatorBinding {
|
|||
route.private_port_start < route.private_port_end,
|
||||
"candidate rollout requires at least two private ports"
|
||||
);
|
||||
let stable = route.stable_socket()?;
|
||||
if let Some(member) = route.member.as_ref() {
|
||||
ensure!(
|
||||
route.driver == RouteDriver::NginxStreamTcp,
|
||||
"only a TCP route may carry a member listener"
|
||||
);
|
||||
require_value(&member.stable_endpoint, "member endpoint")?;
|
||||
let member_stable = route
|
||||
.member_stable_socket()?
|
||||
.context("route has no member listener")?;
|
||||
ensure!(
|
||||
member_stable != stable,
|
||||
"member endpoint equals the route's stable endpoint"
|
||||
);
|
||||
ensure!(
|
||||
member.private_port_start > 0,
|
||||
"member private port range starts at zero"
|
||||
);
|
||||
ensure!(
|
||||
member.max_connections > 0,
|
||||
"member connection limit is zero"
|
||||
);
|
||||
let width = route.private_port_end - route.private_port_start;
|
||||
let member_end = member
|
||||
.private_port_start
|
||||
.checked_add(width)
|
||||
.context("member private port range overflows")?;
|
||||
ensure!(
|
||||
member_end < route.private_port_start
|
||||
|| member.private_port_start > route.private_port_end,
|
||||
"member private port range overlaps the route's private range"
|
||||
);
|
||||
}
|
||||
route.stable_socket()?;
|
||||
require_absolute_path(&route.config_path, "route config")?;
|
||||
require_unit(&route.reload_unit, "route reload unit")?;
|
||||
}
|
||||
|
|
@ -1682,7 +1554,12 @@ impl OperatorBinding {
|
|||
.keys()
|
||||
.chain(self.workload.secret_environment_names())
|
||||
.collect();
|
||||
for owned in IDUNN_OWNED_LAUNCH_ENVIRONMENT {
|
||||
for owned in [
|
||||
IDUNN_RUNTIME_BUNDLE_ENVIRONMENT,
|
||||
IDUNN_RUNTIME_CANDIDATE_BIND_ENVIRONMENT,
|
||||
IDUNN_PROCESS_WRITE_LEASE_ENVIRONMENT,
|
||||
IDUNN_ACTIVATION_CREDENTIAL_ENVIRONMENT,
|
||||
] {
|
||||
ensure!(
|
||||
!workload_names.contains(&owned.to_string()),
|
||||
"operator binding cannot replace Idunn-owned launch environment {owned}"
|
||||
|
|
@ -1704,9 +1581,6 @@ impl OperatorBinding {
|
|||
}
|
||||
if declaration.service.route_required {
|
||||
available_environment.insert(IDUNN_RUNTIME_CANDIDATE_BIND_ENVIRONMENT.into());
|
||||
if self.route.as_ref().is_some_and(|route| route.member.is_some()) {
|
||||
available_environment.insert(IDUNN_RUNTIME_MEMBER_BIND_ENVIRONMENT.into());
|
||||
}
|
||||
}
|
||||
if declaration.write_lease_required() {
|
||||
available_environment.insert(IDUNN_PROCESS_WRITE_LEASE_ENVIRONMENT.into());
|
||||
|
|
@ -2054,10 +1928,10 @@ fn require_host(value: &str) -> Result<()> {
|
|||
}
|
||||
|
||||
#[cfg(test)]
|
||||
pub(crate) mod tests {
|
||||
mod tests {
|
||||
use super::*;
|
||||
|
||||
pub(crate) const RECIPE: &str = r#"
|
||||
const RECIPE: &str = r#"
|
||||
schema = "gamecult.idunn.target_declaration.v1"
|
||||
target = "test-service"
|
||||
source_stamp_environment = "TEST_SERVICE_BUILD_COMMIT"
|
||||
|
|
@ -2129,7 +2003,7 @@ compatibility = "v1"
|
|||
startup = "before-promotion"
|
||||
"#;
|
||||
|
||||
pub(crate) const BINDING: &str = r#"
|
||||
const BINDING: &str = r#"
|
||||
schema = "gamecult.idunn.operator_binding.v2"
|
||||
target = "test-service"
|
||||
profiles = ["aetheria", "full-gamecult"]
|
||||
|
|
@ -2223,76 +2097,6 @@ nodes = ["yggdrasil"]
|
|||
assert!(TargetDeclaration::parse(&input).is_err());
|
||||
}
|
||||
|
||||
const MEMBER_CANARY: &str = "zzcanary";
|
||||
|
||||
fn recipe_with_members(members: &str) -> String {
|
||||
let input = RECIPE.replace("executable = true", &format!("executable_members = [{members}]"));
|
||||
assert_ne!(input, RECIPE, "the fixture lost its executable line");
|
||||
input
|
||||
}
|
||||
|
||||
/// A refused member list is refused by name of the field, never by echoing
|
||||
/// the member.
|
||||
fn assert_members_refused(members: &[&str]) {
|
||||
let control = TargetDeclaration::parse(&recipe_with_members("\"bin/daemon\"")).unwrap();
|
||||
assert!(control.artifacts[0].executable_members.contains(Path::new("bin/daemon")));
|
||||
for member in members {
|
||||
let error = TargetDeclaration::parse(&recipe_with_members(&format!("\"{member}\"")))
|
||||
.expect_err(member);
|
||||
assert!(
|
||||
!format!("{error:#}").contains(MEMBER_CANARY),
|
||||
"the refusal echoed a member path"
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn recipe_executable_member_refuses_parent_traversal() {
|
||||
assert_members_refused(&[
|
||||
"zzcanary/../b",
|
||||
"../zzcanary",
|
||||
"bin/zzcanary/..",
|
||||
"bin/../../zzcanary",
|
||||
]);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn recipe_executable_member_refuses_absolute_path() {
|
||||
assert_members_refused(&["/zzcanary/sh", "/zzcanary"]);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn recipe_executable_member_refuses_empty_and_current_dir() {
|
||||
assert_members_refused(&["", ".", "./", "zzcanary/.."]);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn recipe_refuses_executable_flag_with_members() {
|
||||
let input = RECIPE.replace(
|
||||
"executable = true",
|
||||
"executable = true\nexecutable_members = [\"bin/daemon\"]",
|
||||
);
|
||||
let error = TargetDeclaration::parse(&input).unwrap_err();
|
||||
assert!(format!("{error:#}").contains("executable_members"));
|
||||
assert!(!format!("{error:#}").contains("bin/daemon"));
|
||||
// executable = true alone, and members alone, both stay admitted.
|
||||
assert!(TargetDeclaration::parse(RECIPE).is_ok());
|
||||
assert!(TargetDeclaration::parse(&recipe_with_members("\"bin/daemon\"")).is_ok());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn recipe_admits_members_on_both_source_kinds() {
|
||||
let runner_output = TargetDeclaration::parse(&recipe_with_members("\"a/b\", \"c\"")).unwrap();
|
||||
assert_eq!(runner_output.artifacts[0].source_kind, ArtifactSource::RunnerOutput);
|
||||
assert_eq!(runner_output.artifacts[0].executable_members.len(), 2);
|
||||
let worktree = recipe_with_members("\"a/b\", \"c\"")
|
||||
.replace("source_kind = \"runner-output\"\nrunner = \"rust\"\n", "source_kind = \"worktree-tree\"\n")
|
||||
.replace("expected_sha256 = \"cccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccc\"\n", "");
|
||||
let worktree = TargetDeclaration::parse(&worktree).unwrap();
|
||||
assert_eq!(worktree.artifacts[0].source_kind, ArtifactSource::WorktreeTree);
|
||||
assert_eq!(worktree.artifacts[0].executable_members.len(), 2);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn operator_cannot_omit_an_affordance_derived_from_the_recipe() {
|
||||
let recipe = TargetDeclaration::parse(RECIPE).unwrap();
|
||||
|
|
@ -2426,203 +2230,6 @@ nodes = ["yggdrasil"]
|
|||
}
|
||||
}
|
||||
|
||||
/// The recipe and binding of a routed service whose members connect to a
|
||||
/// listener of their own.
|
||||
pub(crate) fn member_listener_pair() -> (String, String) {
|
||||
let recipe = RECIPE.replace(
|
||||
"required_environment = [",
|
||||
"required_environment = [\"GAMECULT_IDUNN_MEMBER_BIND\", ",
|
||||
);
|
||||
let binding = BINDING.replace(
|
||||
"reload_unit = \"nginx.service\"\n",
|
||||
"reload_unit = \"nginx.service\"\n\n[route.member]\nstable_endpoint = \"http://127.0.0.1:18840\"\nprivate_port_start = 18841\n",
|
||||
);
|
||||
(recipe, binding)
|
||||
}
|
||||
|
||||
fn member_binding_with(replacement: (&str, &str)) -> Result<OperatorBinding> {
|
||||
let (recipe, binding) = member_listener_pair();
|
||||
let recipe = TargetDeclaration::parse(&recipe)?;
|
||||
let binding = OperatorBinding::parse(&binding.replace(replacement.0, replacement.1))?;
|
||||
binding.admit(&recipe)?;
|
||||
Ok(binding)
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn a_member_listener_admits_with_a_recipe_that_requires_the_member_bind() {
|
||||
let binding = member_binding_with(("never-matches", "")).unwrap();
|
||||
let route = binding.route.unwrap();
|
||||
assert_eq!(route.member_stable_socket().unwrap().unwrap().1, 18840);
|
||||
assert_eq!(route.member_candidate_port(18831).unwrap(), Some(18841));
|
||||
assert_eq!(route.member_candidate_port(18839).unwrap(), Some(18849));
|
||||
assert!(route.member_candidate_port(18840).is_err());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn a_binding_without_a_member_listener_has_no_member_ports() {
|
||||
let binding = OperatorBinding::parse(BINDING).unwrap();
|
||||
let route = binding.route.unwrap();
|
||||
assert_eq!(route.member_stable_socket().unwrap(), None);
|
||||
assert_eq!(route.member_candidate_port(18831).unwrap(), None);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn a_member_listener_needs_a_recipe_that_requires_the_member_bind() {
|
||||
let (recipe, binding) = member_listener_pair();
|
||||
// Bound but not required: the recipe never asked for the variable.
|
||||
let plain_recipe = TargetDeclaration::parse(RECIPE).unwrap();
|
||||
assert!(
|
||||
OperatorBinding::parse(&binding)
|
||||
.unwrap()
|
||||
.admit(&plain_recipe)
|
||||
.is_err()
|
||||
);
|
||||
// Required but not bound: the variable would never be set.
|
||||
let recipe = TargetDeclaration::parse(&recipe).unwrap();
|
||||
assert!(
|
||||
OperatorBinding::parse(BINDING)
|
||||
.unwrap()
|
||||
.admit(&recipe)
|
||||
.is_err()
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn only_a_routed_recipe_may_require_the_member_bind() {
|
||||
let recipe = RECIPE
|
||||
.replace("route_required = true", "route_required = false")
|
||||
.replace(
|
||||
"required_environment = [\"GAMECULT_IDUNN_CANDIDATE_BIND\", ",
|
||||
"required_environment = [\"GAMECULT_IDUNN_MEMBER_BIND\", ",
|
||||
);
|
||||
assert!(recipe.contains("GAMECULT_IDUNN_MEMBER_BIND"));
|
||||
assert!(TargetDeclaration::parse(&recipe).is_err());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn member_listener_bindings_are_validated() {
|
||||
let start = |value: &str| {
|
||||
member_binding_with((
|
||||
"private_port_start = 18841",
|
||||
&format!("private_port_start = {value}"),
|
||||
))
|
||||
};
|
||||
// Overlapping the route's own private range, in either direction.
|
||||
assert!(start("18835").is_err());
|
||||
assert!(start("18825").is_err());
|
||||
// Adjacent but disjoint is fine, above the route's range or below it.
|
||||
assert!(start("18840").is_ok());
|
||||
assert!(start("18822").is_ok());
|
||||
assert!(start("18823").is_err());
|
||||
// The connection limit defaults, and zero is refused.
|
||||
let limit = |value: &str| {
|
||||
member_binding_with((
|
||||
"private_port_start = 18841",
|
||||
&format!("private_port_start = 18841
|
||||
max_connections = {value}"),
|
||||
))
|
||||
};
|
||||
assert_eq!(
|
||||
member_binding_with(("never-matches", "")).unwrap().route.unwrap().member.unwrap().max_connections,
|
||||
128
|
||||
);
|
||||
assert_eq!(limit("7").unwrap().route.unwrap().member.unwrap().max_connections, 7);
|
||||
assert!(limit("0").is_err());
|
||||
assert!(limit("65535").is_ok());
|
||||
assert!(limit("65536").is_err());
|
||||
// Zero, and a width that overflows u16.
|
||||
assert!(start("0").is_err());
|
||||
assert!(start("65530").is_err());
|
||||
// The member endpoint may not be the route's stable endpoint.
|
||||
assert!(member_binding_with(("http://127.0.0.1:18840", "http://127.0.0.1:18830")).is_err());
|
||||
// The same transport and address rules as the stable endpoint.
|
||||
assert!(member_binding_with(("http://127.0.0.1:18840", "rudp://127.0.0.1:18840")).is_err());
|
||||
assert!(member_binding_with(("http://127.0.0.1:18840", "http://example.test:18840")).is_err());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn a_udp_route_cannot_carry_a_member_listener() {
|
||||
let (recipe, binding) = member_listener_pair();
|
||||
let binding = binding
|
||||
.replace("nginx-stream-tcp", "nginx-stream-udp")
|
||||
.replace("http://127.0.0.1:18830", "rudp://127.0.0.1:18830")
|
||||
.replace("http://127.0.0.1:18840", "rudp://127.0.0.1:18840");
|
||||
let recipe = TargetDeclaration::parse(&recipe).unwrap();
|
||||
assert!(
|
||||
OperatorBinding::parse(&binding)
|
||||
.and_then(|binding| binding.admit(&recipe))
|
||||
.is_err()
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn idunn_owned_names_are_reserved_in_one_list() {
|
||||
let recipe = TargetDeclaration::parse(RECIPE).unwrap();
|
||||
let owned = [
|
||||
"GAMECULT_IDUNN_RUNTIME_BUNDLE",
|
||||
"GAMECULT_IDUNN_CANDIDATE_BIND",
|
||||
"GAMECULT_IDUNN_MEMBER_BIND",
|
||||
"GAMECULT_IDUNN_PROCESS_WRITE_LEASE",
|
||||
"GAMECULT_IDUNN_ACTIVATION_CREDENTIAL",
|
||||
"CREDENTIALS_DIRECTORY",
|
||||
];
|
||||
let mut listed = IDUNN_OWNED_LAUNCH_ENVIRONMENT.to_vec();
|
||||
listed.sort_unstable();
|
||||
let mut expected = owned.to_vec();
|
||||
expected.sort_unstable();
|
||||
assert_eq!(listed, expected);
|
||||
|
||||
let as_environment = |name: &str| {
|
||||
BINDING.replace(
|
||||
"[workload.argument_bindings]",
|
||||
&format!("[workload.environment]\n{name} = \"value\"\n\n[workload.argument_bindings]"),
|
||||
)
|
||||
};
|
||||
let as_secret = |name: &str| {
|
||||
BINDING.replace(
|
||||
"[workload.secret_files]",
|
||||
&format!("[workload.secret_files]\n{name} = \"/etc/gamecult/test-service/other.cc\""),
|
||||
)
|
||||
};
|
||||
// The fixtures are well formed: an unowned name parses in both
|
||||
// shapes, so the refusals below are about the name.
|
||||
assert!(OperatorBinding::parse(&as_environment("UNOWNED_NAME")).is_ok());
|
||||
assert!(OperatorBinding::parse(&as_secret("UNOWNED_NAME")).is_ok());
|
||||
for name in owned {
|
||||
// Refused when the binding is parsed...
|
||||
assert!(
|
||||
OperatorBinding::parse(&as_environment(name)).is_err(),
|
||||
"{name} accepted as environment at parse"
|
||||
);
|
||||
assert!(
|
||||
OperatorBinding::parse(&as_secret(name)).is_err(),
|
||||
"{name} accepted as a secret file at parse"
|
||||
);
|
||||
// ...and again at admission, for a binding that did not come
|
||||
// through parse. Each site reads the one list.
|
||||
for secret in [false, true] {
|
||||
let mut binding = OperatorBinding::parse(BINDING).unwrap();
|
||||
let WorkloadBinding::SystemdTransient(workload) = &mut binding.workload else {
|
||||
panic!("the fixture is a systemd workload");
|
||||
};
|
||||
if secret {
|
||||
workload.secret_files.insert(name.into(), "/etc/gamecult/test-service/other.cc".into());
|
||||
} else {
|
||||
workload.environment.insert(name.into(), "forged".into());
|
||||
}
|
||||
assert!(
|
||||
binding.workload.validate_launch_environment().is_err(),
|
||||
"{name} (secret: {secret}) accepted by the binding check"
|
||||
);
|
||||
let error = binding.admit(&recipe).unwrap_err().to_string();
|
||||
assert!(
|
||||
error.contains("Idunn-owned launch environment"),
|
||||
"{name} (secret: {secret}) accepted by the launch-input check: {error}"
|
||||
);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn state_slot_path_cannot_escape_the_operator_owned_root() {
|
||||
let input = RECIPE.replace(
|
||||
|
|
@ -2654,17 +2261,6 @@ max_connections = {value}"),
|
|||
assert!(declaration.write_lease_required());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn external_directory_state_is_lease_gated_like_external_file() {
|
||||
let input = RECIPE.replace("kind = \"cultcache-file\"", "kind = \"external-directory\"");
|
||||
let declaration = TargetDeclaration::parse(&input).unwrap();
|
||||
let slot = &declaration.state.as_ref().unwrap().slots[0];
|
||||
assert_eq!(slot.kind, StateKind::ExternalDirectory);
|
||||
assert_eq!(slot.writer, StateWriter::ProcessBoundSingleWriter);
|
||||
assert_eq!(slot.startup, StateStartup::CreateOrOpenAfterWriteLease);
|
||||
assert!(declaration.write_lease_required());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn state_slots_cannot_claim_overlapping_paths() {
|
||||
let input = RECIPE.replace(
|
||||
|
|
|
|||
|
|
@ -475,13 +475,6 @@ pub const DEFAULT_AWAITING_READY_DEADLINE_SECONDS: u32 = 300;
|
|||
pub const DEFAULT_ROUTING_DEADLINE_SECONDS: u32 = 120;
|
||||
pub const DEFAULT_COMMITTING_DEADLINE_SECONDS: u32 = 120;
|
||||
|
||||
/// Idunn's own deadlines, not binding-declared: a new `PhaseDeadlines` field
|
||||
/// would change every v3 plan id (`recomputed_plan_id` re-encodes the plan).
|
||||
/// Warming is timed for every transaction, owner or not.
|
||||
pub const WARMING_DEADLINE_SECONDS: u32 = 600;
|
||||
/// A pre-fencing abort is timed from its beginning.
|
||||
pub const PRE_FENCING_ABORT_DEADLINE_SECONDS: u32 = 300;
|
||||
|
||||
/// The resolved per-phase deadline durations a plan carries. Entering a
|
||||
/// post-fencing phase stamps its deadline from them, and the deadline
|
||||
/// resolver ends the phase once it passes.
|
||||
|
|
@ -734,11 +727,6 @@ pub struct ArtifactReceipt {
|
|||
pub sha256: String,
|
||||
pub size_bytes: u64,
|
||||
pub executable: bool,
|
||||
/// Files sealed executable inside a directory artifact. Skipped when empty
|
||||
/// so a receipt with no members keeps its original encoding, and so its
|
||||
/// sealed release id.
|
||||
#[serde(default, skip_serializing_if = "BTreeSet::is_empty")]
|
||||
pub executable_members: BTreeSet<PathBuf>,
|
||||
}
|
||||
|
||||
impl ArtifactReceipt {
|
||||
|
|
@ -752,9 +740,7 @@ impl ArtifactReceipt {
|
|||
.find(|artifact| artifact.id == self.artifact_id)
|
||||
.with_context(|| format!("plan declares no artifact {}", self.artifact_id))?;
|
||||
ensure!(
|
||||
self.destination == declared.destination
|
||||
&& self.executable == declared.executable
|
||||
&& self.executable_members == declared.executable_members,
|
||||
self.destination == declared.destination && self.executable == declared.executable,
|
||||
"artifact receipt differs from declared output"
|
||||
);
|
||||
if let Some(expected_sha256) = &declared.expected_sha256 {
|
||||
|
|
@ -1432,7 +1418,6 @@ nodes = ["yggdrasil"]
|
|||
sha256: digest(b'c'),
|
||||
size_bytes: 42,
|
||||
executable: true,
|
||||
executable_members: BTreeSet::new(),
|
||||
}
|
||||
}
|
||||
|
||||
|
|
@ -1799,113 +1784,6 @@ nodes = ["yggdrasil"]
|
|||
assert!(changed.validate_against(&plan).is_err());
|
||||
}
|
||||
|
||||
fn hex(bytes: &[u8]) -> String {
|
||||
bytes.iter().map(|byte| format!("{byte:02x}")).collect()
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn receipt_without_members_encodes_as_before() {
|
||||
// Both values were computed outside this code, from the five-field
|
||||
// msgpack array the receipt encoded to at base d74ac8d.
|
||||
const BASE_RECEIPT: &str = "95a66461656d6f6ea773657276696365d9477368613235362d636363636363636363636363636363636363636363636363636363636363636363636363636363636363636363636363636363636363636363636363636363632ac3";
|
||||
const BASE_RELEASE_ID: &str =
|
||||
"sha256-b9143d8f546ad71c493c70037813547aaec10fd87994a6f3bdc77ac3a8b3495e";
|
||||
let receipt = artifact_receipt();
|
||||
let encoded = rmp_serde::to_vec(&receipt).unwrap();
|
||||
assert_eq!(hex(&encoded), BASE_RECEIPT);
|
||||
let bytes = (0..BASE_RECEIPT.len())
|
||||
.step_by(2)
|
||||
.map(|at| u8::from_str_radix(&BASE_RECEIPT[at..at + 2], 16).unwrap())
|
||||
.collect::<Vec<_>>();
|
||||
assert_eq!(rmp_serde::from_slice::<ArtifactReceipt>(&bytes).unwrap(), receipt);
|
||||
let release = SealedRelease {
|
||||
schema: SEALED_RELEASE_SCHEMA.into(),
|
||||
sealed_release_id: String::new(),
|
||||
plan_id: digest(b'a'),
|
||||
artifacts: vec![receipt],
|
||||
external_inputs: vec![],
|
||||
sealed_at_unix_millis: 0,
|
||||
};
|
||||
assert_eq!(release.recomputed_release_id().unwrap(), BASE_RELEASE_ID);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn sealed_release_address_covers_executable_members() {
|
||||
let release_with = |members: &[&str]| SealedRelease {
|
||||
schema: SEALED_RELEASE_SCHEMA.into(),
|
||||
sealed_release_id: String::new(),
|
||||
plan_id: digest(b'a'),
|
||||
artifacts: vec![ArtifactReceipt {
|
||||
executable: false,
|
||||
executable_members: members.iter().map(PathBuf::from).collect(),
|
||||
..artifact_receipt()
|
||||
}],
|
||||
external_inputs: vec![],
|
||||
sealed_at_unix_millis: 0,
|
||||
};
|
||||
let none = release_with(&[]).recomputed_release_id().unwrap();
|
||||
let one = release_with(&["bin/daemon"]).recomputed_release_id().unwrap();
|
||||
let other = release_with(&["bin/other"]).recomputed_release_id().unwrap();
|
||||
let two = release_with(&["bin/daemon", "bin/other"])
|
||||
.recomputed_release_id()
|
||||
.unwrap();
|
||||
let all = [&none, &one, &other, &two];
|
||||
for (index, left) in all.iter().enumerate() {
|
||||
for right in &all[index + 1..] {
|
||||
assert_ne!(left, right);
|
||||
}
|
||||
}
|
||||
|
||||
// The declaration is the other half: a receipt must carry exactly the
|
||||
// declared list, in a plan that declares one and in one that does not.
|
||||
let plain = plan();
|
||||
let mut with_members = artifact_receipt();
|
||||
with_members.executable_members = BTreeSet::from([PathBuf::from("bin/daemon")]);
|
||||
assert!(
|
||||
SealedRelease::new(&plain, vec![with_members], vec![external_input_receipt()], 120)
|
||||
.is_err()
|
||||
);
|
||||
let recipe = RECIPE.replace("executable = true", "executable_members = [\"bin/daemon\", \"lib/tool\"]");
|
||||
let declaring = compile_deployment_plan(
|
||||
recipe.as_bytes(),
|
||||
BINDING.as_bytes(),
|
||||
source(&recipe),
|
||||
"service-incarnation-1",
|
||||
Some(18001),
|
||||
110,
|
||||
&[ready_odin_provider("odin", "odin-yggdrasil", 1)],
|
||||
)
|
||||
.unwrap();
|
||||
let honest = ArtifactReceipt {
|
||||
executable: false,
|
||||
executable_members: BTreeSet::from([PathBuf::from("bin/daemon"), PathBuf::from("lib/tool")]),
|
||||
..artifact_receipt()
|
||||
};
|
||||
let release = SealedRelease::new(
|
||||
&declaring,
|
||||
vec![honest.clone()],
|
||||
vec![external_input_receipt()],
|
||||
120,
|
||||
)
|
||||
.unwrap();
|
||||
release.validate_against(&declaring).unwrap();
|
||||
for drifted in [
|
||||
BTreeSet::from([PathBuf::from("bin/daemon")]),
|
||||
BTreeSet::from([PathBuf::from("bin/daemon"), PathBuf::from("lib/other")]),
|
||||
BTreeSet::new(),
|
||||
] {
|
||||
let mut receipt = honest.clone();
|
||||
receipt.executable_members = drifted;
|
||||
assert!(
|
||||
SealedRelease::new(&declaring, vec![receipt], vec![external_input_receipt()], 120)
|
||||
.is_err()
|
||||
);
|
||||
}
|
||||
let mut changed = release.clone();
|
||||
changed.artifacts[0].executable_members.insert(PathBuf::from("lib/extra"));
|
||||
assert!(changed.validate_against(&declaring).is_err());
|
||||
}
|
||||
|
||||
fn binding_with_deadlines(table: &str) -> String {
|
||||
BINDING.replace("[placement]", &format!("{table}\n[placement]"))
|
||||
}
|
||||
|
|
|
|||
2170
src/drivers.rs
2170
src/drivers.rs
File diff suppressed because it is too large
Load diff
45
src/host.rs
45
src/host.rs
|
|
@ -30,13 +30,13 @@ use serde::Deserialize;
|
|||
|
||||
use crate::control_plane::read_trust_anchor;
|
||||
use crate::deployment::{
|
||||
HostWorkloadBinding, IDUNN_ACTIVATION_CREDENTIAL_ENVIRONMENT,
|
||||
ArtifactSource, HostWorkloadBinding, IDUNN_ACTIVATION_CREDENTIAL_ENVIRONMENT,
|
||||
IDUNN_RUNTIME_BUNDLE_ENVIRONMENT, LaunchArgument, OperatorBinding, TargetDeclaration,
|
||||
};
|
||||
use crate::deployment_plan::{CompiledDeploymentPlan, SealedRelease};
|
||||
use crate::deployment_plan::{ArtifactReceipt, CompiledDeploymentPlan, SealedRelease};
|
||||
use crate::drivers::{
|
||||
HostWorkloadObservation, InstalledReleaseObservation, copy_tree, digest_artifact,
|
||||
release_artifact, remove_tree_inside, seal_artifact, sha256_id, write_runtime_bundle_records,
|
||||
HostWorkloadObservation, InstalledReleaseObservation, copy_artifact, copy_tree,
|
||||
digest_artifact, release_artifact, remove_tree_inside, sha256_id, write_runtime_bundle_records,
|
||||
};
|
||||
use crate::host_actuator::{
|
||||
HOST_ACTUATOR_CHANNEL, HOST_ACTUATOR_CONNECTION_ID, HOST_ACTUATOR_PING_MS,
|
||||
|
|
@ -1048,12 +1048,37 @@ fn materialize(
|
|||
fs::create_dir_all(&staging)?;
|
||||
let mut artifacts = Vec::new();
|
||||
for artifact in &declaration.artifacts {
|
||||
artifacts.push(seal_artifact(
|
||||
artifact,
|
||||
&source,
|
||||
&source.join(&artifact.source),
|
||||
&staging,
|
||||
)?);
|
||||
let source_path = match artifact.source_kind {
|
||||
ArtifactSource::RunnerOutput | ArtifactSource::WorktreeTree => {
|
||||
source.join(&artifact.source)
|
||||
}
|
||||
};
|
||||
ensure!(
|
||||
source_path.exists(),
|
||||
"declared artifact output {} is absent",
|
||||
artifact.id
|
||||
);
|
||||
let destination = staging.join(&artifact.destination);
|
||||
ensure!(
|
||||
destination.starts_with(&staging),
|
||||
"artifact destination escaped its staging root"
|
||||
);
|
||||
copy_artifact(&source, &source_path, &destination)?;
|
||||
let (sha256, size_bytes) = digest_artifact(&destination)?;
|
||||
if let Some(expected) = &artifact.expected_sha256 {
|
||||
ensure!(
|
||||
&sha256 == expected,
|
||||
"artifact {} differs from its recipe-pinned digest",
|
||||
artifact.id
|
||||
);
|
||||
}
|
||||
artifacts.push(ArtifactReceipt {
|
||||
artifact_id: artifact.id.clone(),
|
||||
destination: artifact.destination.clone(),
|
||||
sha256: format!("sha256-{sha256}"),
|
||||
size_bytes,
|
||||
executable: artifact.executable,
|
||||
});
|
||||
}
|
||||
let release = SealedRelease::new(plan, artifacts, Vec::new(), sealed_at_unix_millis)?;
|
||||
Ok((release, staging))
|
||||
|
|
|
|||
|
|
@ -1151,7 +1151,7 @@ mod tests {
|
|||
let answerer = thread::spawn(move || -> Result<()> {
|
||||
for _ in 0..500 {
|
||||
client.poll_resends()?;
|
||||
if let Some(frame) = client.receive_once()? {
|
||||
while let Some(frame) = client.receive_once()? {
|
||||
let envelope = HostActuatorEnvelope::decode(&frame.payload)?;
|
||||
let request = envelope.open_request("raven", &idunn_anchor)?;
|
||||
assert_eq!(envelope.session_nonce, nonce);
|
||||
|
|
|
|||
|
|
@ -32,9 +32,6 @@ pub(crate) const INCIDENT_RETENTION_MILLIS: u64 = 7 * 24 * 60 * 60 * 1000;
|
|||
#[serde(rename_all = "kebab-case")]
|
||||
pub(crate) enum IncidentCondition {
|
||||
ContinuityExhausted,
|
||||
/// A transaction past its deadline in a state Idunn will not end by
|
||||
/// itself. Subject: the transaction id.
|
||||
OperatorRequired,
|
||||
}
|
||||
|
||||
impl IncidentCondition {
|
||||
|
|
@ -43,7 +40,6 @@ impl IncidentCondition {
|
|||
pub(crate) const fn name(self) -> &'static str {
|
||||
match self {
|
||||
Self::ContinuityExhausted => "continuity-exhausted",
|
||||
Self::OperatorRequired => "operator-required",
|
||||
}
|
||||
}
|
||||
|
||||
|
|
@ -53,7 +49,6 @@ impl IncidentCondition {
|
|||
pub(crate) const fn ends_with_admission(self) -> bool {
|
||||
match self {
|
||||
Self::ContinuityExhausted => true,
|
||||
Self::OperatorRequired => false,
|
||||
}
|
||||
}
|
||||
}
|
||||
|
|
@ -276,17 +271,6 @@ impl IncidentStore {
|
|||
Ok(true)
|
||||
}
|
||||
|
||||
/// The subjects of every open incident of `condition`.
|
||||
pub(crate) fn open_subjects(&self, condition: IncidentCondition) -> Result<Vec<String>> {
|
||||
Ok(self
|
||||
.snapshot()?
|
||||
.records
|
||||
.into_iter()
|
||||
.filter(|record| record.is_open() && record.condition == condition)
|
||||
.map(|record| record.subject)
|
||||
.collect())
|
||||
}
|
||||
|
||||
/// Close the open incident for `(condition, subject)`, if there is one.
|
||||
/// `Ok(true)` means this call wrote the closure.
|
||||
pub(crate) fn close(
|
||||
|
|
@ -484,12 +468,11 @@ pub(crate) mod tests {
|
|||
);
|
||||
// The wire spelling readers match on is the spelling the key and
|
||||
// status use.
|
||||
for condition in [IncidentCondition::ContinuityExhausted, IncidentCondition::OperatorRequired] {
|
||||
let payload = rmp_serde::to_vec(&condition).unwrap();
|
||||
assert_eq!(rmp_serde::from_slice::<String>(&payload).unwrap(), condition.name());
|
||||
}
|
||||
assert_eq!(IncidentCondition::OperatorRequired.name(), "operator-required");
|
||||
assert!(!IncidentCondition::OperatorRequired.ends_with_admission());
|
||||
let payload = rmp_serde::to_vec(&IncidentCondition::ContinuityExhausted).unwrap();
|
||||
assert_eq!(
|
||||
rmp_serde::from_slice::<String>(&payload).unwrap(),
|
||||
IncidentCondition::ContinuityExhausted.name()
|
||||
);
|
||||
let payload = rmp_serde::to_vec(&CloseReason::NoLongerAdmitted).unwrap();
|
||||
assert_eq!(
|
||||
rmp_serde::from_slice::<String>(&payload).unwrap(),
|
||||
|
|
|
|||
3238
src/launcher.rs
3238
src/launcher.rs
File diff suppressed because it is too large
Load diff
|
|
@ -4,10 +4,5 @@ pub mod deployment_plan;
|
|||
pub mod drivers;
|
||||
pub mod host;
|
||||
pub mod host_actuator;
|
||||
#[cfg(unix)]
|
||||
pub mod launcher;
|
||||
mod incident;
|
||||
mod runner_evidence;
|
||||
pub mod provisioning;
|
||||
#[cfg(unix)]
|
||||
mod steward_orders;
|
||||
|
|
|
|||
|
|
@ -1,805 +0,0 @@
|
|||
//! The output tail of a failed runner step.
|
||||
//!
|
||||
//! One file, `runner-evidence.cc`, beside `control.cc`, holds one type: the
|
||||
//! last bytes of a failed runner container's stdout and stderr, one record per
|
||||
//! transaction. The engine's pre-fence failure path is its only writer, and
|
||||
//! `idunn status --command` its only reader. It is never `control.cc`, which
|
||||
//! refuses any type it does not know, and never `incidents.cc`, which is
|
||||
//! published and carries no free text.
|
||||
//!
|
||||
//! The tail is raw runner output. It is kept unredacted and readable by root
|
||||
//! alone (mode 0600): root can already read every secret the runner was given.
|
||||
//! It never enters an error, a transaction record or the journal. The status
|
||||
//! reader is a terminal, so everything it prints from this store goes through
|
||||
//! one allowlist sink (`emit`): printable ASCII passes, all else is escaped.
|
||||
|
||||
use std::path::{Path, PathBuf};
|
||||
|
||||
use anyhow::{Context, Result, bail, ensure};
|
||||
use cultcache_rs::{
|
||||
CultCacheEnvelope, DatabaseEntry, SingleFileMessagePackBackingStore,
|
||||
TryCompareExchangeSnapshotOutcome,
|
||||
};
|
||||
use serde::{Deserialize, Serialize};
|
||||
|
||||
use crate::control_plane::{decode_record, require_id, typed_envelope};
|
||||
use crate::drivers::{
|
||||
LockContended, RUNNER_OUTPUT_TAIL_BYTES, RunnerStepFailure, RunnerSubject,
|
||||
exchange_behind_private_lock, runner_status,
|
||||
};
|
||||
|
||||
const RUNNER_FAILURE_SCHEMA: &str = "idunn.runner_failure.v1";
|
||||
|
||||
/// Records kept; the writer drops the oldest past this, so the store needs no
|
||||
/// other owner.
|
||||
pub(crate) const RUNNER_EVIDENCE_RECORDS: usize = 32;
|
||||
|
||||
/// `runner-evidence.cc`, beside the control store.
|
||||
pub(crate) fn runner_evidence_store_path(state_store: &Path) -> PathBuf {
|
||||
state_store.with_file_name("runner-evidence.cc")
|
||||
}
|
||||
|
||||
#[derive(Clone, Copy, Debug, PartialEq, Eq, Serialize, Deserialize)]
|
||||
#[serde(rename_all = "kebab-case")]
|
||||
pub(crate) enum SubjectKind {
|
||||
Step,
|
||||
ExternalInput,
|
||||
}
|
||||
|
||||
#[derive(Clone, PartialEq, Eq, DatabaseEntry)]
|
||||
#[cultcache(type = "idunn.runner_failure", schema = "idunn.runner_failure.v1")]
|
||||
pub(crate) struct RunnerFailureRecord {
|
||||
#[cultcache(key = 0)]
|
||||
pub(crate) schema_version: String,
|
||||
#[cultcache(key = 1)]
|
||||
pub(crate) transaction_id: String,
|
||||
#[cultcache(key = 2)]
|
||||
pub(crate) command_id: String,
|
||||
#[cultcache(key = 3)]
|
||||
pub(crate) target: String,
|
||||
#[cultcache(key = 4)]
|
||||
pub(crate) subject_kind: SubjectKind,
|
||||
#[cultcache(key = 5)]
|
||||
pub(crate) subject_id: String,
|
||||
#[cultcache(key = 6)]
|
||||
pub(crate) runner: String,
|
||||
#[cultcache(key = 7)]
|
||||
pub(crate) exit_code: Option<i32>,
|
||||
#[cultcache(key = 8)]
|
||||
pub(crate) signal: Option<i32>,
|
||||
#[cultcache(key = 9)]
|
||||
pub(crate) stdout_tail: Vec<u8>,
|
||||
#[cultcache(key = 10)]
|
||||
pub(crate) stdout_total_bytes: u64,
|
||||
#[cultcache(key = 11)]
|
||||
pub(crate) stderr_tail: Vec<u8>,
|
||||
#[cultcache(key = 12)]
|
||||
pub(crate) stderr_total_bytes: u64,
|
||||
#[cultcache(key = 13)]
|
||||
pub(crate) recorded_at_unix_millis: u64,
|
||||
}
|
||||
|
||||
/// Tails stay out of `Debug`, so a failed assertion or a logged record never
|
||||
/// prints runner output.
|
||||
impl std::fmt::Debug for RunnerFailureRecord {
|
||||
fn fmt(&self, formatter: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
|
||||
formatter
|
||||
.debug_struct("RunnerFailureRecord")
|
||||
.field("transaction_id", &self.transaction_id)
|
||||
.field("subject_id", &self.subject_id)
|
||||
.finish_non_exhaustive()
|
||||
}
|
||||
}
|
||||
|
||||
impl RunnerFailureRecord {
|
||||
pub(crate) fn from_failure(
|
||||
transaction_id: &str,
|
||||
command_id: &str,
|
||||
target: &str,
|
||||
failure: &RunnerStepFailure,
|
||||
now: u64,
|
||||
) -> Self {
|
||||
let (subject_kind, subject_id) = match &failure.subject {
|
||||
RunnerSubject::Step(id) => (SubjectKind::Step, id),
|
||||
RunnerSubject::ExternalInput(id) => (SubjectKind::ExternalInput, id),
|
||||
};
|
||||
Self {
|
||||
schema_version: RUNNER_FAILURE_SCHEMA.to_owned(),
|
||||
transaction_id: transaction_id.to_owned(),
|
||||
command_id: command_id.to_owned(),
|
||||
target: target.to_owned(),
|
||||
subject_kind,
|
||||
subject_id: subject_id.clone(),
|
||||
runner: failure.runner.clone(),
|
||||
exit_code: failure.exit_code,
|
||||
signal: failure.signal,
|
||||
stdout_tail: failure.stdout_tail.clone(),
|
||||
stdout_total_bytes: failure.stdout_total_bytes,
|
||||
stderr_tail: failure.stderr_tail.clone(),
|
||||
stderr_total_bytes: failure.stderr_total_bytes,
|
||||
recorded_at_unix_millis: now,
|
||||
}
|
||||
}
|
||||
|
||||
pub(crate) fn validate(&self) -> Result<()> {
|
||||
ensure!(
|
||||
self.schema_version == RUNNER_FAILURE_SCHEMA,
|
||||
"runner failure schema is unsupported"
|
||||
);
|
||||
require_id(&self.transaction_id, "runner failure transaction")?;
|
||||
require_id(&self.command_id, "runner failure command")?;
|
||||
require_id(&self.target, "runner failure target")?;
|
||||
require_id(&self.subject_id, "runner failure subject")?;
|
||||
require_id(&self.runner, "runner failure runner")?;
|
||||
for (tail, total, stream) in [
|
||||
(&self.stdout_tail, self.stdout_total_bytes, "stdout"),
|
||||
(&self.stderr_tail, self.stderr_total_bytes, "stderr"),
|
||||
] {
|
||||
ensure!(
|
||||
tail.len() <= RUNNER_OUTPUT_TAIL_BYTES,
|
||||
"runner failure {stream} tail exceeds its bound"
|
||||
);
|
||||
ensure!(
|
||||
tail.len() as u64 <= total,
|
||||
"runner failure {stream} tail exceeds its stream"
|
||||
);
|
||||
}
|
||||
Ok(())
|
||||
}
|
||||
|
||||
fn envelope(&self) -> Result<CultCacheEnvelope> {
|
||||
self.validate()?;
|
||||
typed_envelope(
|
||||
&self.transaction_id,
|
||||
Self::TYPE,
|
||||
RUNNER_FAILURE_SCHEMA,
|
||||
self,
|
||||
self.recorded_at_unix_millis,
|
||||
)
|
||||
}
|
||||
}
|
||||
|
||||
/// One read of the store: the envelopes exactly as stored, and the records
|
||||
/// decoded from them in the same order.
|
||||
fn snapshot(path: &Path) -> Result<(Vec<CultCacheEnvelope>, Vec<RunnerFailureRecord>)> {
|
||||
let envelopes = SingleFileMessagePackBackingStore::new(path)
|
||||
.pull_all_read_only_snapshot()
|
||||
.context("reading the runner evidence store")?;
|
||||
let mut records = Vec::new();
|
||||
for envelope in &envelopes {
|
||||
ensure!(
|
||||
envelope.r#type == RunnerFailureRecord::TYPE
|
||||
&& envelope.schema_id.as_deref() == Some(RUNNER_FAILURE_SCHEMA),
|
||||
"the runner evidence store holds a document that is not a runner failure"
|
||||
);
|
||||
let record: RunnerFailureRecord = decode_record(envelope)?;
|
||||
record.validate()?;
|
||||
ensure!(
|
||||
envelope.key == record.transaction_id,
|
||||
"runner evidence store key differs from the record's transaction"
|
||||
);
|
||||
records.push(record);
|
||||
}
|
||||
Ok((envelopes, records))
|
||||
}
|
||||
|
||||
/// How a write left the store.
|
||||
#[derive(Clone, Copy, Debug, PartialEq, Eq)]
|
||||
pub(crate) enum Kept {
|
||||
Stored,
|
||||
/// The store held bytes that do not decode; it was moved to
|
||||
/// `unreadable_store_path` and the record opened a fresh one. `replaced`
|
||||
/// is the last-written time (unix milliseconds) of an earlier set-aside
|
||||
/// store this one displaced, which is gone: one slot, and it says so.
|
||||
SetAside { replaced: Option<u64> },
|
||||
}
|
||||
|
||||
/// Where an undecodable store is kept. One slot: the next one replaces it, so
|
||||
/// the damage a bad store can cost is one extra file.
|
||||
pub(crate) fn unreadable_store_path(path: &Path) -> PathBuf {
|
||||
path.with_file_name("runner-evidence.unreadable.cc")
|
||||
}
|
||||
|
||||
/// Moves the store over the previous set-aside one, root-only, and reports
|
||||
/// what it displaced. Nothing of the store's content is named: a decode error
|
||||
/// can quote stored values, and none is carried here.
|
||||
fn set_aside_unreadable(path: &Path) -> Result<Option<u64>> {
|
||||
let aside = unreadable_store_path(path);
|
||||
let replaced = std::fs::symlink_metadata(&aside).ok().map(|earlier| {
|
||||
earlier
|
||||
.modified()
|
||||
.ok()
|
||||
.and_then(|time| time.duration_since(std::time::UNIX_EPOCH).ok())
|
||||
.map_or(0, |since| since.as_millis() as u64)
|
||||
});
|
||||
std::fs::rename(path, &aside)
|
||||
.context("the runner evidence store is undecodable and could not be set aside")?;
|
||||
keep_private(&aside)?;
|
||||
Ok(replaced)
|
||||
}
|
||||
|
||||
/// How the writer reads the store's bytes; tests inject an I/O error here.
|
||||
type ReadStore<'a> = &'a dyn Fn(&Path) -> std::io::Result<Vec<u8>>;
|
||||
|
||||
fn unreadable_fault(error: &std::io::Error) -> anyhow::Error {
|
||||
anyhow::anyhow!("the runner evidence store could not be read: {}", error.kind())
|
||||
}
|
||||
|
||||
/// The one writer. Upsert by transaction id; past `RUNNER_EVIDENCE_RECORDS` the
|
||||
/// oldest records go in the same exchange. The file ends at mode 0600.
|
||||
///
|
||||
/// A store is set aside only when bytes that were read do not decode. An I/O
|
||||
/// error leaves the store in place, keeps no evidence for that failure, and
|
||||
/// the error names its kind for the journal.
|
||||
pub(crate) fn record(path: &Path, record: &RunnerFailureRecord) -> Result<Kept> {
|
||||
record_reading(path, record, &|path| std::fs::read(path))
|
||||
}
|
||||
|
||||
fn record_reading(path: &Path, record: &RunnerFailureRecord, read: ReadStore) -> Result<Kept> {
|
||||
let first = match read(path) {
|
||||
Ok(bytes) => Some(bytes),
|
||||
Err(error) if error.kind() == std::io::ErrorKind::NotFound => None,
|
||||
Err(error) => return Err(unreadable_fault(&error)),
|
||||
};
|
||||
let (envelopes, records, how) = match snapshot(path) {
|
||||
Ok((envelopes, records)) => (envelopes, records, Kept::Stored),
|
||||
Err(_) => {
|
||||
let Some(first) = first else {
|
||||
bail!("the runner evidence store could not be decoded");
|
||||
};
|
||||
// Undecodable only if the same bytes are still there to read: a
|
||||
// second read that fails, or finds other bytes, is not a verdict.
|
||||
match read(path) {
|
||||
Ok(second) if second == first => {}
|
||||
Ok(_) => bail!("the runner evidence store changed while it was being read"),
|
||||
Err(error) => return Err(unreadable_fault(&error)),
|
||||
}
|
||||
let replaced = set_aside_unreadable(path)?;
|
||||
(Vec::new(), Vec::new(), Kept::SetAside { replaced })
|
||||
}
|
||||
};
|
||||
let mut kept: Vec<(u64, CultCacheEnvelope)> = envelopes
|
||||
.iter()
|
||||
.zip(&records)
|
||||
.filter(|(_, stored)| stored.transaction_id != record.transaction_id)
|
||||
.map(|(envelope, stored)| (stored.recorded_at_unix_millis, envelope.clone()))
|
||||
.collect();
|
||||
kept.push((record.recorded_at_unix_millis, record.envelope()?));
|
||||
// Stable: records of one instant keep their stored order, so the record
|
||||
// being written survives a tie.
|
||||
kept.sort_by_key(|(recorded_at, _)| *recorded_at);
|
||||
let excess = kept.len().saturating_sub(RUNNER_EVIDENCE_RECORDS);
|
||||
let replacements: Vec<_> = kept
|
||||
.into_iter()
|
||||
.skip(excess)
|
||||
.map(|(_, envelope)| envelope)
|
||||
.collect();
|
||||
match exchange_behind_private_lock(path, &envelopes, &replacements)? {
|
||||
TryCompareExchangeSnapshotOutcome::Exchanged => {}
|
||||
TryCompareExchangeSnapshotOutcome::Mismatch => {
|
||||
bail!("the runner evidence store changed while it was being written")
|
||||
}
|
||||
TryCompareExchangeSnapshotOutcome::LockContended => {
|
||||
return Err(LockContended {
|
||||
store: "runner evidence store",
|
||||
}
|
||||
.into());
|
||||
}
|
||||
}
|
||||
keep_private(path)?;
|
||||
Ok(how)
|
||||
}
|
||||
|
||||
#[cfg(unix)]
|
||||
fn keep_private(path: &Path) -> Result<()> {
|
||||
use std::os::unix::fs::PermissionsExt;
|
||||
std::fs::set_permissions(path, std::fs::Permissions::from_mode(0o600))
|
||||
.context("making the runner evidence store private")
|
||||
}
|
||||
|
||||
#[cfg(not(unix))]
|
||||
fn keep_private(_path: &Path) -> Result<()> {
|
||||
Ok(())
|
||||
}
|
||||
|
||||
/// The one reader. An absent store holds nothing.
|
||||
pub(crate) fn for_transaction(
|
||||
path: &Path,
|
||||
transaction_id: &str,
|
||||
) -> Result<Option<RunnerFailureRecord>> {
|
||||
Ok(snapshot(path)?
|
||||
.1
|
||||
.into_iter()
|
||||
.find(|stored| stored.transaction_id == transaction_id))
|
||||
}
|
||||
|
||||
/// The record as `idunn status --command` shows it, as raw lines. They are
|
||||
/// not text yet: `show_status` is the only caller, and `emit` the only
|
||||
/// writer.
|
||||
fn render(record: &RunnerFailureRecord) -> Vec<Vec<u8>> {
|
||||
let noun = match record.subject_kind {
|
||||
SubjectKind::Step => "step",
|
||||
SubjectKind::ExternalInput => "external input",
|
||||
};
|
||||
let mut lines: Vec<Vec<u8>> = vec![
|
||||
format!(" failed {noun} {}", record.subject_id).into_bytes(),
|
||||
format!(" runner {}", record.runner).into_bytes(),
|
||||
format!(" status {}", runner_status(record.exit_code, record.signal)).into_bytes(),
|
||||
];
|
||||
for (stream, tail, total) in [
|
||||
("stdout", &record.stdout_tail, record.stdout_total_bytes),
|
||||
("stderr", &record.stderr_tail, record.stderr_total_bytes),
|
||||
] {
|
||||
lines.push(format!(" {stream} (last {} of {total} bytes)", tail.len()).into_bytes());
|
||||
let text = tail.strip_suffix(b"\n").unwrap_or(tail);
|
||||
if !text.is_empty() {
|
||||
lines.extend(text.split(|byte| *byte == b'\n').map(|line| {
|
||||
let mut shown = b" | ".to_vec();
|
||||
shown.extend_from_slice(line);
|
||||
shown
|
||||
}));
|
||||
}
|
||||
}
|
||||
lines
|
||||
}
|
||||
|
||||
/// What `idunn status --command` prints for a transaction's runner evidence:
|
||||
/// its record, or one line naming why the store is unreadable. Nothing is
|
||||
/// written when the transaction has no record.
|
||||
pub(crate) fn show_status(
|
||||
out: &mut impl std::io::Write,
|
||||
store: &Path,
|
||||
transaction_id: &str,
|
||||
) -> std::io::Result<()> {
|
||||
let lines = match for_transaction(store, transaction_id) {
|
||||
Ok(Some(record)) => render(&record),
|
||||
Ok(None) => return Ok(()),
|
||||
Err(error) => {
|
||||
vec![format!(" runner evidence unreadable: {}", error.root_cause()).into_bytes()]
|
||||
}
|
||||
};
|
||||
emit(out, &lines)
|
||||
}
|
||||
|
||||
/// The one sink: every byte the status read prints from the evidence store,
|
||||
/// or from an error about it, is written here and nowhere else. An allowlist:
|
||||
/// printable ASCII and tab pass; every other code point is a visible
|
||||
/// `\u{..}` escape and every invalid UTF-8 byte a `\x..` escape. A newline
|
||||
/// inside a line is escaped too: this function writes the only ones.
|
||||
pub(crate) fn emit(out: &mut impl std::io::Write, lines: &[Vec<u8>]) -> std::io::Result<()> {
|
||||
for line in lines {
|
||||
out.write_all(escape(line).as_bytes())?;
|
||||
out.write_all(b"\n")?;
|
||||
}
|
||||
Ok(())
|
||||
}
|
||||
|
||||
pub(crate) fn escape(bytes: &[u8]) -> String {
|
||||
let mut shown = String::with_capacity(bytes.len());
|
||||
for chunk in bytes.utf8_chunks() {
|
||||
for character in chunk.valid().chars() {
|
||||
match character {
|
||||
' '..='~' | '\t' => shown.push(character),
|
||||
_ => shown.push_str(&format!("\\u{{{:x}}}", u32::from(character))),
|
||||
}
|
||||
}
|
||||
for byte in chunk.invalid() {
|
||||
shown.push_str(&format!("\\x{byte:02x}"));
|
||||
}
|
||||
}
|
||||
shown
|
||||
}
|
||||
|
||||
#[cfg(all(test, unix))]
|
||||
mod tests {
|
||||
use super::*;
|
||||
use std::os::unix::fs::PermissionsExt;
|
||||
|
||||
fn failure(subject: RunnerSubject, stdout: &[u8], stderr: &[u8]) -> RunnerStepFailure {
|
||||
RunnerStepFailure {
|
||||
subject,
|
||||
runner: "rust".into(),
|
||||
exit_code: Some(3),
|
||||
signal: None,
|
||||
stdout_tail: stdout.to_vec(),
|
||||
stdout_total_bytes: stdout.len() as u64,
|
||||
stderr_tail: stderr.to_vec(),
|
||||
stderr_total_bytes: stderr.len() as u64,
|
||||
}
|
||||
}
|
||||
|
||||
/// What status writes for a record, through the one sink.
|
||||
fn shown(record: &RunnerFailureRecord) -> String {
|
||||
let mut written = Vec::new();
|
||||
emit(&mut written, &render(record)).expect("a Vec takes any write");
|
||||
String::from_utf8(written).expect("the sink writes ASCII only")
|
||||
}
|
||||
|
||||
fn shown_lines(record: &RunnerFailureRecord) -> Vec<String> {
|
||||
shown(record).lines().map(str::to_owned).collect()
|
||||
}
|
||||
|
||||
fn record_at(transaction: &str, now: u64) -> RunnerFailureRecord {
|
||||
RunnerFailureRecord::from_failure(
|
||||
transaction,
|
||||
"up-service",
|
||||
"service",
|
||||
&failure(RunnerSubject::Step("build".into()), b"out\n", b"err\n"),
|
||||
now,
|
||||
)
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn the_evidence_store_keeps_the_newest_32() -> Result<()> {
|
||||
let temp = tempfile::tempdir()?;
|
||||
let path = runner_evidence_store_path(&temp.path().join("control.cc"));
|
||||
// The literal, not the constant: the spec's bound is 32.
|
||||
for index in 0..=32 {
|
||||
record(&path, &record_at(&format!("tx-{index:02}"), 1000 + index as u64))?;
|
||||
}
|
||||
let kept: Vec<_> = snapshot(&path)?.1.into_iter().map(|stored| stored.transaction_id).collect();
|
||||
assert_eq!(kept.len(), 32);
|
||||
assert!(!kept.contains(&"tx-00".to_owned()), "the oldest survived");
|
||||
assert!(kept.contains(&"tx-32".to_owned()), "the newest was evicted");
|
||||
assert!(kept.contains(&"tx-01".to_owned()), "the second oldest was evicted early");
|
||||
// Rewriting a resident transaction replaces it and drops nothing.
|
||||
let mut again = record_at("tx-05", 5000);
|
||||
again.stdout_tail = b"rewritten\n".to_vec();
|
||||
again.stdout_total_bytes = 10;
|
||||
record(&path, &again)?;
|
||||
let kept = snapshot(&path)?.1;
|
||||
assert_eq!(kept.len(), 32);
|
||||
assert!(kept.iter().any(|stored| stored.transaction_id == "tx-01"));
|
||||
let stored = for_transaction(&path, "tx-05")?.context("rewritten record is gone")?;
|
||||
assert_eq!(stored.stdout_tail, b"rewritten\n");
|
||||
assert_eq!(for_transaction(&path, "tx-00")?, None);
|
||||
Ok(())
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn a_record_round_trips_byte_for_byte() -> Result<()> {
|
||||
let temp = tempfile::tempdir()?;
|
||||
let path = runner_evidence_store_path(&temp.path().join("control.cc"));
|
||||
let written = RunnerFailureRecord::from_failure(
|
||||
"tx-a",
|
||||
"up-service",
|
||||
"service",
|
||||
&failure(RunnerSubject::ExternalInput("toolchain".into()), &[0, 1, 255, 10], b""),
|
||||
77,
|
||||
);
|
||||
record(&path, &written)?;
|
||||
assert_eq!(for_transaction(&path, "tx-a")?, Some(written));
|
||||
assert_eq!(for_transaction(&path, "tx-b")?, None);
|
||||
Ok(())
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn the_evidence_store_is_private() -> Result<()> {
|
||||
let temp = tempfile::tempdir()?;
|
||||
let path = runner_evidence_store_path(&temp.path().join("control.cc"));
|
||||
// Whatever the umask, the mode is the store's own rule, on every write.
|
||||
for index in 0..2 {
|
||||
record(&path, &record_at(&format!("tx-{index}"), 10 + index))?;
|
||||
assert_eq!(std::fs::metadata(&path)?.permissions().mode() & 0o777, 0o600);
|
||||
}
|
||||
Ok(())
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn a_tail_over_its_bound_or_its_stream_is_refused() {
|
||||
let mut oversize = record_at("tx-a", 1);
|
||||
oversize.stdout_tail = vec![b'x'; RUNNER_OUTPUT_TAIL_BYTES + 1];
|
||||
oversize.stdout_total_bytes = u64::MAX;
|
||||
assert!(oversize.validate().is_err());
|
||||
let mut impossible = record_at("tx-a", 1);
|
||||
impossible.stderr_total_bytes = 1;
|
||||
assert!(impossible.validate().is_err());
|
||||
let mut exact = record_at("tx-a", 1);
|
||||
exact.stdout_tail = vec![b'x'; RUNNER_OUTPUT_TAIL_BYTES];
|
||||
exact.stdout_total_bytes = RUNNER_OUTPUT_TAIL_BYTES as u64;
|
||||
assert!(exact.validate().is_ok());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn status_renders_the_tail_with_control_bytes_escaped() {
|
||||
let mut stored = record_at("tx-a", 1);
|
||||
stored.stdout_tail = b"first\n\x1b[2Jsecond\x00end\tTab\nlast\n".to_vec();
|
||||
stored.stdout_total_bytes = 5000;
|
||||
stored.stderr_tail = b"bad\r\x07".to_vec();
|
||||
stored.stderr_total_bytes = 6;
|
||||
let lines = shown_lines(&stored);
|
||||
let text = lines.join("\n");
|
||||
assert!(!text.contains('\u{1b}') && !text.contains('\0') && !text.contains('\r'));
|
||||
assert!(!text.contains('\u{7}'));
|
||||
assert_eq!(
|
||||
lines,
|
||||
vec![
|
||||
" failed step build",
|
||||
" runner rust",
|
||||
" status exit status 3",
|
||||
" stdout (last 30 of 5000 bytes)",
|
||||
" | first",
|
||||
" | \\u{1b}[2Jsecond\\u{0}end\tTab",
|
||||
" | last",
|
||||
" stderr (last 5 of 6 bytes)",
|
||||
" | bad\\u{d}\\u{7}",
|
||||
]
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn debug_names_the_record_and_prints_no_tail() {
|
||||
let mut stored = record_at("tx-a", 1);
|
||||
stored.stdout_tail = b"SECRET-OUTPUT".to_vec();
|
||||
stored.stdout_total_bytes = 13;
|
||||
let debug = format!("{stored:?}");
|
||||
assert!(debug.contains("tx-a") && !debug.contains("SECRET"), "{debug}");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn an_empty_tail_renders_only_its_header() {
|
||||
let mut stored = record_at("tx-a", 1);
|
||||
stored.stdout_tail = Vec::new();
|
||||
stored.stdout_total_bytes = 0;
|
||||
let lines = shown_lines(&stored);
|
||||
assert!(lines.contains(&" stdout (last 0 of 0 bytes)".to_owned()));
|
||||
assert!(!lines.iter().any(|line| line == " | "));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn the_reader_names_an_unreadable_store_as_an_error() -> Result<()> {
|
||||
let temp = tempfile::tempdir()?;
|
||||
let path = runner_evidence_store_path(&temp.path().join("control.cc"));
|
||||
std::fs::write(&path, b"\xc1 not a store")?;
|
||||
assert!(for_transaction(&path, "tx-a").is_err());
|
||||
Ok(())
|
||||
}
|
||||
|
||||
fn files_beside(path: &Path) -> Result<Vec<String>> {
|
||||
let mut names: Vec<String> = std::fs::read_dir(path.parent().context("no parent")?)?
|
||||
.map(|entry| Ok(entry?.file_name().to_string_lossy().into_owned()))
|
||||
.collect::<Result<_>>()?;
|
||||
names.retain(|name| name.starts_with("runner-evidence") && !name.contains("lock"));
|
||||
names.sort();
|
||||
Ok(names)
|
||||
}
|
||||
|
||||
/// A store a newer Idunn wrote: well-formed, but a schema this one does not know.
|
||||
fn write_future_store(path: &Path) -> Result<()> {
|
||||
let mut future = record_at("tx-future", 1);
|
||||
future.schema_version = "idunn.runner_failure.v2".into();
|
||||
let envelope = typed_envelope(
|
||||
&future.transaction_id,
|
||||
RunnerFailureRecord::TYPE,
|
||||
"idunn.runner_failure.v2",
|
||||
&future,
|
||||
1,
|
||||
)?;
|
||||
let outcome = exchange_behind_private_lock(path, &[], &[envelope])?;
|
||||
ensure!(
|
||||
matches!(outcome, TryCompareExchangeSnapshotOutcome::Exchanged),
|
||||
"the future store was not written"
|
||||
);
|
||||
Ok(())
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn a_corrupt_store_is_set_aside_and_the_next_failure_keeps_its_evidence() -> Result<()> {
|
||||
let temp = tempfile::tempdir()?;
|
||||
let path = runner_evidence_store_path(&temp.path().join("control.cc"));
|
||||
let damage = b"\xc1 not a store \x00 damage";
|
||||
std::fs::write(&path, damage)?;
|
||||
let kept = record(&path, &record_at("tx-a", 5))?;
|
||||
assert_eq!(kept, Kept::SetAside { replaced: None });
|
||||
// Kept, not deleted: the damage is where the journal says, root-only.
|
||||
let aside = unreadable_store_path(&path);
|
||||
assert_eq!(std::fs::read(&aside)?, damage);
|
||||
assert_eq!(std::fs::metadata(&aside)?.permissions().mode() & 0o777, 0o600);
|
||||
// The failure's evidence is in a fresh, private store.
|
||||
assert_eq!(
|
||||
for_transaction(&path, "tx-a")?.map(|stored| stored.transaction_id),
|
||||
Some("tx-a".to_owned())
|
||||
);
|
||||
assert_eq!(std::fs::metadata(&path)?.permissions().mode() & 0o777, 0o600);
|
||||
// A healthy store is not set aside again.
|
||||
assert_eq!(record(&path, &record_at("tx-b", 6))?, Kept::Stored);
|
||||
assert_eq!(std::fs::read(&aside)?, damage);
|
||||
Ok(())
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn a_future_version_store_is_set_aside_and_the_next_failure_keeps_its_evidence() -> Result<()> {
|
||||
let temp = tempfile::tempdir()?;
|
||||
let path = runner_evidence_store_path(&temp.path().join("control.cc"));
|
||||
write_future_store(&path)?;
|
||||
let before = std::fs::read(&path)?;
|
||||
assert!(for_transaction(&path, "tx-future").is_err());
|
||||
assert_eq!(
|
||||
record(&path, &record_at("tx-a", 5))?,
|
||||
Kept::SetAside { replaced: None }
|
||||
);
|
||||
assert_eq!(std::fs::read(unreadable_store_path(&path))?, before);
|
||||
assert!(for_transaction(&path, "tx-a")?.is_some());
|
||||
assert_eq!(for_transaction(&path, "tx-future")?, None);
|
||||
Ok(())
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn at_most_one_unreadable_store_is_kept() -> Result<()> {
|
||||
let temp = tempfile::tempdir()?;
|
||||
let path = runner_evidence_store_path(&temp.path().join("control.cc"));
|
||||
let damages: [&[u8]; 3] = [b"\xc1 first", b"\xc1 second", b"\xc1 third"];
|
||||
for (index, damage) in damages.into_iter().enumerate() {
|
||||
std::fs::write(&path, damage)?;
|
||||
record(&path, &record_at(&format!("tx-{index}"), 5 + index as u64))?;
|
||||
assert_eq!(
|
||||
std::fs::read(unreadable_store_path(&path))?,
|
||||
damage,
|
||||
"the newest damage is the one kept"
|
||||
);
|
||||
}
|
||||
assert_eq!(
|
||||
files_beside(&path)?,
|
||||
vec![
|
||||
"runner-evidence.cc".to_owned(),
|
||||
"runner-evidence.unreadable.cc".to_owned()
|
||||
]
|
||||
);
|
||||
Ok(())
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn a_store_that_cannot_be_read_is_left_in_place_and_the_write_fails() -> Result<()> {
|
||||
let temp = tempfile::tempdir()?;
|
||||
let path = runner_evidence_store_path(&temp.path().join("control.cc"));
|
||||
// A directory where the store should be: unreadable, not undecodable.
|
||||
std::fs::create_dir(&path)?;
|
||||
assert!(record(&path, &record_at("tx-a", 1)).is_err());
|
||||
assert!(path.is_dir());
|
||||
assert!(!unreadable_store_path(&path).exists());
|
||||
Ok(())
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn every_code_point_but_printable_ascii_is_a_visible_escape() {
|
||||
// The allowlist, exhaustively: no class list decides anything.
|
||||
for code in 0..=0x10ffffu32 {
|
||||
let Some(character) = char::from_u32(code) else {
|
||||
continue;
|
||||
};
|
||||
let text = character.to_string();
|
||||
let shown = escape(text.as_bytes());
|
||||
if matches!(character, ' '..='~' | '\t') {
|
||||
assert_eq!(shown, text, "U+{code:04X} should pass");
|
||||
} else {
|
||||
assert_eq!(shown, format!("\\u{{{code:x}}}"), "U+{code:04X}");
|
||||
}
|
||||
}
|
||||
// Invalid UTF-8 is escaped byte by byte; valid text round it passes.
|
||||
assert_eq!(escape(b"a\xc1\xffb\xe2\x82"), "a\\xc1\\xffb\\xe2\\x82");
|
||||
assert_eq!(escape(b"ok \xe2\x82\xac"), "ok \\u{20ac}");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn a_newline_inside_a_line_cannot_forge_a_line() {
|
||||
let mut stored = record_at("tx-a", 1);
|
||||
stored.subject_id = "build\n status exit status 0".into();
|
||||
let lines = shown_lines(&stored);
|
||||
assert_eq!(lines[0], " failed step build\\u{a} status exit status 0");
|
||||
assert_eq!(
|
||||
lines.iter().filter(|line| line.starts_with(" status ")).count(),
|
||||
1
|
||||
);
|
||||
}
|
||||
|
||||
/// A store whose first record is well-formed except that its subject kind
|
||||
/// holds `marker`: the decode error quotes it.
|
||||
fn store_with_hostile_subject_kind(path: &Path, marker: &[u8; 4]) -> Result<()> {
|
||||
record(path, &record_at("tx-a", 1))?;
|
||||
let mut bytes = std::fs::read(path)?;
|
||||
let at = bytes
|
||||
.windows(4)
|
||||
.position(|window| window == b"step")
|
||||
.context("no step marker")?;
|
||||
bytes[at..at + 4].copy_from_slice(marker);
|
||||
std::fs::write(path, bytes)?;
|
||||
Ok(())
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn the_unreadable_store_line_is_escaped_with_the_rest() -> Result<()> {
|
||||
let temp = tempfile::tempdir()?;
|
||||
let path = runner_evidence_store_path(&temp.path().join("control.cc"));
|
||||
store_with_hostile_subject_kind(&path, b"\x1b[2J")?;
|
||||
let mut written = Vec::new();
|
||||
show_status(&mut written, &path, "tx-a")?;
|
||||
assert!(!written.contains(&0x1b), "an escape byte reached the terminal");
|
||||
assert!(written.iter().all(|byte| matches!(byte, b' '..=b'~' | b'\t' | b'\n')));
|
||||
let text = String::from_utf8(written)?;
|
||||
assert!(text.starts_with(" runner evidence unreadable: "), "{text}");
|
||||
assert!(text.contains("\\u{1b}[2J"), "the quoted value was not shown escaped: {text}");
|
||||
Ok(())
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn a_failed_set_aside_names_itself_keeps_the_store_and_echoes_nothing() -> Result<()> {
|
||||
let temp = tempfile::tempdir()?;
|
||||
let path = runner_evidence_store_path(&temp.path().join("control.cc"));
|
||||
store_with_hostile_subject_kind(&path, b"CNRY")?;
|
||||
let damage = std::fs::read(&path)?;
|
||||
// Sanity: this damage decodes to an error that quotes the canary.
|
||||
assert!(format!("{:#}", snapshot(&path).unwrap_err()).contains("CNRY"));
|
||||
// A non-empty directory where the aside goes: rename fails even as root.
|
||||
let aside = unreadable_store_path(&path);
|
||||
std::fs::create_dir(&aside)?;
|
||||
std::fs::write(aside.join("occupied"), b"x")?;
|
||||
let error = record(&path, &record_at("tx-b", 2)).unwrap_err();
|
||||
for text in [format!("{error:#}"), format!("{error:?}")] {
|
||||
assert!(text.contains("could not be set aside"), "{text}");
|
||||
assert!(!text.contains("CNRY"), "stored content reached the error: {text}");
|
||||
}
|
||||
// The failure cost nothing: the store is as it was, with no evidence
|
||||
// written over it.
|
||||
assert_eq!(std::fs::read(&path)?, damage);
|
||||
assert!(aside.join("occupied").exists());
|
||||
Ok(())
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn an_io_error_leaves_the_store_in_place_and_keeps_no_evidence() -> Result<()> {
|
||||
let temp = tempfile::tempdir()?;
|
||||
let path = runner_evidence_store_path(&temp.path().join("control.cc"));
|
||||
record(&path, &record_at("tx-a", 1))?;
|
||||
let before = std::fs::read(&path)?;
|
||||
let denied = || std::io::Error::from(std::io::ErrorKind::PermissionDenied);
|
||||
// The first read fails on a healthy store.
|
||||
let error = record_reading(&path, &record_at("tx-b", 2), &|_| Err(denied())).unwrap_err();
|
||||
assert!(format!("{error:#}").contains("permission denied"), "{error:#}");
|
||||
assert_eq!(std::fs::read(&path)?, before, "the store changed");
|
||||
assert!(!unreadable_store_path(&path).exists());
|
||||
assert!(for_transaction(&path, "tx-b")?.is_none());
|
||||
// The second read, the one that confirms the bytes do not decode, fails.
|
||||
std::fs::write(&path, b"\xc1 not a store")?;
|
||||
let calls = std::cell::Cell::new(0usize);
|
||||
let read = |path: &Path| {
|
||||
calls.set(calls.get() + 1);
|
||||
if calls.get() == 2 {
|
||||
return Err(denied());
|
||||
}
|
||||
std::fs::read(path)
|
||||
};
|
||||
let error = record_reading(&path, &record_at("tx-b", 2), &read).unwrap_err();
|
||||
assert!(format!("{error:#}").contains("permission denied"), "{error:#}");
|
||||
assert_eq!(std::fs::read(&path)?, b"\xc1 not a store");
|
||||
assert!(!unreadable_store_path(&path).exists());
|
||||
// Bytes that changed between the reads are no verdict either.
|
||||
let calls = std::cell::Cell::new(0usize);
|
||||
let read = |path: &Path| -> std::io::Result<Vec<u8>> {
|
||||
calls.set(calls.get() + 1);
|
||||
let mut bytes = std::fs::read(path)?;
|
||||
bytes.push(calls.get() as u8);
|
||||
Ok(bytes)
|
||||
};
|
||||
assert!(record_reading(&path, &record_at("tx-b", 2), &read).is_err());
|
||||
assert_eq!(std::fs::read(&path)?, b"\xc1 not a store");
|
||||
assert!(!unreadable_store_path(&path).exists());
|
||||
Ok(())
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn a_second_set_aside_says_it_replaced_the_first() -> Result<()> {
|
||||
let temp = tempfile::tempdir()?;
|
||||
let path = runner_evidence_store_path(&temp.path().join("control.cc"));
|
||||
std::fs::write(&path, b"\xc1 first")?;
|
||||
assert_eq!(
|
||||
record(&path, &record_at("tx-a", 1))?,
|
||||
Kept::SetAside { replaced: None }
|
||||
);
|
||||
std::fs::write(&path, b"\xc1 second")?;
|
||||
let Kept::SetAside { replaced: Some(_) } = record(&path, &record_at("tx-b", 2))? else {
|
||||
bail!("a displaced set-aside store went unreported");
|
||||
};
|
||||
assert_eq!(std::fs::read(unreadable_store_path(&path))?, b"\xc1 second");
|
||||
Ok(())
|
||||
}
|
||||
}
|
||||
File diff suppressed because it is too large
Load diff
16
tests/fixtures/README.md
vendored
16
tests/fixtures/README.md
vendored
|
|
@ -10,19 +10,3 @@ It is checked in because the TypeScript runtime emits a camelCase MessagePack
|
|||
map while Rust `DatabaseEntry` derives tuple payloads. Idunn must prove it can
|
||||
decode the producer's real wire shape rather than round-tripping a fixture it
|
||||
serialized itself.
|
||||
|
||||
# systemd 257 `systemctl show` captures
|
||||
|
||||
`systemd-257/*.show` are the verbatim output of systemd 257.13 for the 27
|
||||
properties `show_unit` requested at `112a7a5`, captured read-only on Yggdrasil
|
||||
on 2026-10-24: `vili-presence-identity-only.show` (a failed idunn-vili unit whose
|
||||
only credential is the presence identity) and
|
||||
`odin-presence-and-one-service-credential.show` (the live Odin unit). systemd 257
|
||||
prints `LoadCredential=[unprintable]` in both. Tests read these bytes; do not edit.
|
||||
|
||||
# Pre-cut systemd observation records
|
||||
|
||||
`pre-cut-systemd-observation.{compact,named}.hex` are the two MessagePack
|
||||
encodings of a `SystemdWorkloadObservation` with `load_credential` set to
|
||||
`[unprintable]`, produced by the code at `112a7a5` (the form live incumbents
|
||||
persisted). They pin that later Idunns still decode such records.
|
||||
|
|
|
|||
|
|
@ -1 +0,0 @@
|
|||
dc0033d94e6964756e6e2d616161616161616161616161616161616161616161616161616161616161616161616161616161616161616161616161616161616161616161616161616161612e73657276696365aa4964756e6e2074657374a3696e7601a465786563a26e6fa56d69786564c3a6753631303030a6753631303030a0a0a0c3c3a9696e76697369626c65a3616c6cc3a430303037a0ad5b756e7072696e7461626c655dcd10920194cdee48cdee48cdee48cdee4894cdee48cdee48cdee48cdee4891cdee480000000000c391010708b52f6f70742f746573742f62696e2f736572766963650101d9477368613235362d31313131313131313131313131313131313131313131313131313131313131313131313131313131313131313131313131313131313131313131313131313131d9477368613235362d61616161616161616161616161616161616161616161616161616161616161616161616161616161616161616161616161616161616161616161616161616161a92f6f70742f74657374a92f72756e2f74657374d9477368613235362d3232323232323232323232323232323232323232323232323232323232323232323232323232323232323232323232323232323232323232323232323232323290d9477368613235362d33333333333333333333333333333333333333333333333333333333333333333333333333333333333333333333333333333333333333333333333333333333d9202f73797374656d2e736c6963652f6964756e6e2d746573742e73657276696365c090aa61637469766174696f6edc0020010101010101010101010101010101010101010101010101010101010101010190
|
||||
|
|
@ -1 +0,0 @@
|
|||
de0033a4756e6974d94e6964756e6e2d616161616161616161616161616161616161616161616161616161616161616161616161616161616161616161616161616161616161616161616161616161612e73657276696365b0756e69745f6465736372697074696f6eaa4964756e6e2074657374ad696e766f636174696f6e5f6964a3696e76d923657865635f6d61696e5f73746172745f74696d657374616d705f6d6f6e6f746f6e696301ac736572766963655f74797065a465786563ae726573746172745f706f6c696379a26e6fa96b696c6c5f6d6f6465a56d69786564ac64796e616d69635f75736572c3ac73797374656d645f75736572a6753631303030ad73797374656d645f67726f7570a6753631303030b4737570706c656d656e746172795f67726f757073a0b76361706162696c6974795f626f756e64696e675f736574a0b4616d6269656e745f6361706162696c6974696573a0ae707269766174655f6d6f756e7473c3ac707269766174655f70696473c3ac70726f746563745f70726f63a9696e76697369626c65ab70726f635f737562736574a3616c6cb16e6f5f6e65775f70726976696c65676573c3a5756d61736ba430303037b2696e61636365737369626c655f7061746873a0af6c6f61645f63726564656e7469616cad5b756e7072696e7461626c655da86d61696e5f706964cd1092b270726f636573735f73746172745f74696d6501ac70726f636573735f7569647394cdee48cdee48cdee48cdee48ac70726f636573735f6769647394cdee48cdee48cdee48cdee48ae70726f636573735f67726f75707391cdee48b770726f636573735f6361705f696e686572697461626c6500b570726f636573735f6361705f7065726d697474656400b570726f636573735f6361705f65666665637469766500b470726f636573735f6361705f626f756e64696e6700b370726f636573735f6361705f616d6269656e7400b970726f636573735f6e6f5f6e65775f70726976696c65676573c3b670726f636573735f6e616d6573706163655f706964739101b26d6f756e745f6e616d6573706163655f696407b07069645f6e616d6573706163655f696408aa65786563757461626c65b52f6f70742f746573742f62696e2f73657276696365b165786563757461626c655f64657669636501b065786563757461626c655f696e6f646501b165786563757461626c655f736861323536d9477368613235362d31313131313131313131313131313131313131313131313131313131313131313131313131313131313131313131313131313131313131313131313131313131b372756e74696d655f696e7374616e63655f6964d9477368613235362d61616161616161616161616161616161616161616161616161616161616161616161616161616161616161616161616161616161616161616161616161616161b1776f726b696e675f6469726563746f7279a92f6f70742f74657374ae72756e74696d655f62756e646c65a92f72756e2f74657374b3636f6d6d616e645f6c696e655f736861323536d9477368613235362d32323232323232323232323232323232323232323232323232323232323232323232323232323232323232323232323232323232323232323232323232323232b1656e7669726f6e6d656e745f6e616d657390bb656e7669726f6e6d656e745f636f6e74726163745f736861323536d9477368613235362d33333333333333333333333333333333333333333333333333333333333333333333333333333333333333333333333333333333333333333333333333333333ad636f6e74726f6c5f67726f7570d9202f73797374656d2e736c6963652f6964756e6e2d746573742e73657276696365b563726564656e7469616c735f6469726563746f7279c0bc706172656e745f6f6e6c795f66696c655f64657363726970746f727390bd61637469766174696f6e5f7369676e65725f6964656e746974795f6964aa61637469766174696f6ebc61637469766174696f6e5f7369676e65725f7075626c69635f6b6579dc00200101010101010101010101010101010101010101010101010101010101010101b3736572766963655f63726564656e7469616c7390
|
||||
|
|
@ -1,28 +0,0 @@
|
|||
Type=exec
|
||||
Restart=no
|
||||
MainPID=2571
|
||||
OpenFile=/run/idunn/activation-credentials/sha256-7cfe5cd1e87839e029c7425eb1b586b6b0d94dd79d7376eed3b156386f2c7af8.credential:gamecult-idunn-runtime-activation-key:read-only
|
||||
OpenFile=/etc/gamecult/odin/runtime-presence-identity.cc:gamecult-runtime-presence-identity:read-only
|
||||
ExecMainStartTimestampMonotonic=20630000
|
||||
ControlGroup=/system.slice/idunn-odin-7cfe5cd1e87839e029c7425eb1b586b6b0d94dd79d7376eed3b156386f2c7af8.service
|
||||
UMask=0007
|
||||
WorkingDirectory=/srv/odin/releases/sha256-e5c7d0b2b9c0a8ffba368aeb08abdbe37ae0d17484a785e60a66ec4a62b2f18c
|
||||
CapabilityBoundingSet=
|
||||
AmbientCapabilities=
|
||||
User=_duc23369f1b8b167db
|
||||
Group=odin
|
||||
DynamicUser=yes
|
||||
LoadCredential=[unprintable]
|
||||
SupplementaryGroups=
|
||||
InaccessiblePaths=/run/idunn/activation-credentials
|
||||
PrivateMounts=yes
|
||||
PrivatePIDs=yes
|
||||
NoNewPrivileges=yes
|
||||
ProtectProc=invisible
|
||||
ProcSubset=all
|
||||
KillMode=mixed
|
||||
Description=Idunn odin sha256-7cfe5cd1e87839e029c7425eb1b586b6b0d94dd79d7376eed3b156386f2c7af8
|
||||
LoadState=loaded
|
||||
ActiveState=active
|
||||
SubState=running
|
||||
InvocationID=ada48921043449f39dacc87b14a1bf70
|
||||
|
|
@ -1,28 +0,0 @@
|
|||
Type=exec
|
||||
Restart=no
|
||||
MainPID=0
|
||||
OpenFile=/run/idunn/activation-credentials/sha256-b507e1705a70f8eee72f355e7db637acab8eb37880b5d25245fda68d80f63041.credential:gamecult-idunn-runtime-activation-key:read-only
|
||||
OpenFile=/etc/gamecult/vili/runtime-presence-identity.cc:gamecult-runtime-presence-identity:read-only
|
||||
ExecMainStartTimestampMonotonic=520083420000
|
||||
ControlGroup=
|
||||
UMask=0007
|
||||
WorkingDirectory=/srv/vili-idunn/releases/sha256-aeaee4e1669742572c45a3f26bc86893c44b2a4149520975eb7d8a28aa931906
|
||||
CapabilityBoundingSet=
|
||||
AmbientCapabilities=
|
||||
User=_du8c06c563aac6d205
|
||||
Group=vili-state
|
||||
DynamicUser=yes
|
||||
LoadCredential=[unprintable]
|
||||
SupplementaryGroups=
|
||||
InaccessiblePaths=/run/idunn/activation-credentials
|
||||
PrivateMounts=yes
|
||||
PrivatePIDs=yes
|
||||
NoNewPrivileges=yes
|
||||
ProtectProc=invisible
|
||||
ProcSubset=all
|
||||
KillMode=mixed
|
||||
Description=Idunn vili sha256-b507e1705a70f8eee72f355e7db637acab8eb37880b5d25245fda68d80f63041
|
||||
LoadState=loaded
|
||||
ActiveState=failed
|
||||
SubState=failed
|
||||
InvocationID=8f7a349f2f8f49fe87ac5d89239260b4
|
||||
Loading…
Reference in a new issue