Imagination r2 for cut tree-executable-members after Soul s1: link census of the real PBS and Blender trees, why lexical normalisation does not prove containment, the leading-dotdot-within-depth rule and its prior art, members on host targets carried not applied, one seal step for collect and host. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
201 lines
13 KiB
Markdown
201 lines
13 KiB
Markdown
# Deployment provenance: map
|
|
|
|
Status: map, Imagination, 2026-10-23 (campaign `idunn-deploy-provenance`,
|
|
session `self-2026-10-23-eureka-b`). The ends are in
|
|
[`deploy-provenance-target.md`](deploy-provenance-target.md). Questions,
|
|
rulings, cut specs and follow-ups live in Eureka's mind; this page holds what
|
|
has no kind: body facts, rationale and sequencing. Line numbers in T1-T12 are
|
|
at `d74ac8d` (origin/main on 2026-10-23); T4, T9 and T13-T16 are at `fd040f0`,
|
|
the head of Hands attempt 1, which revision 2 of the spec builds on.
|
|
|
|
## Cut `tree-executable-members`
|
|
|
|
Spec `idunn-deploy-provenance:cut_spec:cut-tree-executable-members.r2`
|
|
(revision 2, after Soul pass 1 found the real interpreter tree refused). It
|
|
serves `motion-pipeline:follow_up:idunn-tree-executable-members` (Vili's
|
|
admission waits on it) and will serve gamecult-chat C3 too: the presence
|
|
launcher runs a child such as `upstream/bin/daemon` from inside a tree artifact
|
|
(README, launcher fragment), and Synapse needs a Python interpreter there.
|
|
|
|
### Body facts
|
|
|
|
- T1. `ArtifactOutput` (`src/deployment.rs:106-117`) carries one `executable`
|
|
bool. `ArtifactReceipt` (`src/deployment_plan.rs:731-737`) copies it and
|
|
`validate` (`:740-760`) requires it to equal the declaration.
|
|
- T2. `harden_installed_release` (`src/drivers.rs:6848-6855`) builds its
|
|
executable set from `root.join(receipt.destination)` only. `harden_root_tree`
|
|
(`:6858-6892`) seals every directory 0555 and every file 0444 unless that
|
|
exact path is in the set. A directory artifact with `executable = true` is
|
|
therefore silently meaningless today: its files are all 0444. Nothing refuses
|
|
it. No live recipe declares one (CodexConnector, Ghostlight, Heimdall, Odin,
|
|
StreamPixels service and web surveyed): `executable = true` is on files only.
|
|
- T3. A tree output is copied by `copy_artifact`/`copy_tree`
|
|
(`src/drivers.rs:8053-8126`), which recreate symlinks verbatim. The digest
|
|
(`digest_tree`, `:8142-8175`) covers names, file bytes and link targets, not
|
|
mode bits. So runner or git exec bits never reach the release: harden is the
|
|
only mode authority.
|
|
- T4. `validate_release_symlink` (`src/drivers.rs:6897-6916`) keeps a symlink
|
|
only if its target is relative with no `..`. It runs only inside
|
|
`harden_root_tree`, at install, so a tree with a `..` or absolute link copies,
|
|
digests and seals, then fails at `install_release` (Soul s1: copying and
|
|
digesting the real tree twice took 160 s before harden refused it). The rule
|
|
is sound but too narrow: it refuses the real interpreter tree (T13). The
|
|
frozen-source path has a physical resolver (`resolve_frozen_source_symlink`,
|
|
`:6617`) that walks the filesystem with a 40-link budget and a memo. Its
|
|
first version was exponential (Soul S4-1, recorded in `d562a12`), and it needs
|
|
the tree on disk to answer. Revision 2 does not reuse it (T14).
|
|
- T5. The sealed release id hashes `(schema, plan_id, artifacts,
|
|
external_inputs)` (`src/deployment_plan.rs:1000-1011`), and `plan_id` hashes
|
|
the whole plan including `recipe_blob`. A changed member list is therefore
|
|
already a new plan and a new release. The receipt must still carry the list,
|
|
because `harden_installed_release` reads receipts only; its two callers
|
|
(`install_release` `:2788`, `validate_installed_release` `:2822`) re-harden
|
|
every installed release on each observation.
|
|
- T6. Encoding is `rmp_serde::to_vec`, positional. A trailing field with
|
|
`#[serde(default, skip_serializing_if = "BTreeSet::is_empty")]` keeps every
|
|
existing receipt's bytes and release id: the plan's `deadlines` field
|
|
(`:574-577`) is the precedent.
|
|
- T7. `digest_artifact` and `digest_tree` (`:8128-8175`) `fs::read` each file
|
|
whole. A torch CPU tree's largest file (`libtorch_cpu.so`, hundreds of MB)
|
|
is held in Idunn's memory three times per install (seal, install, every
|
|
validate). `hash_frozen_source_tree` (`:6788-6840`) already streams in 64 KiB
|
|
chunks (S9). The cut collapses both onto one streaming helper.
|
|
- T8. `require_leaf_path` (`src/deployment.rs:1951-1958`) only keeps the
|
|
artifact destination a single filename. Members are paths inside that
|
|
destination, so it does not change. `require_relative_path` (`:1936-1949`)
|
|
with `allow_current = false` already refuses absolute, `..`, root and prefix
|
|
components.
|
|
- T9. The host actuator re-spells the seal (`src/host.rs:1050-1086`): copy,
|
|
mode check, digest, pinned-digest check and receipt, the same steps as
|
|
`collect_artifact` (`src/drivers.rs:2305-2326`). Soul s1: deleting its mode
|
|
check survives every test. Revision 2 collapses both onto one `seal_artifact`
|
|
in `drivers.rs`. Each caller keeps only how it finds the source path.
|
|
- T10. The transient unit runs with `ProtectSystem=strict` and the release in
|
|
`ReadOnlyPaths` (`src/drivers.rs:3675-3698`), `/tmp` mounted `noexec`, and no
|
|
`MemoryDenyWriteExecute`. Executing a 0555 file from the release works.
|
|
Shared libraries need no exec bit.
|
|
- T11. Live body on Yggdrasil: `/usr/local/bin/idunn` sha256 `44cf8a14...`,
|
|
installed 2026-10-20 02:34 CEST, before `6e2a31c` (02:40 CEST). `src/` is
|
|
identical between `8760a4f` and `6e2a31c`, so the live code is the
|
|
reincarnation c2 merge. `origin/main` is 26 commits ahead (+4194/-112 in
|
|
`src/`: the presence launcher and member route). Installing this cut ships all
|
|
of them. Idunn's own body changes only through gamecult-ops
|
|
`scripts/install-idunn-yggdrasil-release.sh` (staged binaries, unit,
|
|
bindings). The stage at `/home/gamecultadmin/idunn-stage` is stale (2026-09-11).
|
|
- T13. Link census of the real interpreter tree (Yggdrasil probe of
|
|
`cpython-3.12.15+20261021-x86_64-unknown-linux-gnu-install_only.tar.gz`,
|
|
sha256 `5ff9f060...77191124`; torch adds no links, as Soul s1 counted 1,049
|
|
after pip): 1,049 links. 0 are absolute, 749 have no `..`, and 300 have all
|
|
their `..` components before the first name (every one is
|
|
`share/terminfo/<c>/<name> -> ../<c2>/<name2>`, one `..` each). 0 have a `..`
|
|
after a name, 0 climb above the tree, 0 dangle and 0 resolve outside it.
|
|
Blender 5.2.2 has 79 links, none absolute or with `..` (Soul s1).
|
|
- T14. Lexically normalising `<link's directory>/<target>` does not check
|
|
containment. The kernel resolves a target one component at a time, so a `..`
|
|
after a symlinked name climbs from where that link points, not from where its
|
|
name sits. Probe on Yggdrasil: `root/a/s -> ..` and `root/a/L -> s/../x`
|
|
normalise to `root` and `root/a/x`, both inside, yet `L` reads the file `x`
|
|
beside `root`. `root/a/d -> .` with `root/a/M -> d/../../x` escapes the same
|
|
way. Idunn has met this before: verify-transaction-cut F5 (`D -> .`,
|
|
`L -> D/D/../../../etc/passwd` passed a lexical check).
|
|
The sound lexical rule constrains the target's shape: all its `..` components
|
|
come before its first name, and they number no more than the link's depth
|
|
below the artifact's top. The climb then starts from the link's own
|
|
directory, which the walk reached through real directories only (copy and
|
|
digest never descend through a link), so it lands on a real ancestor inside
|
|
the artifact. After that the target only descends, and any link it passes
|
|
through obeys the same rule. By induction on the kernel's hops (capped at 40,
|
|
then ELOOP), every hop stays inside. The check is one pass over the target's
|
|
components with no filesystem reads, so its cost is linear in the tree and no
|
|
resolver is needed. The shapes it refuses (`s/../x`) occur in neither real
|
|
tree (T13).
|
|
Prior art, from documentation as known to Imagination (high confidence for Go
|
|
and securejoin, medium for the tar details):
|
|
- Go `path/filepath.IsLocal` (1.20) is lexical and says so: it does not account
|
|
for symlinks. It would admit `s/../x`. Used alone, it is the T14 bug.
|
|
- Go `os.Root` (1.24) and `cyphar/filepath-securejoin` (used by runc,
|
|
containerd and umoci) resolve each component inside the root when a path is
|
|
opened, refusing or clamping an escape. They guard one later access, need
|
|
the tree on disk, and check again on every open. Idunn instead certifies a
|
|
sealed tree once, so that the kernel's own resolution stays inside on every
|
|
later access.
|
|
- Zip-slip defences and archive extractors check entry names lexically and
|
|
protect against link targets by refusing to write through links
|
|
(libarchive's `ARCHIVE_EXTRACT_SECURE_SYMLINKS`, and `SECURE_NODOTDOT` for
|
|
names; GNU tar creates such links last). They protect the extraction, not
|
|
what a kept link points at afterwards.
|
|
- OCI layer extraction keeps link targets verbatim, absolute ones included,
|
|
because the runtime resolves them inside the container's root. Idunn runs a
|
|
release in the host's mount namespace, so absolute targets stay refused.
|
|
- Nix keeps link targets verbatim and tracks dependencies by scanning for
|
|
references. The whole store is its root, so it has no per-artifact
|
|
containment to compare with.
|
|
None of these is a static per-link rule for a tree that runs unconfined. The
|
|
T14 shape rule fits that case.
|
|
- T15. Host targets: `idunn-host` observes, starts and stops processes on
|
|
Windows only (the non-Windows `platform` module in `src/host.rs` bails), and
|
|
`host.rs` sets no file modes. On Windows a file runs by its type, not by a
|
|
mode bit, so `executable` and `executable_members` have nothing to set there.
|
|
The file `executable` flag has always been carried and never applied on host
|
|
targets. A recipe does not know which workload kind its binding will choose.
|
|
- T16. The release root holds only artifact destinations. Runner workspaces,
|
|
where external inputs land, are removed before `SealedRelease::new`
|
|
(`src/drivers.rs:2421-2423`). `install_release` (`:2781-2788`) and
|
|
`validate_installed_release` (`:2815-2822`) digest every receipt's
|
|
destination before calling harden. So a rule in the digest walk runs at seal,
|
|
at install and at every validate, over every link harden would see.
|
|
- T12. Disk: `/` on Yggdrasil has 1.3 TB free. A tree release costs about three
|
|
copies (runner workspace, staging, installed). Nothing collects old releases or
|
|
staging (`ghostlight-verse:follow_up:idunn-generation-retention`), so every
|
|
Vili deploy leaves about 7 GiB behind.
|
|
|
|
### Contract, corrected against the body
|
|
|
|
- Any directory artifact may declare `executable_members` (a set of relative
|
|
paths). The follow-up limited it to runner-output; that limit protects no
|
|
invariant, and the launcher's worktree or fetched trees need the same feature.
|
|
- `executable = true` names a file artifact. On a directory output the release
|
|
is refused, and `executable_members` on a file output is refused (T2).
|
|
- A member is refused at recipe parse if it is empty, absolute, or holds `.`,
|
|
`..`, a root or a prefix. At seal it is refused if it is missing, is a
|
|
symlink, passes through a symlink or non-directory, or is not a regular file.
|
|
Refusal happens before `SealedRelease::new`, so no sealed release can name a
|
|
member it cannot seal.
|
|
- Harden seals each member 0555 and fails if any declared executable path was
|
|
not visited as a regular file. "Every declared executable is sealed 0555, or
|
|
the install fails" holds structurally, not by a lookup that can miss.
|
|
- Symlinks: a link in a tree artifact is kept when its target is relative, its
|
|
`..` components all come before its first name, and they number no more than
|
|
the link's depth below the artifact's top (T14). One function,
|
|
`check_release_link`, decides this. Only the digest walk calls it, so a bad
|
|
link refuses the release at seal, before `SealedRelease::new`, and again at
|
|
install and validate (T16). Harden no longer judges links. The climb is
|
|
bounded by the artifact's top, not the release root: no live release has a
|
|
`..` link (the old rule refused them all), so nothing installed is newly
|
|
refused, and each artifact's digest describes a closed tree.
|
|
- Host targets carry `executable` and `executable_members` in the receipt,
|
|
shape-checked at seal by the shared `seal_artifact`, and do not apply them
|
|
(T15). Refusing members there would make a recipe's validity depend on its
|
|
binding.
|
|
- Vili's trees must
|
|
not come from `python -m venv`: its `bin/python` is an absolute link and its
|
|
`pyvenv.cfg` names the build workspace. Install packages into the relocatable
|
|
interpreter tree instead (`motion-pipeline:follow_up:vili-relocatable-python-tree`).
|
|
|
|
### Authority
|
|
|
|
Owner: the seal (`seal_artifact`, called by `collect_artifact` for systemd
|
|
targets and by `materialize` in `host.rs` for host targets) decides
|
|
membership from the recipe and writes it into the receipt. Harden enforces the
|
|
receipt and decides nothing. Runner exec bits and git `100755` modes are dead
|
|
inputs for release modes. Link containment belongs to `check_release_link`,
|
|
which only `digest_tree` calls. `validate_release_symlink` is deleted, and
|
|
harden's symlink arm leaves links as they are.
|
|
|
|
### Acceptance sizing
|
|
|
|
The big-tree job (spec `verification.operator`) needs about 12 GiB of scratch
|
|
on Yggdrasil: a 3.5 GiB tree, its staging copy, downloads and pip cache. It
|
|
removes it at the end. Revision 2 runs it on the trees as shipped, with the 300
|
|
terminfo links (T13). If a tree holds a link the T14 rule refuses, the job
|
|
prints it and stops, and the link is reported to Self, not deleted.
|