Merge cut/vili-admission: Vili is an Idunn target on Yggdrasil (Soul s1-s4)

motion-pipeline:cut_spec:cut-vili-admission.r1 under the operator's ruling
vili-moves-to-yggdrasil-now. Idunn's process write lease is the one writer
authority (hand-run: one abstract-socket bind per real root); the release carries
the pinned python-build-standalone tree and hash-locked packages; admitted live
under Idunn 804ed75 with end to end, continuity restart and rollout verified.
Follow-up: vili-worker-intake-local-callers.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
Meta Vens 2026-10-24 17:06:43 +02:00
commit c5cf4457c6
21 changed files with 2788 additions and 157 deletions

6
.gitmodules vendored
View file

@ -1,3 +1,9 @@
[submodule "vendor/CultLib"]
path = vendor/CultLib
url = https://github.com/GameCult/CultLib.git
[submodule "vendor/kimodo"]
path = vendor/kimodo
url = https://github.com/nv-tlabs/kimodo.git
[submodule "vendor/SOMA-X"]
path = vendor/SOMA-X
url = https://github.com/NVlabs/SOMA-X.git

View file

@ -4,16 +4,15 @@ Vili is the GameCult Persona animation daemon.
It owns the bridge from Persona performance intent to body-motion generation. Its
backend is Kimodo, run by Vili's own daemon as a CPU-only Python process tree on
Linux. Admission to Idunn on Yggdrasil is the next cut; until then nothing
supervises Vili.
Linux. It is built to run on Yggdrasil as an Idunn target (planned, not live; see Front Door).
## Front Door
- **Value:** a supervised path from a text prompt to generated body motion. The daemon spawns one resident Kimodo worker, caps its threads, and returns motion artifacts (`motion.npz`, optional BVH) for a job.
- **Problem:** Personas and games need generated motion, but cold per-request model loads cost minutes. One daemon owns job identity, worker lifecycle, the thread budget and artifact references, so callers touch no model state.
- **Entry points:** `scripts/vili-daemon.mjs` (the daemon; `npm start`, `npm run health`, `npm test`), `scripts/kimodo-resident-worker.py` (the worker it spawns), `scripts/kimodo-text-encode.py` (the one-shot encoder child), `POST /motion/generate` (see Motion Generation), `docs/verse-service-contract.md` and `docs/motion-pipeline-target.md`.
- **Live:** the process tree below, run by hand: `POST /motion/generate`, `GET /health`, `GET /operator-state`, and the CultCache service store. It produces `motion.npz` and optional BVH on the SOMA skeleton; there is no FBX export.
- **Planned:** admission to Idunn on Yggdrasil with a locked virtual environment, then FBX export and a text-prompt-to-Unity-clip path with no hand steps (`docs/motion-pipeline-target.md`).
- **Entry points:** `scripts/vili-daemon.mjs` (the daemon; `npm start`, `npm test`), `scripts/idunn-entry.mjs` (the process Idunn runs), `deployment/idunn/recipe.toml` (the sealed release), `scripts/kimodo-resident-worker.py` (the worker it spawns), `scripts/kimodo-text-encode.py` (the one-shot encoder child), `POST /motion/generate` (see Motion Generation), `docs/verse-service-contract.md` and `docs/motion-pipeline-target.md`.
- **Live:** the process tree below, run by hand for development and tests: `POST /motion/generate`, `GET /health`, `GET /operator-state`, and the CultCache service store. It produces `motion.npz` and optional BVH on the SOMA skeleton; there is no FBX export.
- **Planned:** the Idunn target `vili` on Yggdrasil at `http://10.77.0.1:8824` over WireGuard, with its provisioning runbook in `gamecult-ops`; neither exists until the first `idunn up` lands. FBX export and a text-prompt-to-Unity-clip path with no hand steps (`docs/motion-pipeline-target.md`).
## Authority
@ -37,8 +36,8 @@ vili-daemon.mjs job intake, worker lifecycle, thread cap, nice 10
kimodo-text-encode.py one child per request with cache misses, on CPU
```
Setup: `git submodule update --init`; in `vendor/CultLib`, `npm ci --ignore-scripts` and
`npm run build --workspace=packages/cultcache-ts`; then `npm ci` here. The worker
Setup: `git submodule update --init` (CultLib, Kimodo, SOMA-X); in `vendor/CultLib`, `npm ci --ignore-scripts` and
`npm run build --workspace=packages/cultcache-ts` and `--workspace=packages/cultnet-ts`; then `npm ci --install-links` here. The worker
needs a Python environment with Kimodo's pins and Kimodo itself importable, and a
Hugging Face cache (`<models root>/hub`) holding the Llama 3 base, both LLM2Vec
adapters and the Kimodo checkpoint. Everything runs offline.
@ -59,7 +58,13 @@ The daemon starts the worker on the first job, on a port the worker binds itself
and reports in its ready line, and respawns it after an exit. The worker runs at
nice 10 with `torch` limited to `--threads` (also `OMP_NUM_THREADS` and
`MKL_NUM_THREADS`), so a shared host keeps its other services. Its log is
`<state>/logs/worker.log`. The daemon owns the whole worker process group: it is killed when the worker exits for any reason, on SIGTERM or SIGINT, and at start-up if a crashed daemon left one (`<state>/worker.pid`). A service unit should also own the tree (systemd `KillMode=control-group`).
`<state>/logs/worker.log`. The daemon owns the whole worker process group: it is killed when the worker exits for any reason, on SIGTERM or SIGINT, and, for a hand-run daemon, at start-up if a crashed predecessor left one (`<state>/worker.pid`). Under Idunn the unit's cgroup owns the tree.
A hand-run daemon holds its state root through a listening socket in Linux's abstract namespace, named by the root's real path: the bind is the atomic claim, so a second daemon on a live root prints that it is held and exits 2, killing nothing, and a dead daemon's claim is gone with it (there is no lock file). Two daemons in separate network namespaces do not exclude each other.
### On Yggdrasil
Idunn runs `node vili/scripts/idunn-entry.mjs --state-root <root> --kimodo-commit <sha>` from a sealed release of three trees: `node`, `vili` (the daemon, worker, child, `node_modules`, cultcache-py) and `python` (python-build-standalone CPython 3.12 with the hash-locked packages, Kimodo, MotionCorrection and SOMA-X installed into it; no venv). The weights are the read-only cache `/srv/vili/models`; state is `/var/lib/gamecult/vili`. Idunn's process write lease is the one writer authority: until the daemon holds it, `POST /motion/generate` answers `503 write-lease-pending` and nothing is written; the lease is asked every 250 ms and at every write, so if it is lost the worker tree is SIGKILLed within a second and the daemon answers `503 write-lease-lost`. From SIGTERM on the daemon accepts nothing and writes and spawns nothing (`503 stopping` for a request in flight), stops the worker tree, and exits 0. Idunn proves readiness with its own `POST /cultnet/snapshot` challenge, answered with a signed runtime presence (an `Origin` is still refused).
`POST /motion/generate` refuses any POST that carries an `Origin` or is not
`application/json` with `403 cross-site`; no route answers a preflight.
@ -68,7 +73,7 @@ Vili writes daemon-owned state to `.vili/vili.service.cc`: operator state, the
command boundary and the transport profile. Jobs are `.vili/jobs/<jobId>.json`,
artifacts `.vili/artifacts/<jobId>/`.
Routes: `GET /health`, `GET /operator-state`, `POST /motion/generate`.
Routes: `GET /health`, `GET /operator-state`, `POST /motion/generate`; under Idunn also `POST /cultnet/snapshot`, Idunn's route challenge.
## Motion Generation

View file

@ -0,0 +1,52 @@
// Assembles the node half of Vili's sealed release (run by the recipe's node-build runner):
// node deployment/idunn/assemble-release.mjs release/vili release/node
// release/vili gets the daemon, the Idunn entry, the worker and child sources, the installed node_modules (installed
// with --install-links, so cultcache-ts and cultnet-ts are real copies with their dependencies and no link reaches
// into vendor/), cultcache-py's sources and the lock files the entry hashes; release/node gets the node binary.
// It refuses to finish if any path it wrote is a symlink whose target is absolute: the seal would refuse the tree.
import { chmodSync, copyFileSync, cpSync, lstatSync, mkdirSync, readdirSync, readlinkSync, rmSync } from "node:fs";
import path from "node:path";
const [viliOut, nodeOut] = process.argv.slice(2);
if (!viliOut || !nodeOut || process.argv.length !== 4) {
console.error("usage: assemble-release.mjs <vili output dir> <node output dir>");
process.exit(2);
}
const copy = (from, to) => cpSync(from, to, { recursive: true, verbatimSymlinks: true, errorOnExist: true });
rmSync(viliOut, { recursive: true, force: true });
rmSync(nodeOut, { recursive: true, force: true });
mkdirSync(path.join(viliOut, "scripts"), { recursive: true });
mkdirSync(path.join(viliOut, "deployment", "idunn"), { recursive: true });
mkdirSync(nodeOut, { recursive: true });
// Runtime sources only: the tests and their fixtures stay out of the release.
for (const entry of readdirSync("scripts", { withFileTypes: true })) {
if (!entry.isFile() || !/\.(mjs|py)$/u.test(entry.name) || /^test[-_]/u.test(entry.name)) continue;
copyFileSync(path.join("scripts", entry.name), path.join(viliOut, "scripts", entry.name));
}
copyFileSync("package.json", path.join(viliOut, "package.json"));
copy("node_modules", path.join(viliOut, "node_modules"));
copy(path.join("vendor", "CultLib", "packages", "cultcache-py", "src"), path.join(viliOut, "vendor", "CultLib", "packages", "cultcache-py", "src"));
for (const lock of ["requirements-build.lock", "requirements.lock"]) {
copyFileSync(path.join("deployment", "idunn", lock), path.join(viliOut, "deployment", "idunn", lock));
}
copyFileSync(process.execPath, path.join(nodeOut, "node"));
chmodSync(path.join(nodeOut, "node"), 0o755);
function refuseAbsoluteLinks(directory) {
for (const entry of readdirSync(directory, { withFileTypes: true })) {
const full = path.join(directory, entry.name);
if (lstatSync(full).isSymbolicLink()) {
if (path.isAbsolute(readlinkSync(full))) {
console.error(`absolute symlink in the release: ${path.relative(process.cwd(), full)}`);
process.exit(1);
}
} else if (entry.isDirectory()) {
refuseAbsoluteLinks(full);
}
}
}
refuseAbsoluteLinks(viliOut);
refuseAbsoluteLinks(nodeOut);

View file

@ -0,0 +1,216 @@
schema = "gamecult.idunn.target_declaration.v1"
target = "vili"
source_stamp_environment = "VILI_BUILD_COMMIT"
required_gitlinks = ["vendor/CultLib", "vendor/kimodo", "vendor/SOMA-X"]
# The one external input: a relocatable CPython. The locked site-packages are installed into this tree itself: a copied
# environment would carry an absolute interpreter link and a recorded build path, and the seal refuses both.
[[external_inputs]]
id = "cpython"
url = "https://github.com/astral-sh/python-build-standalone/releases/download/20261021/cpython-3.12.15+20261021-x86_64-unknown-linux-gnu-install_only.tar.gz"
sha256 = "5ff9f0602ce964a529ed1e8555d3abab6341926d724402ba9a15e9e877191124"
runner = "python-build"
destination = "release/cpython.tar.gz"
# ---- node-build: the daemon, the Idunn entry and its dependencies
[[steps]]
id = "install-cultlib-build-dependencies"
phase = "prepare"
runner = "node-build"
working_directory = "vendor/CultLib"
required_environment = ["HOME", "NPM_CONFIG_CACHE"]
argv = ["npm", "ci", "--ignore-scripts"]
[[steps]]
id = "build-cultcache-ts"
phase = "prepare"
runner = "node-build"
working_directory = "vendor/CultLib"
required_environment = ["HOME", "NPM_CONFIG_CACHE"]
argv = ["npm", "run", "build", "--workspace=packages/cultcache-ts"]
[[steps]]
id = "build-cultnet-ts"
phase = "prepare"
runner = "node-build"
working_directory = "vendor/CultLib"
required_environment = ["HOME", "NPM_CONFIG_CACHE"]
argv = ["npm", "run", "build", "--workspace=packages/cultnet-ts"]
[[steps]]
id = "install-vili-dependencies"
phase = "prepare"
runner = "node-build"
required_environment = ["HOME", "NPM_CONFIG_CACHE"]
argv = ["npm", "ci", "--ignore-scripts", "--install-links"]
[[steps]]
id = "test-daemon"
phase = "test"
runner = "node-build"
required_environment = ["HOME", "NPM_CONFIG_CACHE"]
argv = ["node", "--test", "--test-timeout=60000", "scripts/test-vili-daemon.mjs"]
[[steps]]
id = "assemble-vili-release"
phase = "build"
runner = "node-build"
required_environment = ["HOME", "NPM_CONFIG_CACHE"]
argv = ["node", "deployment/idunn/assemble-release.mjs", "release/vili", "release/node"]
# ---- python-build: the interpreter and its locked packages
[[steps]]
id = "unpack-cpython"
phase = "prepare"
runner = "python-build"
argv = ["tar", "-xzf", "release/cpython.tar.gz", "-C", "release"]
[[steps]]
id = "install-build-tools"
phase = "prepare"
runner = "python-build"
required_environment = ["HOME", "PIP_CACHE_DIR", "TMPDIR", "PATH", "PIP_DISABLE_PIP_VERSION_CHECK", "PIP_NO_INPUT"]
argv = ["python3.12", "-m", "pip", "install", "--no-deps", "--require-hashes", "-r", "deployment/idunn/requirements-build.lock"]
[[steps]]
id = "install-locked-packages"
phase = "prepare"
runner = "python-build"
required_environment = ["HOME", "PIP_CACHE_DIR", "TMPDIR", "PATH", "PIP_DISABLE_PIP_VERSION_CHECK", "PIP_NO_INPUT"]
argv = ["python3.12", "-m", "pip", "install", "--no-deps", "--require-hashes", "--no-build-isolation", "-r", "deployment/idunn/requirements.lock"]
[[steps]]
id = "install-vendored-packages"
phase = "prepare"
runner = "python-build"
required_environment = ["HOME", "PIP_CACHE_DIR", "TMPDIR", "PATH", "PIP_DISABLE_PIP_VERSION_CHECK", "PIP_NO_INPUT"]
argv = ["python3.12", "-m", "pip", "install", "--no-deps", "--no-build-isolation", "--no-index", "vendor/SOMA-X", "vendor/kimodo/MotionCorrection", "vendor/kimodo"]
[[steps]]
id = "test-worker"
phase = "test"
runner = "python-build"
working_directory = "."
required_environment = ["HOME", "TMPDIR", "PYTHONPATH", "PATH"]
argv = ["python3.12", "-m", "unittest", "discover", "-s", "scripts", "-p", "test_*.py"]
# Every module the worker and the encoder child import at top level, kimodo.scripts.generate included.
[[steps]]
id = "import-smoke"
phase = "acceptance"
runner = "python-build"
required_environment = ["HOME", "TMPDIR", "PYTHONPATH", "PATH"]
argv = [
"python3.12", "-c",
"import accelerate, cultcache_py, huggingface_hub.constants, kimodo, kimodo.constraints, kimodo.exports.motion_io, kimodo.meta, kimodo.model.llm2vec.llm2vec, kimodo.model.load_model, kimodo.model.loading, kimodo.model.registry, kimodo.scripts.generate, kimodo.tools, msgpack, peft, torch, transformers",
]
[[steps]]
id = "compile-python"
phase = "build"
runner = "python-build"
required_environment = ["PATH"]
argv = ["python3.12", "-m", "compileall", "-q", "-j", "4", "--invalidation-mode", "unchecked-hash", "release/python/lib/python3.12"]
# ---- the sealed release: node, vili, python
[[artifacts]]
id = "node"
source_kind = "runner-output"
runner = "node-build"
source = "release/node/node"
destination = "node"
executable = true
[[artifacts]]
id = "vili"
source_kind = "runner-output"
runner = "node-build"
source = "release/vili"
destination = "vili"
[[artifacts]]
id = "python"
source_kind = "runner-output"
runner = "python-build"
source = "release/python"
destination = "python"
executable_members = ["bin/python3.12"]
[service]
executable_artifact = "node"
required_adjacent_artifacts = ["vili", "python"]
arguments = [
{ kind = "literal", value = "vili/scripts/idunn-entry.mjs" },
{ kind = "literal", value = "--state-root" },
{ kind = "binding", name = "state_root" },
{ kind = "literal", value = "--kimodo-commit" },
{ kind = "literal", value = "6bb58488037dd65360ff0c5d1692b403a23309f7" },
]
transport = "http"
route_required = true
required_environment = [
"GAMECULT_IDUNN_CANDIDATE_BIND",
"GAMECULT_IDUNN_PROCESS_WRITE_LEASE",
"GAMECULT_IDUNN_RUNTIME_BUNDLE",
"VILI_MODELS_ROOT",
"VILI_THREADS",
]
[service.health]
contract = "gamecult.runtime_presence_health.v2"
[state]
schema_generation = "vili-v1"
[[state.slots]]
id = "service-store"
relative_path = "vili.service.cc"
kind = "cultcache-file"
schema = "gamecult.vili.operator_state.v1"
writer = "process-bound-single-writer"
recovery = "rebuildable"
startup = "create-or-open-after-write-lease"
[[state.slots]]
id = "text-embeddings"
relative_path = "embeddings/text-embeddings.cc"
kind = "cultcache-file"
schema = "gamecult.vili.text_embedding.v0"
writer = "process-bound-single-writer"
recovery = "rebuildable"
startup = "create-or-open-after-write-lease"
[[state.slots]]
id = "jobs"
relative_path = "jobs"
kind = "external-directory"
schema = "gamecult.vili.motion_job.v0"
writer = "process-bound-single-writer"
recovery = "preserve"
startup = "create-or-open-after-write-lease"
[[state.slots]]
id = "artifacts"
relative_path = "artifacts"
kind = "external-directory"
schema = "gamecult.vili.motion_artifact.v0"
writer = "process-bound-single-writer"
recovery = "preserve"
startup = "create-or-open-after-write-lease"
[[state.slots]]
id = "logs"
relative_path = "logs"
kind = "external-directory"
schema = "text.log"
writer = "process-bound-single-writer"
recovery = "rebuildable"
startup = "create-or-open-after-write-lease"
[[provides]]
capability = "vili.motion.api"
schema = "http.v1"
compatibility = "v1"

View file

@ -0,0 +1,4 @@
setuptools
wheel
pybind11
cmake

View file

@ -0,0 +1,39 @@
# This file was autogenerated by uv via the following command:
# uv pip compile --generate-hashes --python-version 3.12 --python-platform x86_64-manylinux_2_28 requirements-build.in -o requirements-build.lock
cmake==4.4.4 \
--hash=sha256:02e9599cba932b386dc5c7e29db5dbda0610d13263798ff1521298c18014357d \
--hash=sha256:3c3f40c4656ec1ad1b873dd502bb31b3bba75295ae66b2623e515f2931eddbcb \
--hash=sha256:45a5c5e058e1e456f8da99e8bb83c98df079f385afd0bbea222a7eab6fdc1a8f \
--hash=sha256:6b914a7d9492d6134d43d484500cacd3a70466f6e31fcaf5145cf87128c28114 \
--hash=sha256:7b4a9c0d8146691d91b094633e4cd4f0cc4cdb8e04db0e8d805056db064396e0 \
--hash=sha256:883962a72b1b3a16ff445427f24c8caca9797dd9e563bbe208cbcd89e877f9d3 \
--hash=sha256:959fdcb277c63180928fe2c388063aa26dcfa1968033f7904eed1948ed44283b \
--hash=sha256:9ca4d839ced82f92b69319f50a65addf3051fae5d34464d9ac5b0296aba6ccb3 \
--hash=sha256:a3b4d4a35ba69360e630d2ce33020f42a61e09af28fe81122697989d11be8b39 \
--hash=sha256:ad8e0a38b5707e27882701146bdfffecacd80e7703fa0fc36c77528518d545af \
--hash=sha256:b3029f586853e01ddf2824c1ff124a1c5284f8ae897994758e9c4eeb45903b58 \
--hash=sha256:b4ed2c1a8f9c90e37a1f3095f7ed94979ebf3f94d0e90e63d50f4e72d419773a \
--hash=sha256:c7def2c08d9d0d31bab239209946e45a97520518ae63c0401772206f28b81523 \
--hash=sha256:d25e1d6baad137ddf21e576bf5296cd941bdd0da85baedbf78f673261fd2fca3 \
--hash=sha256:e2d98233f16b5d6b9d4673a0a724ce801fc413811befb8f0b008cbffb7a2ad63 \
--hash=sha256:eb0ff21b4309828f8fc0f0542022768f24b15aa1af3284e16c1eb01cdbe7d3e5 \
--hash=sha256:ed50c732a82ac04f6e5606a0677ddfb49dda9ee130f9d4163c5df217e478417e \
--hash=sha256:f337d18e33b116cabae9a7a8ff423fc0bc3f389da0acefeb724e1397128dcc34 \
--hash=sha256:f773a0544c66370408f451acd487c2c643b0dfebb48c5d64f3638bc1bb820238
# via -r requirements-build.in
packaging==26.3 \
--hash=sha256:94edc256424af38762eb31306eed28beb9f0efc50a8837492c9d6fd6004aed79 \
--hash=sha256:d7193f7c8e4e93f444fde0262bf90af30e16fa0ad0ad44cb553c87339b23cd1c
# via wheel
pybind11==3.1.0 \
--hash=sha256:a1cc06b524ab3edca51f8ad3895f9c4fa20b8b19283173dff4ae781449dc9639 \
--hash=sha256:b8488090f8acffbcb6b5d6a85571a6827a0a2981ffb75e5a0b27b87c4a6b7dd0
# via -r requirements-build.in
setuptools==84.2.1 \
--hash=sha256:3ff8b3b5492a0aa66960f4419365a86b84e013070d4313e4d4a6894f5d64d6bc \
--hash=sha256:ca140b45d5325a9478f880e351484f62fef15f336462e449ca0c03c873d589c2
# via -r requirements-build.in
wheel==0.48.0 \
--hash=sha256:3217dcc807155e45db462d7ef2431f5ddda0d7273b700d05a67b271ceb1287ab \
--hash=sha256:94800765601e9171bf5d58d066e640662842bcedcbab982b2c90787a2c987322
# via -r requirements-build.in

View file

@ -0,0 +1,31 @@
# Vili's CPU worker: torch from the CPU index, Kimodo 6bb58488's runtime pins as the CPU probe installed them
# (motion-pipeline map, Body facts; probe Dockerfile), msgpack for cultcache-py and the worker's imports.
# kimodo, MotionCorrection and SOMA-X install from the vendored gitlinks, never from here.
# Compile on Yggdrasil in a container (never on Starfire):
# uv pip compile --generate-hashes --python-version 3.12 --python-platform x86_64-manylinux_2_28 \
# --index-strategy unsafe-best-match -c requirements-build.lock requirements.in -o requirements.lock
--extra-index-url https://download.pytorch.org/whl/cpu
torch==2.5.1+cpu
msgpack
hydra-core==1.3.2
omegaconf==2.3.0
scipy==1.15.3
einops==0.8.2
tqdm
pydantic==2.12.5
trimesh==4.11.3
pillow
numpy==1.26.4
peft==0.18.1
transformers==5.1.0
accelerate==1.13.0
safetensors==0.7.0
huggingface-hub==1.6.0
tokenizers==0.22.2
psutil
packaging
filelock
boto3
bvhio
gradio_client>=1.0
urllib3>=2.6.3

File diff suppressed because it is too large Load diff

View file

@ -17,13 +17,16 @@ for Odin and Idunn. The request fields and the runtime flags are in `README.md`
## Witnesses
- `.vili/vili.service.cc`: the daemon-owned CultCache store for operator state,
- `.vili/vili.service.cc` (`/var/lib/gamecult/vili/vili.service.cc` under Idunn): the daemon-owned CultCache store for operator state,
the command boundary and the transport profile.
- `.vili/jobs/{jobId}.json`: accepted animation job records.
- `.vili/artifacts/{jobId}/`: generated motion outputs when generation succeeds.
- `.vili/embeddings/text-embeddings.cc`: the worker's text-embedding cache.
- `.vili/worker.pid`: the process group of the running worker, so a restarted
daemon can find and kill a worker its crashed predecessor left behind.
- `.vili/worker.pid`: the hand-run daemon's only: the process group of the running
worker, so a restarted hand-run daemon can find and kill a worker its crashed
predecessor left behind. Under Idunn it is never written; the unit's cgroup owns the tree.
- The hand-run daemon's abstract-socket lock (named by the state root's real path, no file) and Idunn's process write lease (Yggdrasil): the
writer authority for the state root. No state is written before the lease is held.
- `.vili/logs/worker.log`: the worker's stdout and stderr.
## Command Boundary

103
package-lock.json generated
View file

@ -8,7 +8,9 @@
"name": "@gamecult/vili",
"version": "0.1.0",
"dependencies": {
"cultcache-ts": "file:vendor/CultLib/packages/cultcache-ts"
"@gamecult/cultcache-ts": "file:vendor/CultLib/packages/cultcache-ts",
"@msgpack/msgpack": "3.1.3",
"cultnet-ts": "file:vendor/CultLib/packages/cultnet-ts"
}
},
"../CultLib/packages/cultcache-ts": {
@ -29,6 +31,19 @@
"node": ">=20"
}
},
"node_modules/@gamecult/cultcache-ts": {
"version": "0.14.2",
"resolved": "file:vendor/CultLib/packages/cultcache-ts",
"license": "MIT",
"dependencies": {
"@msgpack/msgpack": "^3.1.3",
"koffi": "3.3.0",
"zod": "^3.25.76"
},
"engines": {
"node": ">=20"
}
},
"node_modules/@koromix/koffi-android-arm64": {
"version": "3.3.0",
"resolved": "https://registry.npmjs.org/@koromix/koffi-android-arm64/-/koffi-android-arm64-3.3.0.tgz",
@ -358,19 +373,62 @@
"node": ">= 18"
}
},
"node_modules/@types/node": {
"version": "24.19.2",
"resolved": "https://registry.npmjs.org/@types/node/-/node-24.19.2.tgz",
"integrity": "sha512-93d49wCWJkCyN87MxLpNN6KXPCwFw7bVUg79mGUV1OS40JGWwVCr8n61DXCSqz7CV6/VfPkb7xec7uP2M+QHWQ==",
"dev": true,
"node_modules/ajv": {
"version": "8.20.0",
"resolved": "https://registry.npmjs.org/ajv/-/ajv-8.20.0.tgz",
"integrity": "sha512-Thbli+OlOj+iMPYFBVBfJ3OmCAnaSyNn4M1vz9T6Gka5Jt9ba/HIR56joy65tY6kx/FCF5VXNB819Y7/GUrBGA==",
"license": "MIT",
"dependencies": {
"undici-types": ">=7.24.0 <7.24.7"
"fast-deep-equal": "^3.1.3",
"fast-uri": "^3.0.1",
"json-schema-traverse": "^1.0.0",
"require-from-string": "^2.0.2"
},
"funding": {
"type": "github",
"url": "https://github.com/sponsors/epoberezkin"
}
},
"node_modules/cultcache-ts": {
"resolved": "vendor/CultLib/packages/cultcache-ts",
"link": true
"node_modules/cultnet-ts": {
"version": "0.3.0",
"resolved": "file:vendor/CultLib/packages/cultnet-ts",
"license": "MIT",
"dependencies": {
"@gamecult/cultcache-ts": "^0.14.1",
"@msgpack/msgpack": "^3.1.3",
"ajv": "^8.17.1"
},
"engines": {
"node": ">=20"
}
},
"node_modules/fast-deep-equal": {
"version": "3.1.3",
"resolved": "https://registry.npmjs.org/fast-deep-equal/-/fast-deep-equal-3.1.3.tgz",
"integrity": "sha512-f3qQ9oQy9j2AhBe/H9VC91wLmKBCCU/gDOnKNAYG5hswO7BLKj09Hc5HYNz9cGI++xlpDCIgDaitVs03ATR84Q==",
"license": "MIT"
},
"node_modules/fast-uri": {
"version": "3.1.9",
"resolved": "https://registry.npmjs.org/fast-uri/-/fast-uri-3.1.9.tgz",
"integrity": "sha512-hS/0Wt3Ya043oKXs0sIxrMpKQITXhMLIrVAKKXi4gozGC9v7YRWVnXIZzjNPTO02M9BdxLCWqK1yRXgNziVyJQ==",
"funding": [
{
"type": "github",
"url": "https://github.com/sponsors/fastify"
},
{
"type": "opencollective",
"url": "https://opencollective.com/fastify"
}
],
"license": "BSD-3-Clause"
},
"node_modules/json-schema-traverse": {
"version": "1.0.0",
"resolved": "https://registry.npmjs.org/json-schema-traverse/-/json-schema-traverse-1.0.0.tgz",
"integrity": "sha512-NM8/P9n3XjXhIZn1lLhkFaACTOURQXjWhV4BA/RnOv8xvgqtqpAX9IO4mRQxSx1Rlo4tqzeqb0sOlruaOy3dug==",
"license": "MIT"
},
"node_modules/koffi": {
"version": "3.3.0",
@ -404,27 +462,15 @@
"@koromix/koffi-win32-x64": "3.3.0"
}
},
"node_modules/typescript": {
"version": "5.9.3",
"resolved": "https://registry.npmjs.org/typescript/-/typescript-5.9.3.tgz",
"integrity": "sha512-jl1vZzPDinLr9eUt3J/t7V6FgNEw9QjvBPdysz9KfQDD41fQrC2Y4vKQdiaUpFT4bXlb1RHhLpp8wtm6M5TgSw==",
"dev": true,
"license": "Apache-2.0",
"bin": {
"tsc": "bin/tsc",
"tsserver": "bin/tsserver"
},
"node_modules/require-from-string": {
"version": "2.0.2",
"resolved": "https://registry.npmjs.org/require-from-string/-/require-from-string-2.0.2.tgz",
"integrity": "sha512-Xf0nWe6RseziFMu+Ap9biiUbmplq6S9/p+7w7YXP/JBHhrUDDUhwa+vANyubuqfZWTveU//DYVGsDG7RKL/vEw==",
"license": "MIT",
"engines": {
"node": ">=14.17"
"node": ">=0.10.0"
}
},
"node_modules/undici-types": {
"version": "7.24.6",
"resolved": "https://registry.npmjs.org/undici-types/-/undici-types-7.24.6.tgz",
"integrity": "sha512-WRNW+sJgj5OBN4/0JpHFqtqzhpbnV0GuB+OozA9gCL7a993SmU+1JBZCzLNxYsbMfIeDL+lTsphD5jN5N+n0zg==",
"dev": true,
"license": "MIT"
},
"node_modules/zod": {
"version": "3.25.76",
"resolved": "https://registry.npmjs.org/zod/-/zod-3.25.76.tgz",
@ -437,6 +483,7 @@
"vendor/CultLib/packages/cultcache-ts": {
"name": "@gamecult/cultcache-ts",
"version": "0.14.2",
"extraneous": true,
"license": "MIT",
"dependencies": {
"@msgpack/msgpack": "^3.1.3",

View file

@ -6,10 +6,14 @@
"description": "Vili is the GameCult Persona animation daemon: speech intent in, embodied motion affordance out.",
"scripts": {
"start": "node scripts/vili-daemon.mjs",
"health": "node scripts/vili-daemon.mjs --health",
"test": "node --test --test-timeout=60000 scripts/test-vili-daemon.mjs"
},
"dependencies": {
"cultcache-ts": "file:vendor/CultLib/packages/cultcache-ts"
"@gamecult/cultcache-ts": "file:vendor/CultLib/packages/cultcache-ts",
"@msgpack/msgpack": "3.1.3",
"cultnet-ts": "file:vendor/CultLib/packages/cultnet-ts"
},
"overrides": {
"@gamecult/cultcache-ts": "file:vendor/CultLib/packages/cultcache-ts"
}
}

View file

@ -0,0 +1,39 @@
// Test seam only: the shipped Idunn runtime (scripts/idunn-entry.mjs idunnRuntime) over a signer whose write lease is
// a file, so the lease gate is tested without Idunn's authority files. Present = leased. The daemon's own flags
// arrive on argv as they do for the hand-run daemon.
import { existsSync, appendFileSync, readdirSync } from "node:fs";
import path from "node:path";
import { idunnRuntime } from "../idunn-entry.mjs";
import { serve } from "../vili-daemon.mjs";
const leasePath = process.env.FAKE_IDUNN_LEASE;
const healthLog = process.env.FAKE_IDUNN_HEALTH_LOG;
const answer = process.env.FAKE_IDUNN_ANSWER_THROWS !== "1";
// The lease is gone from the moment a path exists under the state root (one "*" segment matches any entry), so a
// test can lose it between two writes of one request.
const stateRoot = process.env.FAKE_IDUNN_STATE_ROOT;
const loseWhenExists = process.env.FAKE_IDUNN_LOSE_WHEN_EXISTS;
function exists(parts, base) {
if (!parts.length) return true;
const [head, ...rest] = parts;
if (head !== "*") return existsSync(path.join(base, head)) && exists(rest, path.join(base, head));
return existsSync(base) && readdirSync(base).some((entry) => exists(rest, path.join(base, entry)));
}
const signer = {
assertWriteLease() {
if (!existsSync(leasePath)) throw new Error("no lease");
if (loseWhenExists && exists(loseWhenExists.split("/"), stateRoot)) throw new Error("lease lost");
return "lease";
},
answerRouteObservation(request) {
if (!answer) throw new Error("signer refused");
return { schemaId: "fake.presence", recordKey: request.messageId, payload: new Uint8Array([1, 2, 3]) };
},
reportHealth(state) {
appendFileSync(healthLog, `${state}\n`);
},
};
await serve(idunnRuntime(signer, { pollMs: 200 }));

View file

@ -1,12 +1,12 @@
// Stands in for kimodo-resident-worker.py in the daemon's tests. It binds port 0, prints the ready line the
// daemon waits for, and answers /health and /generate with what the daemon gave it. The --model value picks a
// misbehaviour: crash-before-ready, hang-before-ready, sick, slow-health, stubborn, fail-generate,
// refuse-generate, drop-generate, slow-generate, encoder (a grandchild, then no answer), graceful (exits 0.5 s after SIGTERM), die-after-ready. Any other value is just the model the worker serves.
// refuse-generate, drop-generate, slow-generate, writer (appends to <output>.writes every 50 ms, never answers, ignores SIGTERM), slow-to-stop (no answer; prints a line and exits 1.5 s after SIGTERM), chatty (prints a line the moment it is healthy), encoder (a grandchild, then no answer), graceful (exits 0.5 s after SIGTERM), die-after-ready. Any other value is just the model the worker serves.
import http from "node:http";
import os from "node:os";
import path from "node:path";
import { spawn } from "node:child_process";
import { appendFileSync } from "node:fs";
import { appendFileSync, writeFileSync } from "node:fs";
const argv = process.argv.slice(2);
const flag = (name) => {
@ -29,10 +29,16 @@ if (model === "crash-before-ready") process.exit(3);
if (model === "stubborn") {
process.on("SIGTERM", () => {});
const grandchild = spawn(process.execPath, ["-e", "setInterval(() => {}, 1000)"], { stdio: "ignore" });
appendFileSync(spawnLog, `grandchild ${grandchild.pid}\n`);
// the grandchild reports itself once its handler is armed, so a test never reads a pid that would die of the default SIGTERM
spawn(process.execPath, ["-e", `process.on('SIGTERM', () => {}); require('fs').appendFileSync(${JSON.stringify(spawnLog)}, 'grandchild ' + process.pid + '\\n'); setInterval(() => {}, 1000)`], { stdio: "ignore" });
}
if (model === "slow-to-stop") process.on("SIGTERM", () => {
console.log("stopping-line"); // output while it stops
setTimeout(() => process.exit(0), 1500);
});
if (model === "writer") process.on("SIGTERM", () => {}); // a worker that carries on writing through SIGTERM
if (model === "graceful") {
process.on("SIGTERM", () => {
setTimeout(() => {
@ -56,6 +62,8 @@ if (model === "hang-before-ready") {
if (request.method === "GET" && request.url === "/health") {
healthCalls += 1;
if (model === "sick" || model === "die-after-ready" || (model === "slow-health" && healthCalls <= 2)) return send(200, { ok: false });
writeFileSync(path.join(process.env.VILI_EMBEDDINGS_DIR, "stub-healthy"), ""); // a test can act the moment the worker is healthy
if (model === "chatty") console.log("after-healthy-line"); // after stub-healthy exists, which a test can make the loss of the lease
send(200, { ok: true, model: model || "stub-model", displayModel: model || "Stub Model", pid: process.pid, requests });
} else if (request.method === "POST" && request.url === "/generate") {
requests += 1;
@ -63,11 +71,14 @@ if (model === "hang-before-ready") {
if (model === "fail-generate") return send(500, { ok: false });
if (model === "refuse-generate") return send(200, { ok: false });
if (model === "drop-generate") return request.socket.destroy();
if (model === "slow-generate") return;
if (model === "slow-generate" || model === "slow-to-stop") return;
if (model === "writer") {
setInterval(() => appendFileSync(`${payload.output}.writes`, `${Date.now()}\n`), 50);
return;
}
if (model === "encoder") {
// an encoder child of the real worker: a grandchild in the worker's group, then a request that never answers
const encoder = spawn(process.execPath, ["-e", "process.on('SIGTERM', () => {}); setInterval(() => {}, 1000)"], { stdio: "ignore" });
appendFileSync(spawnLog, `grandchild ${encoder.pid}\n`);
spawn(process.execPath, ["-e", `process.on('SIGTERM', () => {}); require('fs').appendFileSync(${JSON.stringify(spawnLog)}, 'grandchild ' + process.pid + '\\n'); setInterval(() => {}, 1000)`], { stdio: "ignore" });
return;
}
send(200, {

145
scripts/idunn-entry.mjs Normal file
View file

@ -0,0 +1,145 @@
#!/usr/bin/env node
// The Idunn entry: the process Idunn's unit runs for target "vili". Idunn owns Vili's lifecycle and hands it one
// writer authority, the process write lease. This file turns Idunn's environment into the daemon's own
// configuration and into the runtime port serve() takes; it adds no routes and decides nothing the daemon decides.
//
// node vili/scripts/idunn-entry.mjs --state-root <absolute> --kimodo-commit <40 hex>
//
// Nothing from the bundle, the activation or the identity reaches the worker: the daemon's childEnv forwards only
// PATH, HOME and LANG besides its own VILI_* settings.
import { createHash } from "node:crypto";
import { readFileSync } from "node:fs";
import path from "node:path";
import { setTimeout as delay } from "node:timers/promises";
import { fileURLToPath, pathToFileURL } from "node:url";
import { decode, encode } from "@msgpack/msgpack";
import { encodeCultNetMessageForWire, parseCultNetMessage } from "cultnet-ts";
import { createIdunnRuntimeSigner, loadIdunnRuntimeAuthorityFromEnvironment } from "cultnet-ts/idunn-runtime";
const HEALTH_CONTRACT = "gamecult.runtime_presence_health.v2";
const LEASE_POLL_MS = 5000;
const here = path.dirname(fileURLToPath(import.meta.url));
// The runtime port serve() takes, over an Idunn runtime signer. Exported so the lease gate is tested against the
// code that ships, with a signer whose lease is a file (scripts/fixtures/fake-idunn-entry.mjs).
export function idunnRuntime(signer, { pollMs = LEASE_POLL_MS } = {}) {
let held = false;
let lost = false;
let onLost = () => {};
// The lease is read at the moment of every question, not once per tick: a loss is seen by the first gate to ask.
const check = () => {
if (!held) return false;
try {
signer.assertWriteLease();
return true;
} catch {
held = false;
lost = true;
onLost();
return false;
}
};
return {
async acquire() {
for (;;) {
try {
signer.assertWriteLease();
held = true;
return;
} catch {
// Idunn has not leased this incarnation yet
}
await delay(pollMs);
}
},
held: check,
refusal: () => (lost ? "write-lease-lost" : "write-lease-pending"),
watch(handler) {
onLost = handler;
check();
},
recordWorker() {},
forgetWorker() {},
answerSnapshot(bytes) {
let request;
try {
request = parseCultNetMessage(decode(bytes), "cultnet.schema.v0");
if (request.schemaVersion !== "cultnet.snapshot_request.v0") throw new Error("not a snapshot request");
} catch {
throw Object.assign(new Error("invalid snapshot request"), { code: "invalid-snapshot-request" });
}
const document = signer.answerRouteObservation(request);
const response = {
schemaVersion: "cultnet.snapshot_response_raw.v0",
messageId: request.messageId,
documents: [document],
};
return Buffer.from(encode(encodeCultNetMessageForWire(response, "cultnet.schema.v0")));
},
reportHealth: (state) => signer.reportHealth(state),
release() {},
};
}
function fail(reason) {
console.error(`vili idunn entry: ${reason}`);
process.exit(2);
}
// host:port, with an optional http:// scheme, as Idunn writes the candidate bind.
function splitBind(bind) {
const match = /^(?:http:\/\/)?([^:/\s]+):(\d{1,5})\/?$/u.exec(bind ?? "");
if (!match || Number(match[2]) > 65535) fail("GAMECULT_IDUNN_CANDIDATE_BIND is not host:port");
return { host: match[1], port: match[2] };
}
function parseEntryArgs(argv) {
if (argv.length !== 4 || argv[0] !== "--state-root" || argv[2] !== "--kimodo-commit") {
fail("arguments must be exactly --state-root <absolute> --kimodo-commit <40 hex>");
}
if (!path.isAbsolute(argv[1])) fail("--state-root must be absolute");
if (!/^[0-9a-f]{40}$/u.test(argv[3])) fail("--kimodo-commit must be 40 lowercase hex characters");
}
// What the daemon reads from its environment, derived from Idunn's bind and the sealed release's own layout:
// the interpreter sits beside the vili tree, and the environment's identity is the hash of the two lock files.
export function daemonEnvironment(bind, entryDir) {
const { host, port } = splitBind(bind);
const locks = ["requirements-build.lock", "requirements.lock"].map((name) =>
readFileSync(path.resolve(entryDir, "..", "deployment", "idunn", name)));
return {
// Idunn writes the bind as host:port; cultnet-ts compares it to the Expected route's endpoint, which has the scheme.
GAMECULT_IDUNN_CANDIDATE_BIND: `http://${host}:${port}`,
VILI_DAEMON_HOST: host,
VILI_DAEMON_PORT: port,
VILI_PYTHON: path.resolve(entryDir, "..", "..", "python", "bin", "python3.12"),
VILI_PYTHON_ENV: `locks-sha256:${createHash("sha256").update(Buffer.concat(locks)).digest("hex")}`,
};
}
async function main() {
parseEntryArgs(process.argv.slice(2));
Object.assign(process.env, daemonEnvironment(process.env.GAMECULT_IDUNN_CANDIDATE_BIND, here));
const authority = loadIdunnRuntimeAuthorityFromEnvironment("vili", HEALTH_CONTRACT);
const signer = createIdunnRuntimeSigner({
authority,
capabilities: [{ capability: "vili.motion.api", schema: "http.v1", compatibility: "v1", capacity: 1 }],
});
if (!signer.requiresWriteLease) fail("Idunn declares no write lease for this runtime");
const { serve } = await import("./vili-daemon.mjs");
await serve(idunnRuntime(signer));
}
if (process.argv[1] && import.meta.url === pathToFileURL(process.argv[1]).href) {
main().catch((error) => {
// The cause, from the failing code: its error code, or the fixed text of a plain Error. An fs error's message
// carries the path, so a coded error is named by its code alone.
const cause = error?.code ?? String(error?.message ?? "").slice(0, 200);
console.error(`vili idunn entry failed: ${error?.name || "Error"} ${cause}`);
process.exit(1);
});
}

View file

@ -3,6 +3,7 @@ import argparse
import json
import os
import shutil
import sys
import time
import traceback
from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer
@ -63,7 +64,7 @@ model, resolved_model = load_model(
text_encoder=text_encoder,
)
model_revision = text_encoding.read_snapshot_revision(HF_HOME, MODEL_NAMES[resolved_model])
worker_identity = {"kimodoCommit": kimodo_commit, "pythonEnv": python_env, "modelRevision": model_revision, "threads": threads}
worker_identity = {"kimodoCommit": kimodo_commit, "pythonEnv": python_env, "modelRevision": model_revision, "threads": threads, "python": sys.version}
model_load_seconds = time.time() - model_load_started_at
model_info = get_model_info(resolved_model)
display_model = model_info.display_name if model_info else resolved_model

View file

@ -40,6 +40,20 @@ def load_encoder(hf_home, mntp_id, supervised_id, dtype):
return LLM2Vec(model=model, tokenizer=tokenizer)
def encode_texts(model, texts):
"""Each text wrapped in the chat header and encoded alone: (vector bytes, seconds) lists."""
vectors, encode_seconds = [], []
for text in texts:
began = time.perf_counter()
features = model.tokenize([chat_header(text)])
with torch.no_grad():
vector = model.forward(features).detach().cpu().to(torch.float32)[0]
assert vector.shape == (EMBEDDING_DIM,)
vectors.append(vector.numpy().tobytes())
encode_seconds.append(time.perf_counter() - began)
return vectors, encode_seconds
def main() -> int:
apply_thread_cap(os.environ)
texts = json.load(sys.stdin)["texts"]
@ -53,15 +67,7 @@ def main() -> int:
p.requires_grad = False
load_seconds = time.perf_counter() - started
vectors, encode_seconds = [], []
for text in texts:
began = time.perf_counter()
features = model.tokenize([chat_header(text)])
with torch.no_grad():
vector = model.forward(features).detach().cpu().to(torch.float32)[0]
assert vector.shape == (EMBEDDING_DIM,)
vectors.append(vector.numpy().tobytes())
encode_seconds.append(time.perf_counter() - began)
vectors, encode_seconds = encode_texts(model, texts)
out.write_bytes(msgpack.packb({
"vectors": vectors,
@ -75,4 +81,5 @@ def main() -> int:
return 0
sys.exit(main())
if __name__ == "__main__":
sys.exit(main())

View file

@ -2,16 +2,19 @@
// Runs the real daemon against scripts/fixtures/stub-worker.mjs as its --python/--worker-entry.
import assert from "node:assert/strict";
import { spawn } from "node:child_process";
import { existsSync, mkdtempSync, readFileSync, rmSync, writeFileSync } from "node:fs";
import { existsSync, mkdirSync, mkdtempSync, readFileSync, readdirSync, rmSync, statSync, symlinkSync, writeFileSync } from "node:fs";
import http from "node:http";
import net from "node:net";
import os from "node:os";
import path from "node:path";
import test from "node:test";
import { fileURLToPath } from "node:url";
import { decode, encode } from "@msgpack/msgpack";
const here = path.dirname(fileURLToPath(import.meta.url));
const daemonPath = path.join(here, "vili-daemon.mjs");
const stubPath = path.join(here, "fixtures", "stub-worker.mjs");
const fakeIdunnEntry = path.join(here, "fixtures", "fake-idunn-entry.mjs");
const THREADS = 5;
async function until(condition, what, seconds = 15) {
@ -40,7 +43,7 @@ function isAlive(pid) {
async function startDaemon({
omit = [], viaEnv = false, threads = String(THREADS), python = process.execPath, nested = false, host = "127.0.0.1",
relativeModels = false, extraArgs = [], root: givenRoot = null,
relativeModels = false, extraArgs = [], root: givenRoot = null, entry = daemonPath, extraEnv = {}, hold = entry === daemonPath,
} = {}) {
const root = givenRoot ?? mkdtempSync(path.join(os.tmpdir(), "vili-test-"));
const stateRoot = nested ? path.join(root, "a", "b", "state") : root;
@ -57,9 +60,9 @@ async function startDaemon({
};
if (threads !== null) settings.threads = threads;
for (const name of omit) delete settings[name];
const env = { ...process.env, LEAK_CHECK: "leaked", LANG: "xx_TEST.UTF-8" };
const env = { ...process.env, LEAK_CHECK: "leaked", LANG: "xx_TEST.UTF-8", ...extraEnv };
for (const name of Object.keys(env)) if (name.startsWith("VILI_")) delete env[name];
const argv = [daemonPath, ...extraArgs];
const argv = [entry, ...extraArgs];
for (const [name, value] of Object.entries(settings)) {
if (viaEnv) env[`VILI_${{ host: "DAEMON_HOST", port: "DAEMON_PORT", "state-root": "STATE_ROOT", "worker-entry": "WORKER_ENTRY" }[name] ?? name.toUpperCase().replace("-", "_")}`] = value;
else argv.push(`--${name}`, value);
@ -71,25 +74,34 @@ async function startDaemon({
proc.stderr.on("data", (chunk) => { errors += chunk; });
const listening = await until(() => /listening on http:\/\/([^:]+):(\d+)/.exec(output), "the daemon to listen");
const port = listening[2];
// the hand-run daemon takes its lock after it listens; a test that wants a served state root waits for it
if (hold) await until(() => /holds the state root/.test(output), "the daemon to hold the state root");
const spawnLog = () => {
const log = path.join(stateRoot, "embeddings", "stub-spawns.log");
return existsSync(log) ? readFileSync(log, "utf8").split("\n").filter(Boolean) : [];
};
return {
root,
stateRoot,
modelsRoot,
proc,
boundHost: listening[1],
port: Number(port),
stderr: () => errors,
stdout: () => output,
spawns: () => spawnLog().filter((line) => /^\d+$/.test(line)).map(Number),
grandchildren: () => spawnLog().filter((line) => line.startsWith("grandchild ")).map((line) => Number(line.split(" ")[1])),
async stop() {
if (proc.exitCode === null && proc.signalCode === null) {
proc.kill("SIGTERM");
await until(() => proc.exitCode !== null || proc.signalCode !== null, "the daemon to exit", 30);
try {
if (proc.exitCode === null && proc.signalCode === null) {
proc.kill("SIGTERM");
await until(() => proc.exitCode !== null || proc.signalCode !== null, "the daemon to exit", 30);
}
} finally {
// a daemon that did not stop on SIGTERM must not outlive the test and hold the run open
if (proc.exitCode === null && proc.signalCode === null) proc.kill("SIGKILL");
rmSync(root, { recursive: true, force: true });
}
rmSync(root, { recursive: true, force: true });
},
};
}
@ -429,17 +441,41 @@ test("a live worker outside the pid file's group is left alone", async () => {
try {
assert.equal((await generate(first)).status, 200);
const [worker] = first.spawns();
first.proc.kill("SIGKILL");
await until(() => first.proc.exitCode !== null || first.proc.signalCode !== null, "the first daemon to die");
writeFileSync(path.join(root, "worker.pid"), `${bystander.pid}\n`);
const second = await startDaemon({ root });
assert.ok(isAlive(worker) && isAlive(bystander.pid), "only the group the pid file names, and only if it is ours, is killed");
second.proc.kill("SIGTERM");
await until(() => second.proc.exitCode !== null, "the second daemon to exit", 30);
process.kill(-worker, "SIGKILL");
} finally {
bystander.kill("SIGKILL");
await first.stop();
}
});
// The marker that proves a group is ours is this state root's own embeddings directory. A live worker of another
// state root, named by a stale pid file, carries another root's marker and is left alone.
test("a pid file naming another state root's live worker kills nothing", async () => {
const other = await startDaemon();
const root = mkdtempSync(path.join(os.tmpdir(), "vili-test-"));
try {
assert.equal((await generate(other)).status, 200);
const [foreign] = other.spawns();
writeFileSync(path.join(root, "worker.pid"), `${foreign}\n`);
const daemon = await startDaemon({ root });
try {
assert.ok(isAlive(foreign), "a worker carrying another root's marker must survive the reclaim");
assert.equal(existsSync(path.join(root, "worker.pid")), false, "the stale pid file is still forgotten");
} finally {
await daemon.stop();
}
} finally {
await other.stop();
}
});
test("a pid file naming a process group that is not the worker's kills nothing", async () => {
const root = mkdtempSync(path.join(os.tmpdir(), "vili-test-"));
const bystander = spawn(process.execPath, ["-e", "setInterval(() => {}, 1000)"], { detached: true, stdio: "ignore" });
@ -655,22 +691,6 @@ test("the typed service state is refreshed while the daemon runs", async () => {
}
});
test("--health asks a running daemon, and fails when there is none", async () => {
const daemon = await startDaemon();
const check = (port) => new Promise((resolve) => {
const proc = spawn(process.execPath, [daemonPath, "--health", "--host", "127.0.0.1", "--port", String(port)], { stdio: "ignore" });
proc.on("exit", resolve);
});
try {
assert.equal(await check(daemon.port), 0);
const gone = daemon.port;
await daemon.stop();
assert.equal(await check(gone), 1);
} finally {
await daemon.stop();
}
});
test("a thread count outside 1 to 16 stops the daemon without echoing it", async () => {
for (const threads of ["0", "17", "99", "100", "8x", "x8", "-1", "1.5"]) {
const proc = spawn(process.execPath, [daemonPath, "--threads", threads], { stdio: ["ignore", "pipe", "pipe"] });
@ -683,3 +703,589 @@ test("a thread count outside 1 to 16 stops the daemon without echoing it", async
assert.ok(!output.replace("from 1 to 16", "").includes(threads), `the refusal repeated ${threads}`);
}
});
const msgpackPost = (daemon, body, headers = {}) =>
new Promise((resolve, reject) => {
const req = http.request({ host: "127.0.0.1", port: daemon.port, method: "POST", path: "/cultnet/snapshot", headers: { "content-type": "application/msgpack", ...headers } }, (res) => {
const chunks = [];
res.on("data", (chunk) => chunks.push(chunk));
res.on("end", () => resolve({ status: res.statusCode, headers: res.headers, raw: Buffer.concat(chunks) }));
});
req.on("error", reject);
req.end(body);
});
const SECRET = "canary-9f3c-never-echoed";
const snapshotRequest = (messageId = "m1") => Buffer.from(encode({ schemaVersion: "cultnet.snapshot_request.v0", messageId }));
// A second daemon on a state root a live daemon holds must refuse, and must not touch the first one's worker.
test("a second daemon on a live state root exits 2 and kills nothing", async () => {
const first = await startDaemon();
let second = null;
try {
assert.equal((await generate(first)).status, 200);
const [worker] = first.spawns();
assert.ok(isAlive(worker));
second = await startDaemon({ root: first.root, hold: false });
await until(() => second.proc.exitCode !== null, "the second daemon to refuse", 20);
assert.equal(second.proc.exitCode, 2);
assert.match(second.stderr(), /held by a running daemon/);
assert.ok(isAlive(worker), "the first daemon's worker must survive the refusal");
assert.equal((await generate(first)).status, 200);
assert.equal(aliveWorkers(first).length, 1);
} finally {
if (second && second.proc.exitCode === null && second.proc.signalCode === null) second.proc.kill("SIGKILL"); // a daemon that failed to refuse must not outlive the test
await first.stop();
}
});
// Two daemons start together over the lock a crashed daemon left. Exactly one owns the root, every time: the
// other exits 2, and one generate leaves one worker on the root.
for (let trial = 1; trial <= 40; trial += 1) {
test(`two daemons racing a crashed daemon's lock leave one owner and one worker (trial ${trial})`, async () => {
const root = mkdtempSync(path.join(os.tmpdir(), "vili-test-"));
const crashed = await startDaemon({ root });
crashed.proc.kill("SIGKILL");
await until(() => crashed.proc.exitCode !== null || crashed.proc.signalCode !== null, "the first daemon to die");
const racers = await Promise.all([startDaemon({ root, hold: false }), startDaemon({ root, hold: false })]);
try {
await until(() => racers.every((racer) => racer.proc.exitCode !== null || /holds the state root/.test(racer.stdout())), "both racers to decide", 20);
await new Promise((resolve) => setTimeout(resolve, 300));
const owners = racers.filter((racer) => racer.proc.exitCode === null);
assert.equal(owners.length, 1, "exactly one daemon stays up");
assert.match(owners[0].stdout(), /holds the state root/);
const loser = racers.find((racer) => racer !== owners[0]);
assert.equal(loser.proc.exitCode, 2);
assert.match(loser.stderr(), /held by a running daemon/);
for (const racer of owners) assert.equal((await generate(racer)).status, 200);
assert.equal(aliveWorkers(owners[0]).length, 1, "one worker on the root");
} finally {
for (const racer of racers) if (racer.proc.exitCode === null && racer.proc.signalCode === null) racer.proc.kill("SIGKILL");
for (const pid of racers[0].spawns()) { try { process.kill(-pid, "SIGKILL"); } catch { /* gone */ } }
rmSync(root, { recursive: true, force: true });
}
});
}
// canonicalPath's fallback: a state root that does not exist yet is resolved through its longest existing prefix, so a
// daemon naming it through a symlink and one naming it directly still contend for one lock.
for (let trial = 1; trial <= 10; trial += 1) {
test(`two daemons naming a not-yet-existing root, one through a symlink, leave one owner (trial ${trial})`, async () => {
const real = mkdtempSync(path.join(os.tmpdir(), "vili-test-"));
const link = `${real}-link`;
symlinkSync(real, link);
const racers = await Promise.all([startDaemon({ root: real, nested: true, hold: false }), startDaemon({ root: link, nested: true, hold: false })]);
try {
await until(() => racers.every((racer) => racer.proc.exitCode !== null || /holds the state root/.test(racer.stdout())), "both racers to decide", 20);
await new Promise((resolve) => setTimeout(resolve, 300));
const owners = racers.filter((racer) => racer.proc.exitCode === null);
assert.equal(owners.length, 1, "exactly one daemon stays up");
const loser = racers.find((racer) => racer !== owners[0]);
assert.equal(loser.proc.exitCode, 2);
assert.match(loser.stderr(), /held by a running daemon/);
} finally {
for (const racer of racers) if (racer.proc.exitCode === null && racer.proc.signalCode === null) racer.proc.kill("SIGKILL");
rmSync(link, { force: true });
rmSync(real, { recursive: true, force: true });
}
});
}
const idunnDaemon = async ({ leased = false, extraEnv = {}, loseWhenExists = null } = {}) => {
const root = mkdtempSync(path.join(os.tmpdir(), "vili-test-"));
const lease = path.join(root, "lease");
const healthLog = path.join(root, "health.log");
if (leased) writeFileSync(lease, "");
const daemon = await startDaemon({
root, nested: true, entry: fakeIdunnEntry, extraEnv: {
FAKE_IDUNN_LEASE: lease, FAKE_IDUNN_HEALTH_LOG: healthLog, ...extraEnv,
...(loseWhenExists ? { FAKE_IDUNN_STATE_ROOT: path.join(root, "a", "b", "state"), FAKE_IDUNN_LOSE_WHEN_EXISTS: loseWhenExists } : {}),
},
});
return Object.assign(daemon, { lease, health: () => (existsSync(healthLog) ? readFileSync(healthLog, "utf8").split("\n").filter(Boolean) : []) });
};
test("before the lease nothing is written, spawned or accepted", async () => {
const daemon = await idunnDaemon();
try {
const answer = await generate(daemon);
assert.equal(answer.status, 503);
assert.equal(answer.body.code, "write-lease-pending");
assert.deepEqual(daemon.spawns(), []);
assert.equal(existsSync(daemon.stateRoot), false, "the state root must not exist before the lease");
assert.deepEqual(daemon.health(), []);
} finally {
await daemon.stop();
}
});
test("with the lease the daemon writes and serves, and no pid file exists", async () => {
const daemon = await idunnDaemon();
try {
writeFileSync(daemon.lease, "");
await until(() => existsSync(path.join(daemon.stateRoot, "vili.service.cc")), "the service store", 10);
const answer = await generate(daemon);
assert.equal(answer.status, 200);
assert.equal(existsSync(path.join(daemon.stateRoot, "worker.pid")), false);
assert.equal(existsSync(path.join(daemon.stateRoot, "daemon.lock")), false);
assert.deepEqual(daemon.health(), ["active"]);
} finally {
await daemon.stop();
}
});
test("losing the lease kills the worker tree, refuses, and writes nothing more", async () => {
const daemon = await idunnDaemon({ leased: true });
try {
await until(() => existsSync(path.join(daemon.stateRoot, "vili.service.cc")), "the service store", 10);
assert.equal((await generate(daemon, { prompt: "x", model: "stubborn" })).status, 200);
const [worker] = daemon.spawns();
const [grandchild] = await until(() => { const found = daemon.grandchildren(); return found.length ? found : null; }, "the armed grandchild");
assert.ok(isAlive(worker) && isAlive(grandchild));
rmSync(daemon.lease);
await until(() => !isAlive(worker) && !isAlive(grandchild), "the worker tree to die", 20);
const answer = await generate(daemon);
assert.equal(answer.status, 503);
assert.equal(answer.body.code, "write-lease-lost");
await until(() => daemon.health().join() === "active,degraded", "the health to say degraded");
const store = path.join(daemon.stateRoot, "vili.service.cc");
const before = statSync(store).mtimeMs;
const jobs = readdirSync(path.join(daemon.stateRoot, "jobs")).length;
await new Promise((resolve) => setTimeout(resolve, 6500));
assert.equal(statSync(store).mtimeMs, before);
assert.equal(readdirSync(path.join(daemon.stateRoot, "jobs")).length, jobs);
} finally {
await daemon.stop();
}
});
// A generate whose headers are in and whose body is still arriving when the lease goes. The lease is gone, so
// nothing is created and nothing is spawned when the body completes, whether the daemon has noticed the loss yet
// (its 5 s tick) or not.
for (const noticed of [false, true]) {
test(`a request in flight when the lease is lost writes and spawns nothing (loss ${noticed ? "noticed" : "not yet noticed"})`, async () => {
const daemon = await idunnDaemon({ leased: true });
try {
await until(() => existsSync(path.join(daemon.stateRoot, "vili.service.cc")), "the service store", 10);
const body = JSON.stringify({ prompt: "x", duration: 2 });
const socket = net.connect(daemon.port, "127.0.0.1");
let raw = "";
socket.on("data", (chunk) => { raw += chunk; });
const closed = new Promise((resolve) => socket.on("close", resolve));
socket.write(`POST /motion/generate HTTP/1.1\r\nhost: vili\r\ncontent-type: application/json\r\ncontent-length: ${body.length}\r\nconnection: close\r\n\r\n${body.slice(0, 5)}`);
await new Promise((resolve) => setTimeout(resolve, 200));
rmSync(daemon.lease);
if (noticed) await until(() => daemon.health().join() === "active,degraded", "the daemon to notice the loss", 15);
socket.write(body.slice(5));
await Promise.race([closed, new Promise((resolve) => setTimeout(resolve, 15000))]);
socket.destroy();
assert.match(raw, /^HTTP\/1\.1 503 /);
assert.match(raw, /write-lease-lost/);
assert.deepEqual(daemon.spawns(), [], "no worker is spawned after the loss");
const artifacts = path.join(daemon.stateRoot, "artifacts");
assert.deepEqual(existsSync(artifacts) ? readdirSync(artifacts) : [], [], "no artifact directory is created");
assert.deepEqual(readdirSync(path.join(daemon.stateRoot, "jobs")), []);
} finally {
await daemon.stop();
}
});
}
// The lease goes between two writes of one request. Each case names the path whose appearance takes the lease, and
// what must not exist afterwards: every write and the spawn are gated where they happen, not by the request's
// earlier checks.
const losses = [
{ when: "artifacts/*", absent: ["artifacts/*/meta.json", "logs", "embeddings"] },
{ when: "artifacts/*", payload: { prompt: "a person waves", duration: 2, constraints: [{ type: "root2d" }] }, absent: ["artifacts/*/constraints.json", "artifacts/*/meta.json"] },
{ when: "artifacts/*/meta.json", absent: ["logs", "embeddings"] },
{ when: "logs", absent: ["embeddings"] },
{ when: "embeddings", absent: ["logs/worker.log"] },
{ when: "embeddings/stub-healthy", absent: ["artifacts/*/request.json", "jobs/*"] },
];
for (const { when, absent, payload } of losses) {
test(`a request is stopped at its next write when the lease goes at ${when}${payload ? " (constraints in the body)" : ""}`, async () => {
const daemon = await idunnDaemon({ leased: true, loseWhenExists: when });
try {
await until(() => existsSync(path.join(daemon.stateRoot, "vili.service.cc")), "the service store", 10);
const answer = await generate(daemon, payload);
assert.equal(answer.status, 503);
assert.equal(answer.body.code, "write-lease-lost");
for (const gone of absent) {
const parts = gone.split("/");
const found = (base, rest) => (rest.length === 0 ? existsSync(base)
: rest[0] === "*" ? existsSync(base) && readdirSync(base).some((entry) => found(path.join(base, entry), rest.slice(1)))
: found(path.join(base, rest[0]), rest.slice(1)));
assert.equal(found(daemon.stateRoot, parts), false, `${gone} must not exist`);
}
if (when === "embeddings") assert.deepEqual(daemon.spawns(), []);
} finally {
await daemon.stop();
}
});
}
// The daemon's own copy of a worker's stdout into worker.log is a write like any other.
test("a worker's output line is not logged once the lease is gone", async () => {
const daemon = await idunnDaemon({ leased: true, loseWhenExists: "embeddings/stub-spawns.log" });
try {
await until(() => existsSync(path.join(daemon.stateRoot, "vili.service.cc")), "the service store", 10);
const answer = await generate(daemon);
assert.equal(answer.status, 503);
assert.equal(answer.body.code, "write-lease-lost");
const log = readFileSync(path.join(daemon.stateRoot, "logs", "worker.log"), "utf8");
assert.ok(!log.includes("stub-stdout-line"), "the daemon must not copy the worker's stdout after the loss");
} finally {
await daemon.stop();
}
});
// The lease goes while the first persist is under way: the daemon survives it, stays degraded, and the rest of the
// startup writes are not made.
test("a lease lost while the state root is opened leaves the daemon up, degraded and writing nothing more", async () => {
const daemon = await idunnDaemon({ leased: true, loseWhenExists: "vili.service.cc" });
try {
await until(() => daemon.health().includes("degraded") || daemon.proc.exitCode !== null, "the loss to be seen", 15);
assert.equal(daemon.proc.exitCode, null, "the daemon must not die of it");
assert.deepEqual(daemon.health(), ["degraded"]);
const store = readFileSync(path.join(daemon.stateRoot, "vili.service.cc"));
assert.ok(store.includes("operator_state") && !store.includes("command_boundary"), "the second document is not written");
const answer = await generate(daemon);
assert.equal(answer.status, 503);
assert.equal(answer.body.code, "write-lease-lost");
} finally {
await daemon.stop();
}
});
test("a request arriving right after the lease is lost writes and spawns nothing", async () => {
const daemon = await idunnDaemon({ leased: true });
try {
await until(() => existsSync(path.join(daemon.stateRoot, "vili.service.cc")), "the service store", 10);
rmSync(daemon.lease);
const answer = await generate(daemon);
assert.equal(answer.status, 503);
assert.equal(answer.body.code, "write-lease-lost");
assert.deepEqual(daemon.spawns(), []);
const artifacts = path.join(daemon.stateRoot, "artifacts");
assert.deepEqual(existsSync(artifacts) ? readdirSync(artifacts) : [], []);
assert.deepEqual(readdirSync(path.join(daemon.stateRoot, "jobs")), []);
} finally {
await daemon.stop();
}
});
test("a worker respawned under the lease is killed when the lease is lost", async () => {
const daemon = await idunnDaemon({ leased: true });
try {
await until(() => existsSync(path.join(daemon.stateRoot, "vili.service.cc")), "the service store", 10);
assert.equal((await generate(daemon)).status, 200);
const [first] = daemon.spawns();
process.kill(-first, "SIGKILL");
await until(async () => (await backendOf(daemon)).state === "stopped", "the daemon to see the worker gone");
assert.equal((await generate(daemon)).status, 200);
const second = daemon.spawns().find((pid) => pid !== first);
assert.ok(second && isAlive(second));
rmSync(daemon.lease);
await until(() => !isAlive(second), "the respawned worker to die", 20);
} finally {
await daemon.stop();
}
});
// Idunn stops an incumbent (SIGTERM to the unit) before it revokes the lease. What Vili owns in that sequence: the
// worker tree is dead when the process ends, it exits 0, and nothing is written from the signal on.
test("on SIGTERM the daemon stops its worker tree, writes nothing more, and exits 0", async () => {
const daemon = await idunnDaemon({ leased: true });
try {
await until(() => existsSync(path.join(daemon.stateRoot, "vili.service.cc")), "the service store", 10);
assert.equal((await generate(daemon, { prompt: "x", model: "stubborn" })).status, 200);
const [worker] = daemon.spawns();
const [grandchild] = await until(() => { const found = daemon.grandchildren(); return found.length ? found : null; }, "the armed grandchild");
const store = path.join(daemon.stateRoot, "vili.service.cc");
const before = statSync(store).mtimeMs;
const jobs = readdirSync(path.join(daemon.stateRoot, "jobs"));
daemon.proc.kill("SIGTERM");
await until(() => daemon.proc.exitCode !== null || daemon.proc.signalCode !== null, "the daemon to exit", 30);
assert.equal(daemon.proc.exitCode, 0);
assert.ok(!isAlive(worker) && !isAlive(grandchild), "the worker tree is dead when the daemon is gone");
assert.equal(statSync(store).mtimeMs, before, "the 5 s surface tick must not write during the stop");
assert.deepEqual(readdirSync(path.join(daemon.stateRoot, "jobs")), jobs);
assert.ok(existsSync(daemon.lease), "the daemon never touches the lease Idunn revokes");
} finally {
await daemon.stop();
}
});
test("a job that finishes after the lease is lost is not recorded", async () => {
const daemon = await idunnDaemon({ leased: true });
try {
await until(() => existsSync(path.join(daemon.stateRoot, "vili.service.cc")), "the service store", 10);
const slow = generate(daemon, { prompt: "x", model: "slow-generate" });
slow.catch(() => {}); // a failing assertion below must not leave this request pending
await until(() => daemon.spawns().length, "the worker");
await until(() => request(daemon.port, "GET", "/health").then((r) => r.body?.state?.backend?.state === "ready"), "the worker to be ready");
rmSync(daemon.lease);
const answer = await Promise.race([slow, new Promise((resolve) => setTimeout(() => resolve({ status: "no answer in 30 s" }), 30000))]);
assert.equal(answer.status, 503);
assert.equal(answer.body.code, "write-lease-lost");
assert.deepEqual(readdirSync(path.join(daemon.stateRoot, "jobs")), []);
} finally {
await daemon.stop();
}
});
test("the route challenge is answered ahead of the cross-site refusal, and an Origin is still refused", async () => {
const daemon = await idunnDaemon();
try {
// the challenge is how Idunn learns the incarnation is warming, so it is answered before the lease
const answered = await msgpackPost(daemon, snapshotRequest("m-42"));
assert.equal(answered.status, 200);
assert.equal(answered.headers["content-type"], "application/msgpack");
const message = decode(answered.raw);
assert.equal(message.schemaVersion, "cultnet.snapshot_response_raw.v0");
assert.equal(message.messageId, "m-42");
assert.equal(message.documents.length, 1);
assert.equal((await msgpackPost(daemon, snapshotRequest(), { origin: "https://evil.example" })).status, 403);
assert.equal((await msgpackPost(daemon, Buffer.from("not msgpack at all"))).status, 400);
assert.equal((await msgpackPost(daemon, Buffer.from(encode({ schemaVersion: "cultnet.snapshot_response_raw.v0", messageId: "x", documents: [] })))).status, 400);
assert.equal((await msgpackPost(daemon, Buffer.alloc(70 * 1024))).status, 400);
const text = await request(daemon.port, "POST", "/motion/generate", { body: "{}", headers: { "content-type": "text/plain" } });
assert.equal(text.status, 403);
} finally {
await daemon.stop();
}
});
test("a signer that cannot answer gives 503 and echoes nothing", async () => {
const daemon = await idunnDaemon({ extraEnv: { FAKE_IDUNN_ANSWER_THROWS: "1" } });
try {
const answered = await msgpackPost(daemon, snapshotRequest(SECRET));
assert.equal(answered.status, 503);
assert.ok(!answered.raw.toString("utf8").includes(SECRET));
assert.ok(!daemon.stderr().includes(SECRET));
} finally {
await daemon.stop();
}
});
test("the hand-run daemon has no route challenge", async () => {
const daemon = await startDaemon();
try {
assert.equal((await msgpackPost(daemon, snapshotRequest())).status, 404);
assert.equal((await msgpackPost(daemon, snapshotRequest(), { origin: "https://evil.example" })).status, 404);
} finally {
await daemon.stop();
}
});
// ---- the Idunn entry's own rules: its argument contract, and what it derives for the daemon
const entryPath = path.join(here, "idunn-entry.mjs");
const runEntry = (args, env = {}) => new Promise((resolve) => {
const proc = spawn(process.execPath, [entryPath, ...args], { env: { PATH: process.env.PATH, ...env }, stdio: ["ignore", "pipe", "pipe"] });
let output = "";
proc.stdout.on("data", (chunk) => { output += chunk; });
proc.stderr.on("data", (chunk) => { output += chunk; });
proc.on("exit", (code) => resolve({ code, output }));
});
test("the entry takes exactly --state-root <absolute> --kimodo-commit <40 hex>, and echoes none of a bad value", async () => {
const commit = "6bb58488037dd65360ff0c5d1692b403a23309f7";
const good = ["--state-root", "/var/lib/gamecult/vili", "--kimodo-commit", commit];
const bads = [
[],
[...good, "--extra", "x"],
["--kimodo-commit", commit, "--state-root", "/var/lib/gamecult/vili"],
["--state-root", "canary-relative-path", "--kimodo-commit", commit],
["--state-root", "/var/lib/gamecult/vili", "--kimodo-commit", "canary-not-a-commit"],
["--state-root", "/var/lib/gamecult/vili", "--kimodo-commit", commit.toUpperCase()],
];
for (const args of bads) {
const answer = await runEntry(args, { GAMECULT_IDUNN_CANDIDATE_BIND: "127.0.0.1:18861" });
assert.equal(answer.code, 2, JSON.stringify(args));
assert.ok(!answer.output.includes("canary"), `the refusal repeated a value of ${JSON.stringify(args)}`);
}
for (const badBind of ["canary-not-a-bind", "127.0.0.1:65536", "127.0.0.1:99999", "127.0.0.1:123456", "127.0.0.1:"]) {
const bind = await runEntry(good, { GAMECULT_IDUNN_CANDIDATE_BIND: badBind });
assert.equal(bind.code, 2, badBind);
assert.match(bind.output, /CANDIDATE_BIND is not host:port/);
assert.ok(!bind.output.includes("canary"));
assert.ok(!bind.output.includes("65536") && !bind.output.includes("99999"));
}
});
// A startup failure that kills the unit names its cause from what the failing code itself says (its error code, or
// the fixed text of a plain Error), and never repeats a value: an fs error's own message carries the path.
test("a failed startup names the cause and repeats no value", async () => {
const good = ["--state-root", "/var/lib/gamecult/vili", "--kimodo-commit", "6bb58488037dd65360ff0c5d1692b403a23309f7"];
const bind = { GAMECULT_IDUNN_CANDIDATE_BIND: "127.0.0.1:18861" };
const unset = await runEntry(good, bind);
assert.equal(unset.code, 1);
assert.match(unset.output, /GAMECULT_IDUNN_RUNTIME_BUNDLE is required/);
const missing = await runEntry(good, { ...bind, GAMECULT_IDUNN_RUNTIME_BUNDLE: "/canary-bundle-dir" });
assert.equal(missing.code, 1);
assert.match(missing.output, /ENOENT/);
assert.ok(!missing.output.includes("canary"), "the failure repeated a value");
});
test("the entry derives the daemon's host, port, interpreter and environment identity from the release", async () => {
const { createHash } = await import("node:crypto");
const { daemonEnvironment } = await import("./idunn-entry.mjs");
const release = mkdtempSync(path.join(os.tmpdir(), "vili-entry-"));
try {
mkdirSync(path.join(release, "vili", "scripts"), { recursive: true });
mkdirSync(path.join(release, "vili", "deployment", "idunn"), { recursive: true });
writeFileSync(path.join(release, "vili", "deployment", "idunn", "requirements-build.lock"), "build lock bytes\n");
writeFileSync(path.join(release, "vili", "deployment", "idunn", "requirements.lock"), "runtime lock bytes\n");
const entryDir = path.join(release, "vili", "scripts");
assert.equal(daemonEnvironment("127.0.0.1:65535", entryDir).VILI_DAEMON_PORT, "65535", "the top of the port range is accepted");
for (const bind of ["127.0.0.1:18861", "http://127.0.0.1:18861"]) {
const derived = daemonEnvironment(bind, entryDir);
assert.equal(derived.VILI_DAEMON_HOST, "127.0.0.1");
assert.equal(derived.VILI_DAEMON_PORT, "18861");
assert.equal(derived.GAMECULT_IDUNN_CANDIDATE_BIND, "http://127.0.0.1:18861", "cultnet-ts compares the bind to the Expected route's endpoint, which carries the scheme Idunn's bare host:port lacks");
assert.equal(derived.VILI_PYTHON, path.join(release, "python", "bin", "python3.12"));
assert.equal(derived.VILI_PYTHON_ENV, `locks-sha256:${createHash("sha256").update("build lock bytes\nruntime lock bytes\n").digest("hex")}`);
}
} finally {
rmSync(release, { recursive: true, force: true });
}
});
// The lease is asked while a generate awaits its worker, not only when the service documents are rewritten. The
// bound is one WATCH_MS ask plus the SIGKILL; the test allows a second. The loss is made just after a rewrite of the
// service documents (whose own gates have asked), the longest wait for an implementation that asks only then.
test("a worker still writing when the lease is lost is dead within a second", async () => {
const daemon = await idunnDaemon({ leased: true });
try {
const store = path.join(daemon.stateRoot, "vili.service.cc");
await until(() => existsSync(store), "the service store", 10);
const pending = generate(daemon, { prompt: "x", model: "writer" });
pending.catch(() => {});
const writes = () => {
const dir = path.join(daemon.stateRoot, "artifacts");
for (const entry of existsSync(dir) ? readdirSync(dir) : []) {
const file = path.join(dir, entry, "motion.writes");
if (existsSync(file)) return file;
}
return null;
};
const file = await until(writes, "the worker to write");
const [worker] = daemon.spawns();
const rewritten = statSync(store).mtimeMs;
await until(() => statSync(store).mtimeMs !== rewritten, "the service documents to be rewritten", 8);
await new Promise((resolve) => setTimeout(resolve, 150)); // the rewrite is done: its own gates have asked
const lostAt = Date.now();
rmSync(daemon.lease);
await until(() => !isAlive(worker), "the worker to die", 10);
assert.ok(Date.now() - lostAt < 1000, `the worker lived ${Date.now() - lostAt} ms after the loss`);
const stamps = readFileSync(file, "utf8").trim().split("\n").map(Number);
assert.ok(Math.max(...stamps) - lostAt < 1000, "the worker wrote more than a second after the loss");
const answer = await Promise.race([pending, new Promise((resolve) => setTimeout(() => resolve({ status: "no answer" }), 10000))]);
assert.equal(answer.status, 503);
assert.equal(answer.body.code, "write-lease-lost");
assert.deepEqual(readdirSync(path.join(daemon.stateRoot, "jobs")), []);
} finally {
await daemon.stop();
}
});
// A request in flight and one arriving during the stop. The worker takes 1.5 s to stop, so the stop is long enough
// for the second request to arrive in it. From the signal on nothing is written and nothing is spawned.
for (const kind of ["fake Idunn entry", "hand-run"]) {
test(`SIGTERM (${kind}): nothing is written or spawned from the signal on, and the worker tree is dead at exit 0`, async () => {
const daemon = kind === "hand-run" ? await startDaemon() : await idunnDaemon({ leased: true });
try {
await until(() => existsSync(path.join(daemon.stateRoot, "vili.service.cc")), "the service store", 10);
const inFlight = generate(daemon, { prompt: "x", model: "slow-to-stop" });
inFlight.catch(() => {});
await until(() => daemon.spawns().length, "the worker");
await until(() => request(daemon.port, "GET", "/health").then((r) => r.body?.state?.backend?.state === "ready"), "the worker to be ready");
const store = path.join(daemon.stateRoot, "vili.service.cc");
const storeBefore = statSync(store).mtimeMs;
const artifacts = readdirSync(path.join(daemon.stateRoot, "artifacts"));
daemon.proc.kill("SIGTERM");
await new Promise((resolve) => setTimeout(resolve, 300));
const second = await generate(daemon).catch((error) => ({ status: error.code }));
assert.notEqual(second.status, 200, "a request arriving during the stop is not served");
await until(() => daemon.proc.exitCode !== null || daemon.proc.signalCode !== null, "the daemon to exit", 30);
assert.equal(daemon.proc.exitCode, 0);
assert.equal(daemon.spawns().length, 1, "no worker is spawned after the signal");
assert.ok(daemon.spawns().every((pid) => !isAlive(pid)), "the worker tree is dead at exit");
assert.deepEqual(readdirSync(path.join(daemon.stateRoot, "artifacts")), artifacts, "no artifact is created after the signal");
assert.deepEqual(readdirSync(path.join(daemon.stateRoot, "jobs")), [], "no job is recorded after the signal");
assert.equal(statSync(store).mtimeMs, storeBefore);
assert.ok(!readFileSync(path.join(daemon.stateRoot, "logs", "worker.log"), "utf8").includes("stopping-line"), "the worker's output while it stops is not copied");
} finally {
await daemon.stop();
}
});
}
// One notion of root identity: a root is the same root under either spelling, for the lock and for reclaim.
for (const [crashedOn, restartedOn] of [["link", "real"], ["real", "link"]]) {
test(`a daemon restarted on the ${restartedOn} spelling of its root reclaims the worker a crash on the ${crashedOn} spelling left`, async () => {
const real = mkdtempSync(path.join(os.tmpdir(), "vili-test-"));
const link = `${real}-link`;
symlinkSync(real, link);
const spelling = { real, link };
const first = await startDaemon({ root: spelling[crashedOn] });
let second = null;
let stray = null;
try {
assert.equal((await generate(first)).status, 200);
[stray] = first.spawns();
first.proc.kill("SIGKILL");
await until(() => first.proc.exitCode !== null || first.proc.signalCode !== null, "the first daemon to die");
assert.ok(isAlive(stray), "the probe must reproduce the orphan before the fix can be measured");
second = await startDaemon({ root: spelling[restartedOn] });
await until(() => !isAlive(stray), "the predecessor's worker to be reclaimed");
assert.equal((await generate(second)).status, 200);
assert.equal(aliveWorkers(second).length, 1, "one live worker on the one root");
} finally {
if (stray) { try { process.kill(-stray, "SIGKILL"); } catch { /* gone */ } }
if (second) await second.stop();
rmSync(link, { force: true });
rmSync(real, { recursive: true, force: true });
}
});
}
// The group is the worker's, and its leader may be the first to go. Any member carrying the root's marker proves it.
test("a restarted daemon kills a worker group whose leader is gone and a member lives", async () => {
const root = mkdtempSync(path.join(os.tmpdir(), "vili-test-"));
const first = await startDaemon({ root });
const stray = generate(first, { prompt: "x", model: "encoder" }).catch(() => null);
const [grandchild] = await until(() => { const found = first.grandchildren(); return found.length ? found : null; }, "the encoder child");
const [worker] = first.spawns();
first.proc.kill("SIGKILL");
await until(() => first.proc.exitCode !== null || first.proc.signalCode !== null, "the first daemon to die");
process.kill(worker, "SIGKILL"); // the leader only
await until(() => !isAlive(worker), "the leader to die");
await stray;
assert.ok(isAlive(grandchild), "the probe must leave a member alive in the leaderless group");
const second = await startDaemon({ root });
try {
await until(() => !isAlive(grandchild), "the leaderless group's member to be killed");
assert.equal(existsSync(path.join(root, "worker.pid")), false);
} finally {
try { process.kill(-worker, "SIGKILL"); } catch { /* gone */ }
await second.stop();
}
});
// The copy of worker output is gated at every line, not at the first: here the lease goes after the first line
// was logged, by the worker's first healthy answer, and the line it prints then must not reach the log.
test("a worker's output line after the first is not logged once the lease is gone", async () => {
const daemon = await idunnDaemon({ leased: true, loseWhenExists: "embeddings/stub-healthy" });
try {
await until(() => existsSync(path.join(daemon.stateRoot, "vili.service.cc")), "the service store", 10);
const answer = await generate(daemon, { prompt: "x", model: "chatty" });
assert.equal(answer.status, 503);
assert.equal(answer.body.code, "write-lease-lost");
await new Promise((resolve) => setTimeout(resolve, 500));
const log = readFileSync(path.join(daemon.stateRoot, "logs", "worker.log"), "utf8");
assert.ok(log.includes("stub-stdout-line"), "the first line was logged under the lease");
assert.ok(!log.includes("after-healthy-line"), "a line after the loss must not be copied");
} finally {
await daemon.stop();
}
});

View file

@ -55,6 +55,9 @@ class Identity(unittest.TestCase):
self.assertNotEqual(key, te.embedding_key("id", "text b"))
self.assertNotEqual(te.embedding_key("id", "a" * 300 + "x"), te.embedding_key("id", "a" * 300 + "y"))
def test_key_is_case_sensitive(self):
self.assertNotEqual(te.embedding_key("id", "A person waves."), te.embedding_key("id", "a person waves."))
def test_chat_header_is_the_llama3_user_turn_pooled_after_the_separator(self):
header = "<|start_header_id|>user<|end_header_id|>" + chr(10) * 2
self.assertEqual(te.chat_header("hello "), header + "!@#$%^&*()hello<|eot_id|>")
@ -269,6 +272,68 @@ class ChildBoundary(unittest.TestCase):
self.assertEqual(len(answer["vectors"]), 2)
class ChildHeader(unittest.TestCase):
def test_every_text_is_tokenized_with_the_chat_header(self):
import importlib.util
import torch
spec = importlib.util.spec_from_file_location("kimodo_text_encode", Path(te.__file__).with_name("kimodo-text-encode.py"))
child = importlib.util.module_from_spec(spec)
spec.loader.exec_module(child)
class Recorder:
def __init__(self):
self.seen = []
def tokenize(self, batch):
self.seen.extend(batch)
return batch
def forward(self, features):
return torch.zeros(1, te.EMBEDDING_DIM)
model = Recorder()
vectors, seconds = child.encode_texts(model, ["a person waves", " padded "])
self.assertEqual(model.seen, [te.chat_header("a person waves"), te.chat_header(" padded ")])
for header in model.seen:
self.assertTrue(header.startswith("<|start_header_id|>user<|end_header_id|>" + chr(10) * 2))
self.assertEqual([len(v) for v in vectors], [te.EMBEDDING_DIM * 4] * 2)
self.assertEqual(len(seconds), 2)
def test_a_vector_of_the_wrong_width_is_refused(self):
import importlib.util
import torch
spec = importlib.util.spec_from_file_location("kimodo_text_encode", Path(te.__file__).with_name("kimodo-text-encode.py"))
child = importlib.util.module_from_spec(spec)
spec.loader.exec_module(child)
class Narrow:
def tokenize(self, batch):
return batch
def forward(self, features):
return torch.zeros(1, te.EMBEDDING_DIM - 1)
with self.assertRaises(AssertionError):
child.encode_texts(Narrow(), ["a person waves"])
# sha256 of the two Kimodo sources the encoder mirrors, at vendor/kimodo 6bb58488 (embedding-identity-complete).
# A Kimodo bump that changes either file must re-read the mirrored loader before these move.
KIMODO_LOADER_SHA256 = {
"model/load_model.py": "008092cd0b7adef1faea9fa9075718a61bea8ca9feaf9acf1b1d6caacaa0380a",
"model/llm2vec/llm2vec.py": "dc6e8b7d38086b4f456e013ab22cd4cc2cd72eb7fd2c13069f037f42aec80857",
}
class KimodoPin(unittest.TestCase):
def test_the_mirrored_loader_sources_are_the_pinned_ones(self):
import hashlib
import kimodo
root = Path(kimodo.__file__).resolve().parent
for relative, expected in KIMODO_LOADER_SHA256.items():
self.assertEqual(hashlib.sha256((root / relative).read_bytes()).hexdigest(), expected, relative)
REFERENCE_MODELS = os.environ.get("VILI_REFERENCE_MODELS")
ROCK = "A person shakes their fist up and down three times, then thrusts a closed fist forward like a rock."
REFERENCE_CODE = """

View file

@ -1,11 +1,13 @@
#!/usr/bin/env node
import http from "node:http";
import os from "node:os";
import { closeSync, openSync, readdirSync, readFileSync, unlinkSync, writeFileSync, writeSync } from "node:fs";
import { createHash } from "node:crypto";
import { closeSync, openSync, readdirSync, readFileSync, realpathSync, unlinkSync, writeFileSync, writeSync } from "node:fs";
import { mkdir, writeFile } from "node:fs/promises";
import path from "node:path";
import readline from "node:readline";
import { fileURLToPath } from "node:url";
import net from "node:net";
import { fileURLToPath, pathToFileURL } from "node:url";
import { createRequire } from "node:module";
import { spawn } from "node:child_process";
@ -16,7 +18,7 @@ const {
SingleFileMessagePackBackingStore,
defineDocumentRegistry,
defineDocumentType,
} = require("cultcache-ts");
} = require("@gamecult/cultcache-ts");
const projectRoot = path.resolve(__dirname, "..");
const providerId = "vili.animation";
const viliStoreFileName = "vili.service.cc";
@ -73,6 +75,17 @@ const viliDocumentRegistry = defineDocumentRegistry(
const MAX_THREADS = 16;
// The state root's one identity: symlinks resolved as far as the path exists. The hand-run lock's name, reclaim's
// marker and the worker's environment all derive from args.stateRoot, so two spellings of one root are one root.
function canonicalPath(target) {
try {
return realpathSync(target);
} catch {
const parent = path.dirname(target);
return parent === target ? target : path.join(canonicalPath(parent), path.basename(target));
}
}
function parseArgs(argv) {
const env = process.env;
const args = {
@ -85,7 +98,6 @@ function parseArgs(argv) {
threads: env.VILI_THREADS || "8",
kimodoCommit: env.VILI_KIMODO_COMMIT || "",
pythonEnv: env.VILI_PYTHON_ENV || "",
health: false,
};
for (let index = 0; index < argv.length; index += 1) {
@ -99,13 +111,13 @@ function parseArgs(argv) {
else if (arg === "--threads") args.threads = argv[++index];
else if (arg === "--kimodo-commit") args.kimodoCommit = argv[++index];
else if (arg === "--python-env") args.pythonEnv = argv[++index];
else if (arg === "--health") args.health = true;
}
if (!/^[0-9]{1,2}$/.test(String(args.threads)) || Number(args.threads) < 1 || Number(args.threads) > MAX_THREADS) {
console.error(`threads must be an integer from 1 to ${MAX_THREADS}`);
process.exit(2);
}
args.threads = Number(args.threads);
args.stateRoot = canonicalPath(args.stateRoot);
if (args.modelsRoot) args.modelsRoot = path.resolve(args.modelsRoot);
return args;
}
@ -245,13 +257,17 @@ function transportProfile() {
}
async function persistSurfaces() {
guard();
await mkdir(args.stateRoot, { recursive: true });
const cache = CultCache.builder()
.withRegistry(viliDocumentRegistry)
.withGenericStore(new SingleFileMessagePackBackingStore(path.join(args.stateRoot, viliStoreFileName)))
.build();
guard();
await cache.put(operatorStateDefinition, "vili", operatorState());
guard();
await cache.put(commandBoundaryDefinition, "vili", commandBoundary());
guard();
await cache.put(transportProfileDefinition, "vili", transportProfile());
}
@ -294,11 +310,9 @@ function waitForExit(proc, timeoutMs) {
}
// The worker leads its own process group (its pid is the group id), so the encoder child it spawns is in the group
// too. The daemon owns the whole group: it is killed when the worker exits for any reason, on a requested stop, and
// at the next start if a crashed daemon left it behind (reclaimStrayWorker). Under systemd, KillMode=control-group
// is the same ownership from outside the process.
const workerPidFile = () => path.join(args.stateRoot, "worker.pid");
// too. The daemon owns the whole group: it is killed when the worker exits for any reason and on a requested stop.
// A crashed hand-run daemon's group is reclaimed by the next hand-run daemon (handRunRuntime). Under Idunn the
// unit's cgroup is the same ownership from outside the process.
function killGroup(pgid, signal) {
try {
process.kill(-pgid, signal);
@ -307,61 +321,109 @@ function killGroup(pgid, signal) {
}
}
function forgetWorkerPid(pid) {
try {
if (readFileSync(workerPidFile(), "utf8").trim() === String(pid)) unlinkSync(workerPidFile());
} catch {
// no pid file, or another worker's
}
}
// The hand-run way of owning a state root (tests, development): a Unix socket lock, the worker's pid file and the
// /proc reclaim. Idunn's runtime (scripts/idunn-entry.mjs) replaces all three with its process write lease.
export function handRunRuntime(stateRoot) {
const pidFile = path.join(stateRoot, "worker.pid");
let lock = null;
// A worker left by a crashed daemon: the pid file names its group, and the group is ours only if some member
// still carries this state root's embeddings directory in its environment (the pid may have been reused).
function reclaimStrayWorker() {
let pgid;
try {
pgid = Number(readFileSync(workerPidFile(), "utf8").trim());
} catch {
return;
}
const marker = `VILI_EMBEDDINGS_DIR=${path.join(args.stateRoot, "embeddings")}`;
let ours = false;
try {
for (const entry of readdirSync("/proc")) {
if (!/^\d+$/.test(entry)) continue;
try {
const stat = readFileSync(`/proc/${entry}/stat`, "utf8");
const fields = stat.slice(stat.lastIndexOf(")") + 2).split(" ");
if (Number(fields[2]) !== pgid) continue;
if (readFileSync(`/proc/${entry}/environ`, "utf8").split("\0").includes(marker)) ours = true;
} catch {
// the process exited while we looked
}
// The lock is a listening socket in Linux's abstract namespace, named by the state root, which is canonical (its real path, see canonicalPath). A bind is
// atomic and exclusive and the kernel drops it with its owner, so there is no lock file, no stale lock to probe
// or take over, and no takeover to race: a bind either succeeds (this daemon owns the root) or it does not.
const listen = () => new Promise((resolve, reject) => {
const name = createHash("sha256").update(stateRoot).digest("hex");
const server = net.createServer((socket) => socket.destroy());
server.once("error", reject);
server.listen(`\0vili-state-root:${name}`, () => {
server.removeListener("error", reject);
lock = server;
resolve();
});
});
// A worker left by a crashed daemon: the pid file names its group, and the group is ours only if some member
// still carries this state root's embeddings directory in its environment (the pid may have been reused).
function reclaimStrayWorker() {
let pgid;
try {
pgid = Number(readFileSync(pidFile, "utf8").trim());
} catch {
return;
}
const marker = `VILI_EMBEDDINGS_DIR=${path.join(stateRoot, "embeddings")}`;
let ours = false;
try {
for (const entry of readdirSync("/proc")) {
if (!/^\d+$/.test(entry)) continue;
try {
const stat = readFileSync(`/proc/${entry}/stat`, "utf8");
const fields = stat.slice(stat.lastIndexOf(")") + 2).split(" ");
if (Number(fields[2]) !== pgid) continue;
if (readFileSync(`/proc/${entry}/environ`, "utf8").split("\0").includes(marker)) ours = true;
} catch {
// the process exited while we looked
}
}
} catch {
// no /proc: nothing can be verified, so nothing is killed
}
if (ours) killGroup(pgid, "SIGKILL");
try {
unlinkSync(pidFile);
} catch {
// already gone
}
} catch {
// no /proc: nothing can be verified, so nothing is killed
}
if (ours) killGroup(pgid, "SIGKILL");
try {
unlinkSync(workerPidFile());
} catch {
// already gone
}
return {
async acquire() {
await mkdir(stateRoot, { recursive: true });
try {
await listen();
} catch (error) {
if (error?.code !== "EADDRINUSE") throw error;
console.error("the state root is held by a running daemon");
process.exit(2);
}
reclaimStrayWorker(); // synchronous: no request is served before a predecessor's worker is gone
},
held: () => lock !== null,
refusal: () => "write-lease-pending",
watch() {},
recordWorker(pid) {
writeFileSync(pidFile, `${pid}\n`);
},
forgetWorker(pid) {
try {
if (readFileSync(pidFile, "utf8").trim() === String(pid)) unlinkSync(pidFile);
} catch {
// no pid file, or another worker's
}
},
release() {
if (lock) lock.close();
lock = null;
},
};
}
async function stopWorker() {
// A requested stop gives the worker its grace (SIGTERM, then SIGKILL); a lost writer gets SIGKILL at once.
async function stopWorker(signals = ["SIGTERM", "SIGKILL"]) {
const proc = worker.proc;
if (!proc || proc.pid === undefined) return;
worker.state = "stopped";
for (const signal of ["SIGTERM", "SIGKILL"]) {
for (const signal of signals) {
killGroup(proc.pid, signal);
if (await waitForExit(proc, signal === "SIGTERM" ? 10000 : 5000)) break;
}
}
async function startWorker(model, timeoutMs) {
guard();
await mkdir(path.join(args.stateRoot, "logs"), { recursive: true });
guard();
await mkdir(path.join(args.stateRoot, "embeddings"), { recursive: true });
guard(); // synchronous with the log open and the spawn below: no await between them
const logFd = openSync(path.join(args.stateRoot, "logs", "worker.log"), "a");
const argv = [args.workerEntry, "--host", "127.0.0.1", "--port", "0"];
if (model) argv.push("--model", model);
@ -374,10 +436,10 @@ async function startWorker(model, timeoutMs) {
});
closeSync(logFd);
if (proc.pid !== undefined) {
writeFileSync(workerPidFile(), `${proc.pid}\n`);
runtime.recordWorker(proc.pid);
proc.once("exit", () => {
killGroup(proc.pid, "SIGKILL"); // whatever the worker left running, an encoder child included
forgetWorkerPid(proc.pid);
runtime.forgetWorker(proc.pid);
});
}
Object.assign(worker, { state: "starting", proc, pid: proc.pid ?? null, port: null, servedModels: [], lastError: null });
@ -418,7 +480,7 @@ async function startWorker(model, timeoutMs) {
} catch {
// not JSON: an ordinary log line
}
writeSync(sink, `${line}\n`);
if (mayWrite()) writeSync(sink, `${line}\n`);
});
lines.on("close", () => closeSync(sink));
}),
@ -463,6 +525,16 @@ async function readRequestBody(request) {
}
async function generateMotion(request, response) {
try {
await generate(request, response);
} catch (error) {
if (!(error instanceof LeaseLost)) throw error;
json(response, 503, { ok: false, code: stopping ? "stopping" : runtime.refusal() });
}
}
async function generate(request, response) {
guard();
if (workerUnconfigured()) {
json(response, 503, { ok: false, code: "worker-unconfigured" });
return;
@ -492,6 +564,7 @@ async function generateMotion(request, response) {
const jobId = `vili-${Date.now()}`;
const artifactDir = path.join(args.stateRoot, "artifacts", jobId);
guard();
await mkdir(artifactDir, { recursive: true });
const duration = firstDefined(body.durationSeconds, body.duration, 4);
const bodyMeta = typeof body.meta === "object" && body.meta ? body.meta : {};
@ -528,6 +601,7 @@ async function generateMotion(request, response) {
return undefined;
})();
if (body.constraints !== undefined && typeof body.constraints !== "string") {
guard();
await writeFile(path.join(artifactDir, "constraints.json"), JSON.stringify(body.constraints, null, 2));
}
const requestRecord = {
@ -551,6 +625,7 @@ async function generateMotion(request, response) {
saveExampleDir: boolValue(firstDefined(body.saveExampleDir, body.save_example_dir)),
inputFolder: firstDefined(body.inputFolder, body.input_folder),
};
guard();
await writeFile(path.join(artifactDir, "meta.json"), JSON.stringify(meta, null, 2));
let result;
@ -580,6 +655,7 @@ async function generateMotion(request, response) {
saveExampleDir: boolValue(firstDefined(body.saveExampleDir, body.save_example_dir)),
output: path.join(artifactDir, "motion"),
};
guard();
await writeFile(path.join(artifactDir, "request.json"), JSON.stringify(workerRequest, null, 2));
const answer = await fetch(`http://127.0.0.1:${worker.port}/generate`, {
method: "POST",
@ -590,9 +666,11 @@ async function generateMotion(request, response) {
const workerBody = await answer.json();
result = { ok: answer.ok && workerBody?.ok !== false, error: answer.ok ? null : `the worker answered ${answer.status}`, body: workerBody };
} catch (error) {
if (error instanceof LeaseLost) throw error;
result = { ok: false, error: `the worker request failed (${error?.name || "Error"})`, body: null };
}
guard(); // the writer may have been lost while the job ran
const job = {
schema: "gamecult.vili.motion_job.v0",
jobId,
@ -612,8 +690,48 @@ async function generateMotion(request, response) {
json(response, result.ok ? 200 : 502, job);
}
const SNAPSHOT_BODY_MAX = 64 * 1024;
// Idunn's route challenge: a msgpack POST from the host, answered from local authority. A browser page can send
// it too, so an Origin is refused exactly as on every other POST. Only the Idunn runtime answers; the hand-run
// daemon has no such route.
async function answerRouteChallenge(request, response) {
if (!runtime.answerSnapshot) {
text(response, 404, "not found\n");
return;
}
if (request.headers.origin !== undefined) {
json(response, 403, { ok: false, code: "cross-site" });
return;
}
const chunks = [];
let size = 0;
for await (const chunk of request) {
size += chunk.length;
if (size > SNAPSHOT_BODY_MAX) {
json(response, 400, { ok: false, code: "invalid-snapshot-request" });
return;
}
chunks.push(chunk);
}
let answer;
try {
answer = runtime.answerSnapshot(Buffer.concat(chunks));
} catch (error) {
if (error?.code === "invalid-snapshot-request") json(response, 400, { ok: false, code: error.code });
else json(response, 503, { ok: false, code: "runtime-presence-unavailable" });
return;
}
response.writeHead(200, { "content-type": "application/msgpack", "cache-control": "no-store" });
response.end(answer);
}
async function handleRequest(request, response) {
const url = new URL(request.url, `http://${request.headers.host || `${args.host}:${args.port}`}`);
if (url.pathname === "/cultnet/snapshot" && request.method === "POST") {
await answerRouteChallenge(request, response);
return;
}
if (request.method === "POST" && refuseCrossSite(request, response)) return;
if (url.pathname === "/" || url.pathname === "/health") {
@ -627,23 +745,33 @@ async function handleRequest(request, response) {
}
}
async function healthCheck() {
const host = args.host === "0.0.0.0" ? "127.0.0.1" : args.host;
const response = await fetch(`http://${host}:${args.port}/health`);
if (!response.ok) throw new Error(`health returned ${response.status}`);
const body = await response.json();
if (!body.ok) throw new Error("health returned ok=false");
let runtime;
let stopping = false;
class LeaseLost extends Error {}
// The one write authority: the runtime holds the state root (the lease, or the lock) and the daemon is not stopping.
// held() is read at the moment of every ask, so a loss is seen by the first gate to ask.
function mayWrite() {
return !stopping && runtime.held();
}
if (args.health) {
healthCheck().then(() => process.exit(0), (error) => {
console.error(error.message);
process.exit(1);
});
} else {
await mkdir(path.join(args.stateRoot, "jobs"), { recursive: true });
await persistSurfaces();
// The one write gate. Every write to the state root and every spawn calls it, immediately before it happens (not once
// per request), so a request admitted before the lease went, or before SIGTERM, still stops at its next write.
function guard() {
if (!mayWrite()) throw new LeaseLost("the writer was lost");
}
// How often the lease is asked while nothing else asks, and so how long a worker can outlive a loss (plus the
// SIGKILL): a generate can await its worker for 30 minutes. The service documents are rewritten every PERSIST_EVERY asks.
const WATCH_MS = 250;
const PERSIST_EVERY = 20;
// The one daemon. `rt` owns the state root: the hand-run runtime (socket lock) or Idunn's (process write lease).
// The server listens first, so Idunn's challenge can be answered while the lease is still pending; nothing is
// written to the state root until the runtime holds it.
export async function serve(rt) {
runtime = rt;
const server = http.createServer((request, response) => {
handleRequest(request, response).catch((error) => {
// The error's message can carry the request; name only its class.
@ -652,20 +780,55 @@ if (args.health) {
else response.end();
});
});
await new Promise((resolve) => server.listen(args.port, args.host, resolve));
console.log(`Vili listening on http://${args.host}:${server.address().port}`);
server.listen(args.port, args.host, () => {
reclaimStrayWorker(); // synchronous: no request is served before a predecessor's worker is gone
console.log(`Vili listening on http://${args.host}:${server.address().port}`);
});
const surfaceTimer = setInterval(() => persistSurfaces().catch((error) => console.error(`persist failed: ${error?.name || "Error"}`)), 5000);
let tick = null;
// From the signal on nothing is accepted, written or spawned (guard() refuses what is already in flight); then the
// worker tree is stopped, and only then does the process end.
const shutdown = async () => {
clearInterval(surfaceTimer);
await stopWorker();
if (stopping) return;
stopping = true;
server.close();
server.closeIdleConnections();
clearInterval(tick);
await stopWorker();
runtime.release();
process.exit(0);
};
process.on("SIGTERM", shutdown);
process.on("SIGINT", shutdown);
await runtime.acquire();
let lost = false;
const onLost = () => {
if (lost) return;
lost = true;
clearInterval(tick);
stopWorker(["SIGKILL"]).finally(() => runtime.reportHealth?.("degraded"));
};
runtime.watch(onLost); // held() reports a loss through this the moment any gate sees it
try {
guard();
await mkdir(path.join(args.stateRoot, "jobs"), { recursive: true });
await persistSurfaces();
} catch (error) {
if (error instanceof LeaseLost) return; // lost while opening the state root: onLost has run, and nothing is active
throw error;
}
let asks = 0;
tick = setInterval(() => {
runtime.watch(onLost);
if (lost || ++asks % PERSIST_EVERY !== 0) return;
persistSurfaces().catch((error) => {
if (!(error instanceof LeaseLost)) console.error(`persist failed: ${error?.name || "Error"}`);
});
}, WATCH_MS);
if (lost) return;
runtime.reportHealth?.("active");
console.log("Vili holds the state root");
}
if (process.argv[1] && import.meta.url === pathToFileURL(process.argv[1]).href) {
await serve(handRunRuntime(args.stateRoot));
}

1
vendor/SOMA-X vendored Submodule

@ -0,0 +1 @@
Subproject commit 56a3d957a549c66425b500fddf7f3183bc6fbdff

1
vendor/kimodo vendored Submodule

@ -0,0 +1 @@
Subproject commit 6bb58488037dd65360ff0c5d1692b403a23309f7