Merge cut/vili-admission: Vili is an Idunn target on Yggdrasil (Soul s1-s4)
motion-pipeline:cut_spec:cut-vili-admission.r1 under the operator's ruling vili-moves-to-yggdrasil-now. Idunn's process write lease is the one writer authority (hand-run: one abstract-socket bind per real root); the release carries the pinned python-build-standalone tree and hash-locked packages; admitted live under Idunn 804ed75 with end to end, continuity restart and rollout verified. Follow-up: vili-worker-intake-local-callers. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
commit
c5cf4457c6
21 changed files with 2788 additions and 157 deletions
6
.gitmodules
vendored
6
.gitmodules
vendored
|
|
@ -1,3 +1,9 @@
|
|||
[submodule "vendor/CultLib"]
|
||||
path = vendor/CultLib
|
||||
url = https://github.com/GameCult/CultLib.git
|
||||
[submodule "vendor/kimodo"]
|
||||
path = vendor/kimodo
|
||||
url = https://github.com/nv-tlabs/kimodo.git
|
||||
[submodule "vendor/SOMA-X"]
|
||||
path = vendor/SOMA-X
|
||||
url = https://github.com/NVlabs/SOMA-X.git
|
||||
|
|
|
|||
23
README.md
23
README.md
|
|
@ -4,16 +4,15 @@ Vili is the GameCult Persona animation daemon.
|
|||
|
||||
It owns the bridge from Persona performance intent to body-motion generation. Its
|
||||
backend is Kimodo, run by Vili's own daemon as a CPU-only Python process tree on
|
||||
Linux. Admission to Idunn on Yggdrasil is the next cut; until then nothing
|
||||
supervises Vili.
|
||||
Linux. It is built to run on Yggdrasil as an Idunn target (planned, not live; see Front Door).
|
||||
|
||||
## Front Door
|
||||
|
||||
- **Value:** a supervised path from a text prompt to generated body motion. The daemon spawns one resident Kimodo worker, caps its threads, and returns motion artifacts (`motion.npz`, optional BVH) for a job.
|
||||
- **Problem:** Personas and games need generated motion, but cold per-request model loads cost minutes. One daemon owns job identity, worker lifecycle, the thread budget and artifact references, so callers touch no model state.
|
||||
- **Entry points:** `scripts/vili-daemon.mjs` (the daemon; `npm start`, `npm run health`, `npm test`), `scripts/kimodo-resident-worker.py` (the worker it spawns), `scripts/kimodo-text-encode.py` (the one-shot encoder child), `POST /motion/generate` (see Motion Generation), `docs/verse-service-contract.md` and `docs/motion-pipeline-target.md`.
|
||||
- **Live:** the process tree below, run by hand: `POST /motion/generate`, `GET /health`, `GET /operator-state`, and the CultCache service store. It produces `motion.npz` and optional BVH on the SOMA skeleton; there is no FBX export.
|
||||
- **Planned:** admission to Idunn on Yggdrasil with a locked virtual environment, then FBX export and a text-prompt-to-Unity-clip path with no hand steps (`docs/motion-pipeline-target.md`).
|
||||
- **Entry points:** `scripts/vili-daemon.mjs` (the daemon; `npm start`, `npm test`), `scripts/idunn-entry.mjs` (the process Idunn runs), `deployment/idunn/recipe.toml` (the sealed release), `scripts/kimodo-resident-worker.py` (the worker it spawns), `scripts/kimodo-text-encode.py` (the one-shot encoder child), `POST /motion/generate` (see Motion Generation), `docs/verse-service-contract.md` and `docs/motion-pipeline-target.md`.
|
||||
- **Live:** the process tree below, run by hand for development and tests: `POST /motion/generate`, `GET /health`, `GET /operator-state`, and the CultCache service store. It produces `motion.npz` and optional BVH on the SOMA skeleton; there is no FBX export.
|
||||
- **Planned:** the Idunn target `vili` on Yggdrasil at `http://10.77.0.1:8824` over WireGuard, with its provisioning runbook in `gamecult-ops`; neither exists until the first `idunn up` lands. FBX export and a text-prompt-to-Unity-clip path with no hand steps (`docs/motion-pipeline-target.md`).
|
||||
|
||||
## Authority
|
||||
|
||||
|
|
@ -37,8 +36,8 @@ vili-daemon.mjs job intake, worker lifecycle, thread cap, nice 10
|
|||
kimodo-text-encode.py one child per request with cache misses, on CPU
|
||||
```
|
||||
|
||||
Setup: `git submodule update --init`; in `vendor/CultLib`, `npm ci --ignore-scripts` and
|
||||
`npm run build --workspace=packages/cultcache-ts`; then `npm ci` here. The worker
|
||||
Setup: `git submodule update --init` (CultLib, Kimodo, SOMA-X); in `vendor/CultLib`, `npm ci --ignore-scripts` and
|
||||
`npm run build --workspace=packages/cultcache-ts` and `--workspace=packages/cultnet-ts`; then `npm ci --install-links` here. The worker
|
||||
needs a Python environment with Kimodo's pins and Kimodo itself importable, and a
|
||||
Hugging Face cache (`<models root>/hub`) holding the Llama 3 base, both LLM2Vec
|
||||
adapters and the Kimodo checkpoint. Everything runs offline.
|
||||
|
|
@ -59,7 +58,13 @@ The daemon starts the worker on the first job, on a port the worker binds itself
|
|||
and reports in its ready line, and respawns it after an exit. The worker runs at
|
||||
nice 10 with `torch` limited to `--threads` (also `OMP_NUM_THREADS` and
|
||||
`MKL_NUM_THREADS`), so a shared host keeps its other services. Its log is
|
||||
`<state>/logs/worker.log`. The daemon owns the whole worker process group: it is killed when the worker exits for any reason, on SIGTERM or SIGINT, and at start-up if a crashed daemon left one (`<state>/worker.pid`). A service unit should also own the tree (systemd `KillMode=control-group`).
|
||||
`<state>/logs/worker.log`. The daemon owns the whole worker process group: it is killed when the worker exits for any reason, on SIGTERM or SIGINT, and, for a hand-run daemon, at start-up if a crashed predecessor left one (`<state>/worker.pid`). Under Idunn the unit's cgroup owns the tree.
|
||||
|
||||
A hand-run daemon holds its state root through a listening socket in Linux's abstract namespace, named by the root's real path: the bind is the atomic claim, so a second daemon on a live root prints that it is held and exits 2, killing nothing, and a dead daemon's claim is gone with it (there is no lock file). Two daemons in separate network namespaces do not exclude each other.
|
||||
|
||||
### On Yggdrasil
|
||||
|
||||
Idunn runs `node vili/scripts/idunn-entry.mjs --state-root <root> --kimodo-commit <sha>` from a sealed release of three trees: `node`, `vili` (the daemon, worker, child, `node_modules`, cultcache-py) and `python` (python-build-standalone CPython 3.12 with the hash-locked packages, Kimodo, MotionCorrection and SOMA-X installed into it; no venv). The weights are the read-only cache `/srv/vili/models`; state is `/var/lib/gamecult/vili`. Idunn's process write lease is the one writer authority: until the daemon holds it, `POST /motion/generate` answers `503 write-lease-pending` and nothing is written; the lease is asked every 250 ms and at every write, so if it is lost the worker tree is SIGKILLed within a second and the daemon answers `503 write-lease-lost`. From SIGTERM on the daemon accepts nothing and writes and spawns nothing (`503 stopping` for a request in flight), stops the worker tree, and exits 0. Idunn proves readiness with its own `POST /cultnet/snapshot` challenge, answered with a signed runtime presence (an `Origin` is still refused).
|
||||
|
||||
`POST /motion/generate` refuses any POST that carries an `Origin` or is not
|
||||
`application/json` with `403 cross-site`; no route answers a preflight.
|
||||
|
|
@ -68,7 +73,7 @@ Vili writes daemon-owned state to `.vili/vili.service.cc`: operator state, the
|
|||
command boundary and the transport profile. Jobs are `.vili/jobs/<jobId>.json`,
|
||||
artifacts `.vili/artifacts/<jobId>/`.
|
||||
|
||||
Routes: `GET /health`, `GET /operator-state`, `POST /motion/generate`.
|
||||
Routes: `GET /health`, `GET /operator-state`, `POST /motion/generate`; under Idunn also `POST /cultnet/snapshot`, Idunn's route challenge.
|
||||
|
||||
## Motion Generation
|
||||
|
||||
|
|
|
|||
52
deployment/idunn/assemble-release.mjs
Normal file
52
deployment/idunn/assemble-release.mjs
Normal file
|
|
@ -0,0 +1,52 @@
|
|||
// Assembles the node half of Vili's sealed release (run by the recipe's node-build runner):
|
||||
// node deployment/idunn/assemble-release.mjs release/vili release/node
|
||||
// release/vili gets the daemon, the Idunn entry, the worker and child sources, the installed node_modules (installed
|
||||
// with --install-links, so cultcache-ts and cultnet-ts are real copies with their dependencies and no link reaches
|
||||
// into vendor/), cultcache-py's sources and the lock files the entry hashes; release/node gets the node binary.
|
||||
// It refuses to finish if any path it wrote is a symlink whose target is absolute: the seal would refuse the tree.
|
||||
import { chmodSync, copyFileSync, cpSync, lstatSync, mkdirSync, readdirSync, readlinkSync, rmSync } from "node:fs";
|
||||
import path from "node:path";
|
||||
|
||||
const [viliOut, nodeOut] = process.argv.slice(2);
|
||||
if (!viliOut || !nodeOut || process.argv.length !== 4) {
|
||||
console.error("usage: assemble-release.mjs <vili output dir> <node output dir>");
|
||||
process.exit(2);
|
||||
}
|
||||
|
||||
const copy = (from, to) => cpSync(from, to, { recursive: true, verbatimSymlinks: true, errorOnExist: true });
|
||||
|
||||
rmSync(viliOut, { recursive: true, force: true });
|
||||
rmSync(nodeOut, { recursive: true, force: true });
|
||||
mkdirSync(path.join(viliOut, "scripts"), { recursive: true });
|
||||
mkdirSync(path.join(viliOut, "deployment", "idunn"), { recursive: true });
|
||||
mkdirSync(nodeOut, { recursive: true });
|
||||
|
||||
// Runtime sources only: the tests and their fixtures stay out of the release.
|
||||
for (const entry of readdirSync("scripts", { withFileTypes: true })) {
|
||||
if (!entry.isFile() || !/\.(mjs|py)$/u.test(entry.name) || /^test[-_]/u.test(entry.name)) continue;
|
||||
copyFileSync(path.join("scripts", entry.name), path.join(viliOut, "scripts", entry.name));
|
||||
}
|
||||
copyFileSync("package.json", path.join(viliOut, "package.json"));
|
||||
copy("node_modules", path.join(viliOut, "node_modules"));
|
||||
copy(path.join("vendor", "CultLib", "packages", "cultcache-py", "src"), path.join(viliOut, "vendor", "CultLib", "packages", "cultcache-py", "src"));
|
||||
for (const lock of ["requirements-build.lock", "requirements.lock"]) {
|
||||
copyFileSync(path.join("deployment", "idunn", lock), path.join(viliOut, "deployment", "idunn", lock));
|
||||
}
|
||||
copyFileSync(process.execPath, path.join(nodeOut, "node"));
|
||||
chmodSync(path.join(nodeOut, "node"), 0o755);
|
||||
|
||||
function refuseAbsoluteLinks(directory) {
|
||||
for (const entry of readdirSync(directory, { withFileTypes: true })) {
|
||||
const full = path.join(directory, entry.name);
|
||||
if (lstatSync(full).isSymbolicLink()) {
|
||||
if (path.isAbsolute(readlinkSync(full))) {
|
||||
console.error(`absolute symlink in the release: ${path.relative(process.cwd(), full)}`);
|
||||
process.exit(1);
|
||||
}
|
||||
} else if (entry.isDirectory()) {
|
||||
refuseAbsoluteLinks(full);
|
||||
}
|
||||
}
|
||||
}
|
||||
refuseAbsoluteLinks(viliOut);
|
||||
refuseAbsoluteLinks(nodeOut);
|
||||
216
deployment/idunn/recipe.toml
Normal file
216
deployment/idunn/recipe.toml
Normal file
|
|
@ -0,0 +1,216 @@
|
|||
schema = "gamecult.idunn.target_declaration.v1"
|
||||
target = "vili"
|
||||
source_stamp_environment = "VILI_BUILD_COMMIT"
|
||||
required_gitlinks = ["vendor/CultLib", "vendor/kimodo", "vendor/SOMA-X"]
|
||||
|
||||
# The one external input: a relocatable CPython. The locked site-packages are installed into this tree itself: a copied
|
||||
# environment would carry an absolute interpreter link and a recorded build path, and the seal refuses both.
|
||||
[[external_inputs]]
|
||||
id = "cpython"
|
||||
url = "https://github.com/astral-sh/python-build-standalone/releases/download/20261021/cpython-3.12.15+20261021-x86_64-unknown-linux-gnu-install_only.tar.gz"
|
||||
sha256 = "5ff9f0602ce964a529ed1e8555d3abab6341926d724402ba9a15e9e877191124"
|
||||
runner = "python-build"
|
||||
destination = "release/cpython.tar.gz"
|
||||
|
||||
# ---- node-build: the daemon, the Idunn entry and its dependencies
|
||||
|
||||
[[steps]]
|
||||
id = "install-cultlib-build-dependencies"
|
||||
phase = "prepare"
|
||||
runner = "node-build"
|
||||
working_directory = "vendor/CultLib"
|
||||
required_environment = ["HOME", "NPM_CONFIG_CACHE"]
|
||||
argv = ["npm", "ci", "--ignore-scripts"]
|
||||
|
||||
[[steps]]
|
||||
id = "build-cultcache-ts"
|
||||
phase = "prepare"
|
||||
runner = "node-build"
|
||||
working_directory = "vendor/CultLib"
|
||||
required_environment = ["HOME", "NPM_CONFIG_CACHE"]
|
||||
argv = ["npm", "run", "build", "--workspace=packages/cultcache-ts"]
|
||||
|
||||
[[steps]]
|
||||
id = "build-cultnet-ts"
|
||||
phase = "prepare"
|
||||
runner = "node-build"
|
||||
working_directory = "vendor/CultLib"
|
||||
required_environment = ["HOME", "NPM_CONFIG_CACHE"]
|
||||
argv = ["npm", "run", "build", "--workspace=packages/cultnet-ts"]
|
||||
|
||||
[[steps]]
|
||||
id = "install-vili-dependencies"
|
||||
phase = "prepare"
|
||||
runner = "node-build"
|
||||
required_environment = ["HOME", "NPM_CONFIG_CACHE"]
|
||||
argv = ["npm", "ci", "--ignore-scripts", "--install-links"]
|
||||
|
||||
[[steps]]
|
||||
id = "test-daemon"
|
||||
phase = "test"
|
||||
runner = "node-build"
|
||||
required_environment = ["HOME", "NPM_CONFIG_CACHE"]
|
||||
argv = ["node", "--test", "--test-timeout=60000", "scripts/test-vili-daemon.mjs"]
|
||||
|
||||
[[steps]]
|
||||
id = "assemble-vili-release"
|
||||
phase = "build"
|
||||
runner = "node-build"
|
||||
required_environment = ["HOME", "NPM_CONFIG_CACHE"]
|
||||
argv = ["node", "deployment/idunn/assemble-release.mjs", "release/vili", "release/node"]
|
||||
|
||||
# ---- python-build: the interpreter and its locked packages
|
||||
|
||||
[[steps]]
|
||||
id = "unpack-cpython"
|
||||
phase = "prepare"
|
||||
runner = "python-build"
|
||||
argv = ["tar", "-xzf", "release/cpython.tar.gz", "-C", "release"]
|
||||
|
||||
[[steps]]
|
||||
id = "install-build-tools"
|
||||
phase = "prepare"
|
||||
runner = "python-build"
|
||||
required_environment = ["HOME", "PIP_CACHE_DIR", "TMPDIR", "PATH", "PIP_DISABLE_PIP_VERSION_CHECK", "PIP_NO_INPUT"]
|
||||
argv = ["python3.12", "-m", "pip", "install", "--no-deps", "--require-hashes", "-r", "deployment/idunn/requirements-build.lock"]
|
||||
|
||||
[[steps]]
|
||||
id = "install-locked-packages"
|
||||
phase = "prepare"
|
||||
runner = "python-build"
|
||||
required_environment = ["HOME", "PIP_CACHE_DIR", "TMPDIR", "PATH", "PIP_DISABLE_PIP_VERSION_CHECK", "PIP_NO_INPUT"]
|
||||
argv = ["python3.12", "-m", "pip", "install", "--no-deps", "--require-hashes", "--no-build-isolation", "-r", "deployment/idunn/requirements.lock"]
|
||||
|
||||
[[steps]]
|
||||
id = "install-vendored-packages"
|
||||
phase = "prepare"
|
||||
runner = "python-build"
|
||||
required_environment = ["HOME", "PIP_CACHE_DIR", "TMPDIR", "PATH", "PIP_DISABLE_PIP_VERSION_CHECK", "PIP_NO_INPUT"]
|
||||
argv = ["python3.12", "-m", "pip", "install", "--no-deps", "--no-build-isolation", "--no-index", "vendor/SOMA-X", "vendor/kimodo/MotionCorrection", "vendor/kimodo"]
|
||||
|
||||
[[steps]]
|
||||
id = "test-worker"
|
||||
phase = "test"
|
||||
runner = "python-build"
|
||||
working_directory = "."
|
||||
required_environment = ["HOME", "TMPDIR", "PYTHONPATH", "PATH"]
|
||||
argv = ["python3.12", "-m", "unittest", "discover", "-s", "scripts", "-p", "test_*.py"]
|
||||
|
||||
# Every module the worker and the encoder child import at top level, kimodo.scripts.generate included.
|
||||
[[steps]]
|
||||
id = "import-smoke"
|
||||
phase = "acceptance"
|
||||
runner = "python-build"
|
||||
required_environment = ["HOME", "TMPDIR", "PYTHONPATH", "PATH"]
|
||||
argv = [
|
||||
"python3.12", "-c",
|
||||
"import accelerate, cultcache_py, huggingface_hub.constants, kimodo, kimodo.constraints, kimodo.exports.motion_io, kimodo.meta, kimodo.model.llm2vec.llm2vec, kimodo.model.load_model, kimodo.model.loading, kimodo.model.registry, kimodo.scripts.generate, kimodo.tools, msgpack, peft, torch, transformers",
|
||||
]
|
||||
|
||||
[[steps]]
|
||||
id = "compile-python"
|
||||
phase = "build"
|
||||
runner = "python-build"
|
||||
required_environment = ["PATH"]
|
||||
argv = ["python3.12", "-m", "compileall", "-q", "-j", "4", "--invalidation-mode", "unchecked-hash", "release/python/lib/python3.12"]
|
||||
|
||||
# ---- the sealed release: node, vili, python
|
||||
|
||||
[[artifacts]]
|
||||
id = "node"
|
||||
source_kind = "runner-output"
|
||||
runner = "node-build"
|
||||
source = "release/node/node"
|
||||
destination = "node"
|
||||
executable = true
|
||||
|
||||
[[artifacts]]
|
||||
id = "vili"
|
||||
source_kind = "runner-output"
|
||||
runner = "node-build"
|
||||
source = "release/vili"
|
||||
destination = "vili"
|
||||
|
||||
[[artifacts]]
|
||||
id = "python"
|
||||
source_kind = "runner-output"
|
||||
runner = "python-build"
|
||||
source = "release/python"
|
||||
destination = "python"
|
||||
executable_members = ["bin/python3.12"]
|
||||
|
||||
[service]
|
||||
executable_artifact = "node"
|
||||
required_adjacent_artifacts = ["vili", "python"]
|
||||
arguments = [
|
||||
{ kind = "literal", value = "vili/scripts/idunn-entry.mjs" },
|
||||
{ kind = "literal", value = "--state-root" },
|
||||
{ kind = "binding", name = "state_root" },
|
||||
{ kind = "literal", value = "--kimodo-commit" },
|
||||
{ kind = "literal", value = "6bb58488037dd65360ff0c5d1692b403a23309f7" },
|
||||
]
|
||||
transport = "http"
|
||||
route_required = true
|
||||
required_environment = [
|
||||
"GAMECULT_IDUNN_CANDIDATE_BIND",
|
||||
"GAMECULT_IDUNN_PROCESS_WRITE_LEASE",
|
||||
"GAMECULT_IDUNN_RUNTIME_BUNDLE",
|
||||
"VILI_MODELS_ROOT",
|
||||
"VILI_THREADS",
|
||||
]
|
||||
|
||||
[service.health]
|
||||
contract = "gamecult.runtime_presence_health.v2"
|
||||
|
||||
[state]
|
||||
schema_generation = "vili-v1"
|
||||
|
||||
[[state.slots]]
|
||||
id = "service-store"
|
||||
relative_path = "vili.service.cc"
|
||||
kind = "cultcache-file"
|
||||
schema = "gamecult.vili.operator_state.v1"
|
||||
writer = "process-bound-single-writer"
|
||||
recovery = "rebuildable"
|
||||
startup = "create-or-open-after-write-lease"
|
||||
|
||||
[[state.slots]]
|
||||
id = "text-embeddings"
|
||||
relative_path = "embeddings/text-embeddings.cc"
|
||||
kind = "cultcache-file"
|
||||
schema = "gamecult.vili.text_embedding.v0"
|
||||
writer = "process-bound-single-writer"
|
||||
recovery = "rebuildable"
|
||||
startup = "create-or-open-after-write-lease"
|
||||
|
||||
[[state.slots]]
|
||||
id = "jobs"
|
||||
relative_path = "jobs"
|
||||
kind = "external-directory"
|
||||
schema = "gamecult.vili.motion_job.v0"
|
||||
writer = "process-bound-single-writer"
|
||||
recovery = "preserve"
|
||||
startup = "create-or-open-after-write-lease"
|
||||
|
||||
[[state.slots]]
|
||||
id = "artifacts"
|
||||
relative_path = "artifacts"
|
||||
kind = "external-directory"
|
||||
schema = "gamecult.vili.motion_artifact.v0"
|
||||
writer = "process-bound-single-writer"
|
||||
recovery = "preserve"
|
||||
startup = "create-or-open-after-write-lease"
|
||||
|
||||
[[state.slots]]
|
||||
id = "logs"
|
||||
relative_path = "logs"
|
||||
kind = "external-directory"
|
||||
schema = "text.log"
|
||||
writer = "process-bound-single-writer"
|
||||
recovery = "rebuildable"
|
||||
startup = "create-or-open-after-write-lease"
|
||||
|
||||
[[provides]]
|
||||
capability = "vili.motion.api"
|
||||
schema = "http.v1"
|
||||
compatibility = "v1"
|
||||
4
deployment/idunn/requirements-build.in
Normal file
4
deployment/idunn/requirements-build.in
Normal file
|
|
@ -0,0 +1,4 @@
|
|||
setuptools
|
||||
wheel
|
||||
pybind11
|
||||
cmake
|
||||
39
deployment/idunn/requirements-build.lock
Normal file
39
deployment/idunn/requirements-build.lock
Normal file
|
|
@ -0,0 +1,39 @@
|
|||
# This file was autogenerated by uv via the following command:
|
||||
# uv pip compile --generate-hashes --python-version 3.12 --python-platform x86_64-manylinux_2_28 requirements-build.in -o requirements-build.lock
|
||||
cmake==4.4.4 \
|
||||
--hash=sha256:02e9599cba932b386dc5c7e29db5dbda0610d13263798ff1521298c18014357d \
|
||||
--hash=sha256:3c3f40c4656ec1ad1b873dd502bb31b3bba75295ae66b2623e515f2931eddbcb \
|
||||
--hash=sha256:45a5c5e058e1e456f8da99e8bb83c98df079f385afd0bbea222a7eab6fdc1a8f \
|
||||
--hash=sha256:6b914a7d9492d6134d43d484500cacd3a70466f6e31fcaf5145cf87128c28114 \
|
||||
--hash=sha256:7b4a9c0d8146691d91b094633e4cd4f0cc4cdb8e04db0e8d805056db064396e0 \
|
||||
--hash=sha256:883962a72b1b3a16ff445427f24c8caca9797dd9e563bbe208cbcd89e877f9d3 \
|
||||
--hash=sha256:959fdcb277c63180928fe2c388063aa26dcfa1968033f7904eed1948ed44283b \
|
||||
--hash=sha256:9ca4d839ced82f92b69319f50a65addf3051fae5d34464d9ac5b0296aba6ccb3 \
|
||||
--hash=sha256:a3b4d4a35ba69360e630d2ce33020f42a61e09af28fe81122697989d11be8b39 \
|
||||
--hash=sha256:ad8e0a38b5707e27882701146bdfffecacd80e7703fa0fc36c77528518d545af \
|
||||
--hash=sha256:b3029f586853e01ddf2824c1ff124a1c5284f8ae897994758e9c4eeb45903b58 \
|
||||
--hash=sha256:b4ed2c1a8f9c90e37a1f3095f7ed94979ebf3f94d0e90e63d50f4e72d419773a \
|
||||
--hash=sha256:c7def2c08d9d0d31bab239209946e45a97520518ae63c0401772206f28b81523 \
|
||||
--hash=sha256:d25e1d6baad137ddf21e576bf5296cd941bdd0da85baedbf78f673261fd2fca3 \
|
||||
--hash=sha256:e2d98233f16b5d6b9d4673a0a724ce801fc413811befb8f0b008cbffb7a2ad63 \
|
||||
--hash=sha256:eb0ff21b4309828f8fc0f0542022768f24b15aa1af3284e16c1eb01cdbe7d3e5 \
|
||||
--hash=sha256:ed50c732a82ac04f6e5606a0677ddfb49dda9ee130f9d4163c5df217e478417e \
|
||||
--hash=sha256:f337d18e33b116cabae9a7a8ff423fc0bc3f389da0acefeb724e1397128dcc34 \
|
||||
--hash=sha256:f773a0544c66370408f451acd487c2c643b0dfebb48c5d64f3638bc1bb820238
|
||||
# via -r requirements-build.in
|
||||
packaging==26.3 \
|
||||
--hash=sha256:94edc256424af38762eb31306eed28beb9f0efc50a8837492c9d6fd6004aed79 \
|
||||
--hash=sha256:d7193f7c8e4e93f444fde0262bf90af30e16fa0ad0ad44cb553c87339b23cd1c
|
||||
# via wheel
|
||||
pybind11==3.1.0 \
|
||||
--hash=sha256:a1cc06b524ab3edca51f8ad3895f9c4fa20b8b19283173dff4ae781449dc9639 \
|
||||
--hash=sha256:b8488090f8acffbcb6b5d6a85571a6827a0a2981ffb75e5a0b27b87c4a6b7dd0
|
||||
# via -r requirements-build.in
|
||||
setuptools==84.2.1 \
|
||||
--hash=sha256:3ff8b3b5492a0aa66960f4419365a86b84e013070d4313e4d4a6894f5d64d6bc \
|
||||
--hash=sha256:ca140b45d5325a9478f880e351484f62fef15f336462e449ca0c03c873d589c2
|
||||
# via -r requirements-build.in
|
||||
wheel==0.48.0 \
|
||||
--hash=sha256:3217dcc807155e45db462d7ef2431f5ddda0d7273b700d05a67b271ceb1287ab \
|
||||
--hash=sha256:94800765601e9171bf5d58d066e640662842bcedcbab982b2c90787a2c987322
|
||||
# via -r requirements-build.in
|
||||
31
deployment/idunn/requirements.in
Normal file
31
deployment/idunn/requirements.in
Normal file
|
|
@ -0,0 +1,31 @@
|
|||
# Vili's CPU worker: torch from the CPU index, Kimodo 6bb58488's runtime pins as the CPU probe installed them
|
||||
# (motion-pipeline map, Body facts; probe Dockerfile), msgpack for cultcache-py and the worker's imports.
|
||||
# kimodo, MotionCorrection and SOMA-X install from the vendored gitlinks, never from here.
|
||||
# Compile on Yggdrasil in a container (never on Starfire):
|
||||
# uv pip compile --generate-hashes --python-version 3.12 --python-platform x86_64-manylinux_2_28 \
|
||||
# --index-strategy unsafe-best-match -c requirements-build.lock requirements.in -o requirements.lock
|
||||
--extra-index-url https://download.pytorch.org/whl/cpu
|
||||
torch==2.5.1+cpu
|
||||
msgpack
|
||||
hydra-core==1.3.2
|
||||
omegaconf==2.3.0
|
||||
scipy==1.15.3
|
||||
einops==0.8.2
|
||||
tqdm
|
||||
pydantic==2.12.5
|
||||
trimesh==4.11.3
|
||||
pillow
|
||||
numpy==1.26.4
|
||||
peft==0.18.1
|
||||
transformers==5.1.0
|
||||
accelerate==1.13.0
|
||||
safetensors==0.7.0
|
||||
huggingface-hub==1.6.0
|
||||
tokenizers==0.22.2
|
||||
psutil
|
||||
packaging
|
||||
filelock
|
||||
boto3
|
||||
bvhio
|
||||
gradio_client>=1.0
|
||||
urllib3>=2.6.3
|
||||
1185
deployment/idunn/requirements.lock
Normal file
1185
deployment/idunn/requirements.lock
Normal file
File diff suppressed because it is too large
Load diff
|
|
@ -17,13 +17,16 @@ for Odin and Idunn. The request fields and the runtime flags are in `README.md`
|
|||
|
||||
## Witnesses
|
||||
|
||||
- `.vili/vili.service.cc`: the daemon-owned CultCache store for operator state,
|
||||
- `.vili/vili.service.cc` (`/var/lib/gamecult/vili/vili.service.cc` under Idunn): the daemon-owned CultCache store for operator state,
|
||||
the command boundary and the transport profile.
|
||||
- `.vili/jobs/{jobId}.json`: accepted animation job records.
|
||||
- `.vili/artifacts/{jobId}/`: generated motion outputs when generation succeeds.
|
||||
- `.vili/embeddings/text-embeddings.cc`: the worker's text-embedding cache.
|
||||
- `.vili/worker.pid`: the process group of the running worker, so a restarted
|
||||
daemon can find and kill a worker its crashed predecessor left behind.
|
||||
- `.vili/worker.pid`: the hand-run daemon's only: the process group of the running
|
||||
worker, so a restarted hand-run daemon can find and kill a worker its crashed
|
||||
predecessor left behind. Under Idunn it is never written; the unit's cgroup owns the tree.
|
||||
- The hand-run daemon's abstract-socket lock (named by the state root's real path, no file) and Idunn's process write lease (Yggdrasil): the
|
||||
writer authority for the state root. No state is written before the lease is held.
|
||||
- `.vili/logs/worker.log`: the worker's stdout and stderr.
|
||||
|
||||
## Command Boundary
|
||||
|
|
|
|||
103
package-lock.json
generated
103
package-lock.json
generated
|
|
@ -8,7 +8,9 @@
|
|||
"name": "@gamecult/vili",
|
||||
"version": "0.1.0",
|
||||
"dependencies": {
|
||||
"cultcache-ts": "file:vendor/CultLib/packages/cultcache-ts"
|
||||
"@gamecult/cultcache-ts": "file:vendor/CultLib/packages/cultcache-ts",
|
||||
"@msgpack/msgpack": "3.1.3",
|
||||
"cultnet-ts": "file:vendor/CultLib/packages/cultnet-ts"
|
||||
}
|
||||
},
|
||||
"../CultLib/packages/cultcache-ts": {
|
||||
|
|
@ -29,6 +31,19 @@
|
|||
"node": ">=20"
|
||||
}
|
||||
},
|
||||
"node_modules/@gamecult/cultcache-ts": {
|
||||
"version": "0.14.2",
|
||||
"resolved": "file:vendor/CultLib/packages/cultcache-ts",
|
||||
"license": "MIT",
|
||||
"dependencies": {
|
||||
"@msgpack/msgpack": "^3.1.3",
|
||||
"koffi": "3.3.0",
|
||||
"zod": "^3.25.76"
|
||||
},
|
||||
"engines": {
|
||||
"node": ">=20"
|
||||
}
|
||||
},
|
||||
"node_modules/@koromix/koffi-android-arm64": {
|
||||
"version": "3.3.0",
|
||||
"resolved": "https://registry.npmjs.org/@koromix/koffi-android-arm64/-/koffi-android-arm64-3.3.0.tgz",
|
||||
|
|
@ -358,19 +373,62 @@
|
|||
"node": ">= 18"
|
||||
}
|
||||
},
|
||||
"node_modules/@types/node": {
|
||||
"version": "24.19.2",
|
||||
"resolved": "https://registry.npmjs.org/@types/node/-/node-24.19.2.tgz",
|
||||
"integrity": "sha512-93d49wCWJkCyN87MxLpNN6KXPCwFw7bVUg79mGUV1OS40JGWwVCr8n61DXCSqz7CV6/VfPkb7xec7uP2M+QHWQ==",
|
||||
"dev": true,
|
||||
"node_modules/ajv": {
|
||||
"version": "8.20.0",
|
||||
"resolved": "https://registry.npmjs.org/ajv/-/ajv-8.20.0.tgz",
|
||||
"integrity": "sha512-Thbli+OlOj+iMPYFBVBfJ3OmCAnaSyNn4M1vz9T6Gka5Jt9ba/HIR56joy65tY6kx/FCF5VXNB819Y7/GUrBGA==",
|
||||
"license": "MIT",
|
||||
"dependencies": {
|
||||
"undici-types": ">=7.24.0 <7.24.7"
|
||||
"fast-deep-equal": "^3.1.3",
|
||||
"fast-uri": "^3.0.1",
|
||||
"json-schema-traverse": "^1.0.0",
|
||||
"require-from-string": "^2.0.2"
|
||||
},
|
||||
"funding": {
|
||||
"type": "github",
|
||||
"url": "https://github.com/sponsors/epoberezkin"
|
||||
}
|
||||
},
|
||||
"node_modules/cultcache-ts": {
|
||||
"resolved": "vendor/CultLib/packages/cultcache-ts",
|
||||
"link": true
|
||||
"node_modules/cultnet-ts": {
|
||||
"version": "0.3.0",
|
||||
"resolved": "file:vendor/CultLib/packages/cultnet-ts",
|
||||
"license": "MIT",
|
||||
"dependencies": {
|
||||
"@gamecult/cultcache-ts": "^0.14.1",
|
||||
"@msgpack/msgpack": "^3.1.3",
|
||||
"ajv": "^8.17.1"
|
||||
},
|
||||
"engines": {
|
||||
"node": ">=20"
|
||||
}
|
||||
},
|
||||
"node_modules/fast-deep-equal": {
|
||||
"version": "3.1.3",
|
||||
"resolved": "https://registry.npmjs.org/fast-deep-equal/-/fast-deep-equal-3.1.3.tgz",
|
||||
"integrity": "sha512-f3qQ9oQy9j2AhBe/H9VC91wLmKBCCU/gDOnKNAYG5hswO7BLKj09Hc5HYNz9cGI++xlpDCIgDaitVs03ATR84Q==",
|
||||
"license": "MIT"
|
||||
},
|
||||
"node_modules/fast-uri": {
|
||||
"version": "3.1.9",
|
||||
"resolved": "https://registry.npmjs.org/fast-uri/-/fast-uri-3.1.9.tgz",
|
||||
"integrity": "sha512-hS/0Wt3Ya043oKXs0sIxrMpKQITXhMLIrVAKKXi4gozGC9v7YRWVnXIZzjNPTO02M9BdxLCWqK1yRXgNziVyJQ==",
|
||||
"funding": [
|
||||
{
|
||||
"type": "github",
|
||||
"url": "https://github.com/sponsors/fastify"
|
||||
},
|
||||
{
|
||||
"type": "opencollective",
|
||||
"url": "https://opencollective.com/fastify"
|
||||
}
|
||||
],
|
||||
"license": "BSD-3-Clause"
|
||||
},
|
||||
"node_modules/json-schema-traverse": {
|
||||
"version": "1.0.0",
|
||||
"resolved": "https://registry.npmjs.org/json-schema-traverse/-/json-schema-traverse-1.0.0.tgz",
|
||||
"integrity": "sha512-NM8/P9n3XjXhIZn1lLhkFaACTOURQXjWhV4BA/RnOv8xvgqtqpAX9IO4mRQxSx1Rlo4tqzeqb0sOlruaOy3dug==",
|
||||
"license": "MIT"
|
||||
},
|
||||
"node_modules/koffi": {
|
||||
"version": "3.3.0",
|
||||
|
|
@ -404,27 +462,15 @@
|
|||
"@koromix/koffi-win32-x64": "3.3.0"
|
||||
}
|
||||
},
|
||||
"node_modules/typescript": {
|
||||
"version": "5.9.3",
|
||||
"resolved": "https://registry.npmjs.org/typescript/-/typescript-5.9.3.tgz",
|
||||
"integrity": "sha512-jl1vZzPDinLr9eUt3J/t7V6FgNEw9QjvBPdysz9KfQDD41fQrC2Y4vKQdiaUpFT4bXlb1RHhLpp8wtm6M5TgSw==",
|
||||
"dev": true,
|
||||
"license": "Apache-2.0",
|
||||
"bin": {
|
||||
"tsc": "bin/tsc",
|
||||
"tsserver": "bin/tsserver"
|
||||
},
|
||||
"node_modules/require-from-string": {
|
||||
"version": "2.0.2",
|
||||
"resolved": "https://registry.npmjs.org/require-from-string/-/require-from-string-2.0.2.tgz",
|
||||
"integrity": "sha512-Xf0nWe6RseziFMu+Ap9biiUbmplq6S9/p+7w7YXP/JBHhrUDDUhwa+vANyubuqfZWTveU//DYVGsDG7RKL/vEw==",
|
||||
"license": "MIT",
|
||||
"engines": {
|
||||
"node": ">=14.17"
|
||||
"node": ">=0.10.0"
|
||||
}
|
||||
},
|
||||
"node_modules/undici-types": {
|
||||
"version": "7.24.6",
|
||||
"resolved": "https://registry.npmjs.org/undici-types/-/undici-types-7.24.6.tgz",
|
||||
"integrity": "sha512-WRNW+sJgj5OBN4/0JpHFqtqzhpbnV0GuB+OozA9gCL7a993SmU+1JBZCzLNxYsbMfIeDL+lTsphD5jN5N+n0zg==",
|
||||
"dev": true,
|
||||
"license": "MIT"
|
||||
},
|
||||
"node_modules/zod": {
|
||||
"version": "3.25.76",
|
||||
"resolved": "https://registry.npmjs.org/zod/-/zod-3.25.76.tgz",
|
||||
|
|
@ -437,6 +483,7 @@
|
|||
"vendor/CultLib/packages/cultcache-ts": {
|
||||
"name": "@gamecult/cultcache-ts",
|
||||
"version": "0.14.2",
|
||||
"extraneous": true,
|
||||
"license": "MIT",
|
||||
"dependencies": {
|
||||
"@msgpack/msgpack": "^3.1.3",
|
||||
|
|
|
|||
|
|
@ -6,10 +6,14 @@
|
|||
"description": "Vili is the GameCult Persona animation daemon: speech intent in, embodied motion affordance out.",
|
||||
"scripts": {
|
||||
"start": "node scripts/vili-daemon.mjs",
|
||||
"health": "node scripts/vili-daemon.mjs --health",
|
||||
"test": "node --test --test-timeout=60000 scripts/test-vili-daemon.mjs"
|
||||
},
|
||||
"dependencies": {
|
||||
"cultcache-ts": "file:vendor/CultLib/packages/cultcache-ts"
|
||||
"@gamecult/cultcache-ts": "file:vendor/CultLib/packages/cultcache-ts",
|
||||
"@msgpack/msgpack": "3.1.3",
|
||||
"cultnet-ts": "file:vendor/CultLib/packages/cultnet-ts"
|
||||
},
|
||||
"overrides": {
|
||||
"@gamecult/cultcache-ts": "file:vendor/CultLib/packages/cultcache-ts"
|
||||
}
|
||||
}
|
||||
|
|
|
|||
39
scripts/fixtures/fake-idunn-entry.mjs
Normal file
39
scripts/fixtures/fake-idunn-entry.mjs
Normal file
|
|
@ -0,0 +1,39 @@
|
|||
// Test seam only: the shipped Idunn runtime (scripts/idunn-entry.mjs idunnRuntime) over a signer whose write lease is
|
||||
// a file, so the lease gate is tested without Idunn's authority files. Present = leased. The daemon's own flags
|
||||
// arrive on argv as they do for the hand-run daemon.
|
||||
import { existsSync, appendFileSync, readdirSync } from "node:fs";
|
||||
import path from "node:path";
|
||||
import { idunnRuntime } from "../idunn-entry.mjs";
|
||||
import { serve } from "../vili-daemon.mjs";
|
||||
|
||||
const leasePath = process.env.FAKE_IDUNN_LEASE;
|
||||
const healthLog = process.env.FAKE_IDUNN_HEALTH_LOG;
|
||||
const answer = process.env.FAKE_IDUNN_ANSWER_THROWS !== "1";
|
||||
// The lease is gone from the moment a path exists under the state root (one "*" segment matches any entry), so a
|
||||
// test can lose it between two writes of one request.
|
||||
const stateRoot = process.env.FAKE_IDUNN_STATE_ROOT;
|
||||
const loseWhenExists = process.env.FAKE_IDUNN_LOSE_WHEN_EXISTS;
|
||||
|
||||
function exists(parts, base) {
|
||||
if (!parts.length) return true;
|
||||
const [head, ...rest] = parts;
|
||||
if (head !== "*") return existsSync(path.join(base, head)) && exists(rest, path.join(base, head));
|
||||
return existsSync(base) && readdirSync(base).some((entry) => exists(rest, path.join(base, entry)));
|
||||
}
|
||||
|
||||
const signer = {
|
||||
assertWriteLease() {
|
||||
if (!existsSync(leasePath)) throw new Error("no lease");
|
||||
if (loseWhenExists && exists(loseWhenExists.split("/"), stateRoot)) throw new Error("lease lost");
|
||||
return "lease";
|
||||
},
|
||||
answerRouteObservation(request) {
|
||||
if (!answer) throw new Error("signer refused");
|
||||
return { schemaId: "fake.presence", recordKey: request.messageId, payload: new Uint8Array([1, 2, 3]) };
|
||||
},
|
||||
reportHealth(state) {
|
||||
appendFileSync(healthLog, `${state}\n`);
|
||||
},
|
||||
};
|
||||
|
||||
await serve(idunnRuntime(signer, { pollMs: 200 }));
|
||||
|
|
@ -1,12 +1,12 @@
|
|||
// Stands in for kimodo-resident-worker.py in the daemon's tests. It binds port 0, prints the ready line the
|
||||
// daemon waits for, and answers /health and /generate with what the daemon gave it. The --model value picks a
|
||||
// misbehaviour: crash-before-ready, hang-before-ready, sick, slow-health, stubborn, fail-generate,
|
||||
// refuse-generate, drop-generate, slow-generate, encoder (a grandchild, then no answer), graceful (exits 0.5 s after SIGTERM), die-after-ready. Any other value is just the model the worker serves.
|
||||
// refuse-generate, drop-generate, slow-generate, writer (appends to <output>.writes every 50 ms, never answers, ignores SIGTERM), slow-to-stop (no answer; prints a line and exits 1.5 s after SIGTERM), chatty (prints a line the moment it is healthy), encoder (a grandchild, then no answer), graceful (exits 0.5 s after SIGTERM), die-after-ready. Any other value is just the model the worker serves.
|
||||
import http from "node:http";
|
||||
import os from "node:os";
|
||||
import path from "node:path";
|
||||
import { spawn } from "node:child_process";
|
||||
import { appendFileSync } from "node:fs";
|
||||
import { appendFileSync, writeFileSync } from "node:fs";
|
||||
|
||||
const argv = process.argv.slice(2);
|
||||
const flag = (name) => {
|
||||
|
|
@ -29,10 +29,16 @@ if (model === "crash-before-ready") process.exit(3);
|
|||
|
||||
if (model === "stubborn") {
|
||||
process.on("SIGTERM", () => {});
|
||||
const grandchild = spawn(process.execPath, ["-e", "setInterval(() => {}, 1000)"], { stdio: "ignore" });
|
||||
appendFileSync(spawnLog, `grandchild ${grandchild.pid}\n`);
|
||||
// the grandchild reports itself once its handler is armed, so a test never reads a pid that would die of the default SIGTERM
|
||||
spawn(process.execPath, ["-e", `process.on('SIGTERM', () => {}); require('fs').appendFileSync(${JSON.stringify(spawnLog)}, 'grandchild ' + process.pid + '\\n'); setInterval(() => {}, 1000)`], { stdio: "ignore" });
|
||||
}
|
||||
|
||||
if (model === "slow-to-stop") process.on("SIGTERM", () => {
|
||||
console.log("stopping-line"); // output while it stops
|
||||
setTimeout(() => process.exit(0), 1500);
|
||||
});
|
||||
if (model === "writer") process.on("SIGTERM", () => {}); // a worker that carries on writing through SIGTERM
|
||||
|
||||
if (model === "graceful") {
|
||||
process.on("SIGTERM", () => {
|
||||
setTimeout(() => {
|
||||
|
|
@ -56,6 +62,8 @@ if (model === "hang-before-ready") {
|
|||
if (request.method === "GET" && request.url === "/health") {
|
||||
healthCalls += 1;
|
||||
if (model === "sick" || model === "die-after-ready" || (model === "slow-health" && healthCalls <= 2)) return send(200, { ok: false });
|
||||
writeFileSync(path.join(process.env.VILI_EMBEDDINGS_DIR, "stub-healthy"), ""); // a test can act the moment the worker is healthy
|
||||
if (model === "chatty") console.log("after-healthy-line"); // after stub-healthy exists, which a test can make the loss of the lease
|
||||
send(200, { ok: true, model: model || "stub-model", displayModel: model || "Stub Model", pid: process.pid, requests });
|
||||
} else if (request.method === "POST" && request.url === "/generate") {
|
||||
requests += 1;
|
||||
|
|
@ -63,11 +71,14 @@ if (model === "hang-before-ready") {
|
|||
if (model === "fail-generate") return send(500, { ok: false });
|
||||
if (model === "refuse-generate") return send(200, { ok: false });
|
||||
if (model === "drop-generate") return request.socket.destroy();
|
||||
if (model === "slow-generate") return;
|
||||
if (model === "slow-generate" || model === "slow-to-stop") return;
|
||||
if (model === "writer") {
|
||||
setInterval(() => appendFileSync(`${payload.output}.writes`, `${Date.now()}\n`), 50);
|
||||
return;
|
||||
}
|
||||
if (model === "encoder") {
|
||||
// an encoder child of the real worker: a grandchild in the worker's group, then a request that never answers
|
||||
const encoder = spawn(process.execPath, ["-e", "process.on('SIGTERM', () => {}); setInterval(() => {}, 1000)"], { stdio: "ignore" });
|
||||
appendFileSync(spawnLog, `grandchild ${encoder.pid}\n`);
|
||||
spawn(process.execPath, ["-e", `process.on('SIGTERM', () => {}); require('fs').appendFileSync(${JSON.stringify(spawnLog)}, 'grandchild ' + process.pid + '\\n'); setInterval(() => {}, 1000)`], { stdio: "ignore" });
|
||||
return;
|
||||
}
|
||||
send(200, {
|
||||
|
|
|
|||
145
scripts/idunn-entry.mjs
Normal file
145
scripts/idunn-entry.mjs
Normal file
|
|
@ -0,0 +1,145 @@
|
|||
#!/usr/bin/env node
|
||||
// The Idunn entry: the process Idunn's unit runs for target "vili". Idunn owns Vili's lifecycle and hands it one
|
||||
// writer authority, the process write lease. This file turns Idunn's environment into the daemon's own
|
||||
// configuration and into the runtime port serve() takes; it adds no routes and decides nothing the daemon decides.
|
||||
//
|
||||
// node vili/scripts/idunn-entry.mjs --state-root <absolute> --kimodo-commit <40 hex>
|
||||
//
|
||||
// Nothing from the bundle, the activation or the identity reaches the worker: the daemon's childEnv forwards only
|
||||
// PATH, HOME and LANG besides its own VILI_* settings.
|
||||
import { createHash } from "node:crypto";
|
||||
import { readFileSync } from "node:fs";
|
||||
import path from "node:path";
|
||||
import { setTimeout as delay } from "node:timers/promises";
|
||||
import { fileURLToPath, pathToFileURL } from "node:url";
|
||||
import { decode, encode } from "@msgpack/msgpack";
|
||||
import { encodeCultNetMessageForWire, parseCultNetMessage } from "cultnet-ts";
|
||||
import { createIdunnRuntimeSigner, loadIdunnRuntimeAuthorityFromEnvironment } from "cultnet-ts/idunn-runtime";
|
||||
|
||||
const HEALTH_CONTRACT = "gamecult.runtime_presence_health.v2";
|
||||
const LEASE_POLL_MS = 5000;
|
||||
const here = path.dirname(fileURLToPath(import.meta.url));
|
||||
|
||||
// The runtime port serve() takes, over an Idunn runtime signer. Exported so the lease gate is tested against the
|
||||
// code that ships, with a signer whose lease is a file (scripts/fixtures/fake-idunn-entry.mjs).
|
||||
export function idunnRuntime(signer, { pollMs = LEASE_POLL_MS } = {}) {
|
||||
let held = false;
|
||||
let lost = false;
|
||||
let onLost = () => {};
|
||||
|
||||
// The lease is read at the moment of every question, not once per tick: a loss is seen by the first gate to ask.
|
||||
const check = () => {
|
||||
if (!held) return false;
|
||||
try {
|
||||
signer.assertWriteLease();
|
||||
return true;
|
||||
} catch {
|
||||
held = false;
|
||||
lost = true;
|
||||
onLost();
|
||||
return false;
|
||||
}
|
||||
};
|
||||
|
||||
return {
|
||||
async acquire() {
|
||||
for (;;) {
|
||||
try {
|
||||
signer.assertWriteLease();
|
||||
held = true;
|
||||
return;
|
||||
} catch {
|
||||
// Idunn has not leased this incarnation yet
|
||||
}
|
||||
await delay(pollMs);
|
||||
}
|
||||
},
|
||||
held: check,
|
||||
refusal: () => (lost ? "write-lease-lost" : "write-lease-pending"),
|
||||
watch(handler) {
|
||||
onLost = handler;
|
||||
check();
|
||||
},
|
||||
recordWorker() {},
|
||||
forgetWorker() {},
|
||||
answerSnapshot(bytes) {
|
||||
let request;
|
||||
try {
|
||||
request = parseCultNetMessage(decode(bytes), "cultnet.schema.v0");
|
||||
if (request.schemaVersion !== "cultnet.snapshot_request.v0") throw new Error("not a snapshot request");
|
||||
} catch {
|
||||
throw Object.assign(new Error("invalid snapshot request"), { code: "invalid-snapshot-request" });
|
||||
}
|
||||
const document = signer.answerRouteObservation(request);
|
||||
const response = {
|
||||
schemaVersion: "cultnet.snapshot_response_raw.v0",
|
||||
messageId: request.messageId,
|
||||
documents: [document],
|
||||
};
|
||||
return Buffer.from(encode(encodeCultNetMessageForWire(response, "cultnet.schema.v0")));
|
||||
},
|
||||
reportHealth: (state) => signer.reportHealth(state),
|
||||
release() {},
|
||||
};
|
||||
}
|
||||
|
||||
function fail(reason) {
|
||||
console.error(`vili idunn entry: ${reason}`);
|
||||
process.exit(2);
|
||||
}
|
||||
|
||||
// host:port, with an optional http:// scheme, as Idunn writes the candidate bind.
|
||||
function splitBind(bind) {
|
||||
const match = /^(?:http:\/\/)?([^:/\s]+):(\d{1,5})\/?$/u.exec(bind ?? "");
|
||||
if (!match || Number(match[2]) > 65535) fail("GAMECULT_IDUNN_CANDIDATE_BIND is not host:port");
|
||||
return { host: match[1], port: match[2] };
|
||||
}
|
||||
|
||||
function parseEntryArgs(argv) {
|
||||
if (argv.length !== 4 || argv[0] !== "--state-root" || argv[2] !== "--kimodo-commit") {
|
||||
fail("arguments must be exactly --state-root <absolute> --kimodo-commit <40 hex>");
|
||||
}
|
||||
if (!path.isAbsolute(argv[1])) fail("--state-root must be absolute");
|
||||
if (!/^[0-9a-f]{40}$/u.test(argv[3])) fail("--kimodo-commit must be 40 lowercase hex characters");
|
||||
}
|
||||
|
||||
// What the daemon reads from its environment, derived from Idunn's bind and the sealed release's own layout:
|
||||
// the interpreter sits beside the vili tree, and the environment's identity is the hash of the two lock files.
|
||||
export function daemonEnvironment(bind, entryDir) {
|
||||
const { host, port } = splitBind(bind);
|
||||
const locks = ["requirements-build.lock", "requirements.lock"].map((name) =>
|
||||
readFileSync(path.resolve(entryDir, "..", "deployment", "idunn", name)));
|
||||
return {
|
||||
// Idunn writes the bind as host:port; cultnet-ts compares it to the Expected route's endpoint, which has the scheme.
|
||||
GAMECULT_IDUNN_CANDIDATE_BIND: `http://${host}:${port}`,
|
||||
VILI_DAEMON_HOST: host,
|
||||
VILI_DAEMON_PORT: port,
|
||||
VILI_PYTHON: path.resolve(entryDir, "..", "..", "python", "bin", "python3.12"),
|
||||
VILI_PYTHON_ENV: `locks-sha256:${createHash("sha256").update(Buffer.concat(locks)).digest("hex")}`,
|
||||
};
|
||||
}
|
||||
|
||||
async function main() {
|
||||
parseEntryArgs(process.argv.slice(2));
|
||||
Object.assign(process.env, daemonEnvironment(process.env.GAMECULT_IDUNN_CANDIDATE_BIND, here));
|
||||
|
||||
const authority = loadIdunnRuntimeAuthorityFromEnvironment("vili", HEALTH_CONTRACT);
|
||||
const signer = createIdunnRuntimeSigner({
|
||||
authority,
|
||||
capabilities: [{ capability: "vili.motion.api", schema: "http.v1", compatibility: "v1", capacity: 1 }],
|
||||
});
|
||||
if (!signer.requiresWriteLease) fail("Idunn declares no write lease for this runtime");
|
||||
|
||||
const { serve } = await import("./vili-daemon.mjs");
|
||||
await serve(idunnRuntime(signer));
|
||||
}
|
||||
|
||||
if (process.argv[1] && import.meta.url === pathToFileURL(process.argv[1]).href) {
|
||||
main().catch((error) => {
|
||||
// The cause, from the failing code: its error code, or the fixed text of a plain Error. An fs error's message
|
||||
// carries the path, so a coded error is named by its code alone.
|
||||
const cause = error?.code ?? String(error?.message ?? "").slice(0, 200);
|
||||
console.error(`vili idunn entry failed: ${error?.name || "Error"} ${cause}`);
|
||||
process.exit(1);
|
||||
});
|
||||
}
|
||||
|
|
@ -3,6 +3,7 @@ import argparse
|
|||
import json
|
||||
import os
|
||||
import shutil
|
||||
import sys
|
||||
import time
|
||||
import traceback
|
||||
from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer
|
||||
|
|
@ -63,7 +64,7 @@ model, resolved_model = load_model(
|
|||
text_encoder=text_encoder,
|
||||
)
|
||||
model_revision = text_encoding.read_snapshot_revision(HF_HOME, MODEL_NAMES[resolved_model])
|
||||
worker_identity = {"kimodoCommit": kimodo_commit, "pythonEnv": python_env, "modelRevision": model_revision, "threads": threads}
|
||||
worker_identity = {"kimodoCommit": kimodo_commit, "pythonEnv": python_env, "modelRevision": model_revision, "threads": threads, "python": sys.version}
|
||||
model_load_seconds = time.time() - model_load_started_at
|
||||
model_info = get_model_info(resolved_model)
|
||||
display_model = model_info.display_name if model_info else resolved_model
|
||||
|
|
|
|||
|
|
@ -40,6 +40,20 @@ def load_encoder(hf_home, mntp_id, supervised_id, dtype):
|
|||
return LLM2Vec(model=model, tokenizer=tokenizer)
|
||||
|
||||
|
||||
def encode_texts(model, texts):
|
||||
"""Each text wrapped in the chat header and encoded alone: (vector bytes, seconds) lists."""
|
||||
vectors, encode_seconds = [], []
|
||||
for text in texts:
|
||||
began = time.perf_counter()
|
||||
features = model.tokenize([chat_header(text)])
|
||||
with torch.no_grad():
|
||||
vector = model.forward(features).detach().cpu().to(torch.float32)[0]
|
||||
assert vector.shape == (EMBEDDING_DIM,)
|
||||
vectors.append(vector.numpy().tobytes())
|
||||
encode_seconds.append(time.perf_counter() - began)
|
||||
return vectors, encode_seconds
|
||||
|
||||
|
||||
def main() -> int:
|
||||
apply_thread_cap(os.environ)
|
||||
texts = json.load(sys.stdin)["texts"]
|
||||
|
|
@ -53,15 +67,7 @@ def main() -> int:
|
|||
p.requires_grad = False
|
||||
load_seconds = time.perf_counter() - started
|
||||
|
||||
vectors, encode_seconds = [], []
|
||||
for text in texts:
|
||||
began = time.perf_counter()
|
||||
features = model.tokenize([chat_header(text)])
|
||||
with torch.no_grad():
|
||||
vector = model.forward(features).detach().cpu().to(torch.float32)[0]
|
||||
assert vector.shape == (EMBEDDING_DIM,)
|
||||
vectors.append(vector.numpy().tobytes())
|
||||
encode_seconds.append(time.perf_counter() - began)
|
||||
vectors, encode_seconds = encode_texts(model, texts)
|
||||
|
||||
out.write_bytes(msgpack.packb({
|
||||
"vectors": vectors,
|
||||
|
|
@ -75,4 +81,5 @@ def main() -> int:
|
|||
return 0
|
||||
|
||||
|
||||
sys.exit(main())
|
||||
if __name__ == "__main__":
|
||||
sys.exit(main())
|
||||
|
|
|
|||
|
|
@ -2,16 +2,19 @@
|
|||
// Runs the real daemon against scripts/fixtures/stub-worker.mjs as its --python/--worker-entry.
|
||||
import assert from "node:assert/strict";
|
||||
import { spawn } from "node:child_process";
|
||||
import { existsSync, mkdtempSync, readFileSync, rmSync, writeFileSync } from "node:fs";
|
||||
import { existsSync, mkdirSync, mkdtempSync, readFileSync, readdirSync, rmSync, statSync, symlinkSync, writeFileSync } from "node:fs";
|
||||
import http from "node:http";
|
||||
import net from "node:net";
|
||||
import os from "node:os";
|
||||
import path from "node:path";
|
||||
import test from "node:test";
|
||||
import { fileURLToPath } from "node:url";
|
||||
import { decode, encode } from "@msgpack/msgpack";
|
||||
|
||||
const here = path.dirname(fileURLToPath(import.meta.url));
|
||||
const daemonPath = path.join(here, "vili-daemon.mjs");
|
||||
const stubPath = path.join(here, "fixtures", "stub-worker.mjs");
|
||||
const fakeIdunnEntry = path.join(here, "fixtures", "fake-idunn-entry.mjs");
|
||||
const THREADS = 5;
|
||||
|
||||
async function until(condition, what, seconds = 15) {
|
||||
|
|
@ -40,7 +43,7 @@ function isAlive(pid) {
|
|||
|
||||
async function startDaemon({
|
||||
omit = [], viaEnv = false, threads = String(THREADS), python = process.execPath, nested = false, host = "127.0.0.1",
|
||||
relativeModels = false, extraArgs = [], root: givenRoot = null,
|
||||
relativeModels = false, extraArgs = [], root: givenRoot = null, entry = daemonPath, extraEnv = {}, hold = entry === daemonPath,
|
||||
} = {}) {
|
||||
const root = givenRoot ?? mkdtempSync(path.join(os.tmpdir(), "vili-test-"));
|
||||
const stateRoot = nested ? path.join(root, "a", "b", "state") : root;
|
||||
|
|
@ -57,9 +60,9 @@ async function startDaemon({
|
|||
};
|
||||
if (threads !== null) settings.threads = threads;
|
||||
for (const name of omit) delete settings[name];
|
||||
const env = { ...process.env, LEAK_CHECK: "leaked", LANG: "xx_TEST.UTF-8" };
|
||||
const env = { ...process.env, LEAK_CHECK: "leaked", LANG: "xx_TEST.UTF-8", ...extraEnv };
|
||||
for (const name of Object.keys(env)) if (name.startsWith("VILI_")) delete env[name];
|
||||
const argv = [daemonPath, ...extraArgs];
|
||||
const argv = [entry, ...extraArgs];
|
||||
for (const [name, value] of Object.entries(settings)) {
|
||||
if (viaEnv) env[`VILI_${{ host: "DAEMON_HOST", port: "DAEMON_PORT", "state-root": "STATE_ROOT", "worker-entry": "WORKER_ENTRY" }[name] ?? name.toUpperCase().replace("-", "_")}`] = value;
|
||||
else argv.push(`--${name}`, value);
|
||||
|
|
@ -71,25 +74,34 @@ async function startDaemon({
|
|||
proc.stderr.on("data", (chunk) => { errors += chunk; });
|
||||
const listening = await until(() => /listening on http:\/\/([^:]+):(\d+)/.exec(output), "the daemon to listen");
|
||||
const port = listening[2];
|
||||
// the hand-run daemon takes its lock after it listens; a test that wants a served state root waits for it
|
||||
if (hold) await until(() => /holds the state root/.test(output), "the daemon to hold the state root");
|
||||
const spawnLog = () => {
|
||||
const log = path.join(stateRoot, "embeddings", "stub-spawns.log");
|
||||
return existsSync(log) ? readFileSync(log, "utf8").split("\n").filter(Boolean) : [];
|
||||
};
|
||||
return {
|
||||
root,
|
||||
stateRoot,
|
||||
modelsRoot,
|
||||
proc,
|
||||
boundHost: listening[1],
|
||||
port: Number(port),
|
||||
stderr: () => errors,
|
||||
stdout: () => output,
|
||||
spawns: () => spawnLog().filter((line) => /^\d+$/.test(line)).map(Number),
|
||||
grandchildren: () => spawnLog().filter((line) => line.startsWith("grandchild ")).map((line) => Number(line.split(" ")[1])),
|
||||
async stop() {
|
||||
if (proc.exitCode === null && proc.signalCode === null) {
|
||||
proc.kill("SIGTERM");
|
||||
await until(() => proc.exitCode !== null || proc.signalCode !== null, "the daemon to exit", 30);
|
||||
try {
|
||||
if (proc.exitCode === null && proc.signalCode === null) {
|
||||
proc.kill("SIGTERM");
|
||||
await until(() => proc.exitCode !== null || proc.signalCode !== null, "the daemon to exit", 30);
|
||||
}
|
||||
} finally {
|
||||
// a daemon that did not stop on SIGTERM must not outlive the test and hold the run open
|
||||
if (proc.exitCode === null && proc.signalCode === null) proc.kill("SIGKILL");
|
||||
rmSync(root, { recursive: true, force: true });
|
||||
}
|
||||
rmSync(root, { recursive: true, force: true });
|
||||
},
|
||||
};
|
||||
}
|
||||
|
|
@ -429,17 +441,41 @@ test("a live worker outside the pid file's group is left alone", async () => {
|
|||
try {
|
||||
assert.equal((await generate(first)).status, 200);
|
||||
const [worker] = first.spawns();
|
||||
first.proc.kill("SIGKILL");
|
||||
await until(() => first.proc.exitCode !== null || first.proc.signalCode !== null, "the first daemon to die");
|
||||
writeFileSync(path.join(root, "worker.pid"), `${bystander.pid}\n`);
|
||||
const second = await startDaemon({ root });
|
||||
assert.ok(isAlive(worker) && isAlive(bystander.pid), "only the group the pid file names, and only if it is ours, is killed");
|
||||
second.proc.kill("SIGTERM");
|
||||
await until(() => second.proc.exitCode !== null, "the second daemon to exit", 30);
|
||||
process.kill(-worker, "SIGKILL");
|
||||
} finally {
|
||||
bystander.kill("SIGKILL");
|
||||
await first.stop();
|
||||
}
|
||||
});
|
||||
|
||||
// The marker that proves a group is ours is this state root's own embeddings directory. A live worker of another
|
||||
// state root, named by a stale pid file, carries another root's marker and is left alone.
|
||||
test("a pid file naming another state root's live worker kills nothing", async () => {
|
||||
const other = await startDaemon();
|
||||
const root = mkdtempSync(path.join(os.tmpdir(), "vili-test-"));
|
||||
try {
|
||||
assert.equal((await generate(other)).status, 200);
|
||||
const [foreign] = other.spawns();
|
||||
writeFileSync(path.join(root, "worker.pid"), `${foreign}\n`);
|
||||
const daemon = await startDaemon({ root });
|
||||
try {
|
||||
assert.ok(isAlive(foreign), "a worker carrying another root's marker must survive the reclaim");
|
||||
assert.equal(existsSync(path.join(root, "worker.pid")), false, "the stale pid file is still forgotten");
|
||||
} finally {
|
||||
await daemon.stop();
|
||||
}
|
||||
} finally {
|
||||
await other.stop();
|
||||
}
|
||||
});
|
||||
|
||||
test("a pid file naming a process group that is not the worker's kills nothing", async () => {
|
||||
const root = mkdtempSync(path.join(os.tmpdir(), "vili-test-"));
|
||||
const bystander = spawn(process.execPath, ["-e", "setInterval(() => {}, 1000)"], { detached: true, stdio: "ignore" });
|
||||
|
|
@ -655,22 +691,6 @@ test("the typed service state is refreshed while the daemon runs", async () => {
|
|||
}
|
||||
});
|
||||
|
||||
test("--health asks a running daemon, and fails when there is none", async () => {
|
||||
const daemon = await startDaemon();
|
||||
const check = (port) => new Promise((resolve) => {
|
||||
const proc = spawn(process.execPath, [daemonPath, "--health", "--host", "127.0.0.1", "--port", String(port)], { stdio: "ignore" });
|
||||
proc.on("exit", resolve);
|
||||
});
|
||||
try {
|
||||
assert.equal(await check(daemon.port), 0);
|
||||
const gone = daemon.port;
|
||||
await daemon.stop();
|
||||
assert.equal(await check(gone), 1);
|
||||
} finally {
|
||||
await daemon.stop();
|
||||
}
|
||||
});
|
||||
|
||||
test("a thread count outside 1 to 16 stops the daemon without echoing it", async () => {
|
||||
for (const threads of ["0", "17", "99", "100", "8x", "x8", "-1", "1.5"]) {
|
||||
const proc = spawn(process.execPath, [daemonPath, "--threads", threads], { stdio: ["ignore", "pipe", "pipe"] });
|
||||
|
|
@ -683,3 +703,589 @@ test("a thread count outside 1 to 16 stops the daemon without echoing it", async
|
|||
assert.ok(!output.replace("from 1 to 16", "").includes(threads), `the refusal repeated ${threads}`);
|
||||
}
|
||||
});
|
||||
|
||||
const msgpackPost = (daemon, body, headers = {}) =>
|
||||
new Promise((resolve, reject) => {
|
||||
const req = http.request({ host: "127.0.0.1", port: daemon.port, method: "POST", path: "/cultnet/snapshot", headers: { "content-type": "application/msgpack", ...headers } }, (res) => {
|
||||
const chunks = [];
|
||||
res.on("data", (chunk) => chunks.push(chunk));
|
||||
res.on("end", () => resolve({ status: res.statusCode, headers: res.headers, raw: Buffer.concat(chunks) }));
|
||||
});
|
||||
req.on("error", reject);
|
||||
req.end(body);
|
||||
});
|
||||
const SECRET = "canary-9f3c-never-echoed";
|
||||
const snapshotRequest = (messageId = "m1") => Buffer.from(encode({ schemaVersion: "cultnet.snapshot_request.v0", messageId }));
|
||||
|
||||
// A second daemon on a state root a live daemon holds must refuse, and must not touch the first one's worker.
|
||||
test("a second daemon on a live state root exits 2 and kills nothing", async () => {
|
||||
const first = await startDaemon();
|
||||
let second = null;
|
||||
try {
|
||||
assert.equal((await generate(first)).status, 200);
|
||||
const [worker] = first.spawns();
|
||||
assert.ok(isAlive(worker));
|
||||
second = await startDaemon({ root: first.root, hold: false });
|
||||
await until(() => second.proc.exitCode !== null, "the second daemon to refuse", 20);
|
||||
assert.equal(second.proc.exitCode, 2);
|
||||
assert.match(second.stderr(), /held by a running daemon/);
|
||||
assert.ok(isAlive(worker), "the first daemon's worker must survive the refusal");
|
||||
assert.equal((await generate(first)).status, 200);
|
||||
assert.equal(aliveWorkers(first).length, 1);
|
||||
} finally {
|
||||
if (second && second.proc.exitCode === null && second.proc.signalCode === null) second.proc.kill("SIGKILL"); // a daemon that failed to refuse must not outlive the test
|
||||
await first.stop();
|
||||
}
|
||||
});
|
||||
|
||||
// Two daemons start together over the lock a crashed daemon left. Exactly one owns the root, every time: the
|
||||
// other exits 2, and one generate leaves one worker on the root.
|
||||
for (let trial = 1; trial <= 40; trial += 1) {
|
||||
test(`two daemons racing a crashed daemon's lock leave one owner and one worker (trial ${trial})`, async () => {
|
||||
const root = mkdtempSync(path.join(os.tmpdir(), "vili-test-"));
|
||||
const crashed = await startDaemon({ root });
|
||||
crashed.proc.kill("SIGKILL");
|
||||
await until(() => crashed.proc.exitCode !== null || crashed.proc.signalCode !== null, "the first daemon to die");
|
||||
const racers = await Promise.all([startDaemon({ root, hold: false }), startDaemon({ root, hold: false })]);
|
||||
try {
|
||||
await until(() => racers.every((racer) => racer.proc.exitCode !== null || /holds the state root/.test(racer.stdout())), "both racers to decide", 20);
|
||||
await new Promise((resolve) => setTimeout(resolve, 300));
|
||||
const owners = racers.filter((racer) => racer.proc.exitCode === null);
|
||||
assert.equal(owners.length, 1, "exactly one daemon stays up");
|
||||
assert.match(owners[0].stdout(), /holds the state root/);
|
||||
const loser = racers.find((racer) => racer !== owners[0]);
|
||||
assert.equal(loser.proc.exitCode, 2);
|
||||
assert.match(loser.stderr(), /held by a running daemon/);
|
||||
for (const racer of owners) assert.equal((await generate(racer)).status, 200);
|
||||
assert.equal(aliveWorkers(owners[0]).length, 1, "one worker on the root");
|
||||
} finally {
|
||||
for (const racer of racers) if (racer.proc.exitCode === null && racer.proc.signalCode === null) racer.proc.kill("SIGKILL");
|
||||
for (const pid of racers[0].spawns()) { try { process.kill(-pid, "SIGKILL"); } catch { /* gone */ } }
|
||||
rmSync(root, { recursive: true, force: true });
|
||||
}
|
||||
});
|
||||
}
|
||||
|
||||
// canonicalPath's fallback: a state root that does not exist yet is resolved through its longest existing prefix, so a
|
||||
// daemon naming it through a symlink and one naming it directly still contend for one lock.
|
||||
for (let trial = 1; trial <= 10; trial += 1) {
|
||||
test(`two daemons naming a not-yet-existing root, one through a symlink, leave one owner (trial ${trial})`, async () => {
|
||||
const real = mkdtempSync(path.join(os.tmpdir(), "vili-test-"));
|
||||
const link = `${real}-link`;
|
||||
symlinkSync(real, link);
|
||||
const racers = await Promise.all([startDaemon({ root: real, nested: true, hold: false }), startDaemon({ root: link, nested: true, hold: false })]);
|
||||
try {
|
||||
await until(() => racers.every((racer) => racer.proc.exitCode !== null || /holds the state root/.test(racer.stdout())), "both racers to decide", 20);
|
||||
await new Promise((resolve) => setTimeout(resolve, 300));
|
||||
const owners = racers.filter((racer) => racer.proc.exitCode === null);
|
||||
assert.equal(owners.length, 1, "exactly one daemon stays up");
|
||||
const loser = racers.find((racer) => racer !== owners[0]);
|
||||
assert.equal(loser.proc.exitCode, 2);
|
||||
assert.match(loser.stderr(), /held by a running daemon/);
|
||||
} finally {
|
||||
for (const racer of racers) if (racer.proc.exitCode === null && racer.proc.signalCode === null) racer.proc.kill("SIGKILL");
|
||||
rmSync(link, { force: true });
|
||||
rmSync(real, { recursive: true, force: true });
|
||||
}
|
||||
});
|
||||
}
|
||||
const idunnDaemon = async ({ leased = false, extraEnv = {}, loseWhenExists = null } = {}) => {
|
||||
const root = mkdtempSync(path.join(os.tmpdir(), "vili-test-"));
|
||||
const lease = path.join(root, "lease");
|
||||
const healthLog = path.join(root, "health.log");
|
||||
if (leased) writeFileSync(lease, "");
|
||||
const daemon = await startDaemon({
|
||||
root, nested: true, entry: fakeIdunnEntry, extraEnv: {
|
||||
FAKE_IDUNN_LEASE: lease, FAKE_IDUNN_HEALTH_LOG: healthLog, ...extraEnv,
|
||||
...(loseWhenExists ? { FAKE_IDUNN_STATE_ROOT: path.join(root, "a", "b", "state"), FAKE_IDUNN_LOSE_WHEN_EXISTS: loseWhenExists } : {}),
|
||||
},
|
||||
});
|
||||
return Object.assign(daemon, { lease, health: () => (existsSync(healthLog) ? readFileSync(healthLog, "utf8").split("\n").filter(Boolean) : []) });
|
||||
};
|
||||
|
||||
test("before the lease nothing is written, spawned or accepted", async () => {
|
||||
const daemon = await idunnDaemon();
|
||||
try {
|
||||
const answer = await generate(daemon);
|
||||
assert.equal(answer.status, 503);
|
||||
assert.equal(answer.body.code, "write-lease-pending");
|
||||
assert.deepEqual(daemon.spawns(), []);
|
||||
assert.equal(existsSync(daemon.stateRoot), false, "the state root must not exist before the lease");
|
||||
assert.deepEqual(daemon.health(), []);
|
||||
} finally {
|
||||
await daemon.stop();
|
||||
}
|
||||
});
|
||||
|
||||
test("with the lease the daemon writes and serves, and no pid file exists", async () => {
|
||||
const daemon = await idunnDaemon();
|
||||
try {
|
||||
writeFileSync(daemon.lease, "");
|
||||
await until(() => existsSync(path.join(daemon.stateRoot, "vili.service.cc")), "the service store", 10);
|
||||
const answer = await generate(daemon);
|
||||
assert.equal(answer.status, 200);
|
||||
assert.equal(existsSync(path.join(daemon.stateRoot, "worker.pid")), false);
|
||||
assert.equal(existsSync(path.join(daemon.stateRoot, "daemon.lock")), false);
|
||||
assert.deepEqual(daemon.health(), ["active"]);
|
||||
} finally {
|
||||
await daemon.stop();
|
||||
}
|
||||
});
|
||||
|
||||
test("losing the lease kills the worker tree, refuses, and writes nothing more", async () => {
|
||||
const daemon = await idunnDaemon({ leased: true });
|
||||
try {
|
||||
await until(() => existsSync(path.join(daemon.stateRoot, "vili.service.cc")), "the service store", 10);
|
||||
assert.equal((await generate(daemon, { prompt: "x", model: "stubborn" })).status, 200);
|
||||
const [worker] = daemon.spawns();
|
||||
const [grandchild] = await until(() => { const found = daemon.grandchildren(); return found.length ? found : null; }, "the armed grandchild");
|
||||
assert.ok(isAlive(worker) && isAlive(grandchild));
|
||||
rmSync(daemon.lease);
|
||||
await until(() => !isAlive(worker) && !isAlive(grandchild), "the worker tree to die", 20);
|
||||
const answer = await generate(daemon);
|
||||
assert.equal(answer.status, 503);
|
||||
assert.equal(answer.body.code, "write-lease-lost");
|
||||
await until(() => daemon.health().join() === "active,degraded", "the health to say degraded");
|
||||
const store = path.join(daemon.stateRoot, "vili.service.cc");
|
||||
const before = statSync(store).mtimeMs;
|
||||
const jobs = readdirSync(path.join(daemon.stateRoot, "jobs")).length;
|
||||
await new Promise((resolve) => setTimeout(resolve, 6500));
|
||||
assert.equal(statSync(store).mtimeMs, before);
|
||||
assert.equal(readdirSync(path.join(daemon.stateRoot, "jobs")).length, jobs);
|
||||
} finally {
|
||||
await daemon.stop();
|
||||
}
|
||||
});
|
||||
|
||||
// A generate whose headers are in and whose body is still arriving when the lease goes. The lease is gone, so
|
||||
// nothing is created and nothing is spawned when the body completes, whether the daemon has noticed the loss yet
|
||||
// (its 5 s tick) or not.
|
||||
for (const noticed of [false, true]) {
|
||||
test(`a request in flight when the lease is lost writes and spawns nothing (loss ${noticed ? "noticed" : "not yet noticed"})`, async () => {
|
||||
const daemon = await idunnDaemon({ leased: true });
|
||||
try {
|
||||
await until(() => existsSync(path.join(daemon.stateRoot, "vili.service.cc")), "the service store", 10);
|
||||
const body = JSON.stringify({ prompt: "x", duration: 2 });
|
||||
const socket = net.connect(daemon.port, "127.0.0.1");
|
||||
let raw = "";
|
||||
socket.on("data", (chunk) => { raw += chunk; });
|
||||
const closed = new Promise((resolve) => socket.on("close", resolve));
|
||||
socket.write(`POST /motion/generate HTTP/1.1\r\nhost: vili\r\ncontent-type: application/json\r\ncontent-length: ${body.length}\r\nconnection: close\r\n\r\n${body.slice(0, 5)}`);
|
||||
await new Promise((resolve) => setTimeout(resolve, 200));
|
||||
rmSync(daemon.lease);
|
||||
if (noticed) await until(() => daemon.health().join() === "active,degraded", "the daemon to notice the loss", 15);
|
||||
socket.write(body.slice(5));
|
||||
await Promise.race([closed, new Promise((resolve) => setTimeout(resolve, 15000))]);
|
||||
socket.destroy();
|
||||
assert.match(raw, /^HTTP\/1\.1 503 /);
|
||||
assert.match(raw, /write-lease-lost/);
|
||||
assert.deepEqual(daemon.spawns(), [], "no worker is spawned after the loss");
|
||||
const artifacts = path.join(daemon.stateRoot, "artifacts");
|
||||
assert.deepEqual(existsSync(artifacts) ? readdirSync(artifacts) : [], [], "no artifact directory is created");
|
||||
assert.deepEqual(readdirSync(path.join(daemon.stateRoot, "jobs")), []);
|
||||
} finally {
|
||||
await daemon.stop();
|
||||
}
|
||||
});
|
||||
}
|
||||
|
||||
// The lease goes between two writes of one request. Each case names the path whose appearance takes the lease, and
|
||||
// what must not exist afterwards: every write and the spawn are gated where they happen, not by the request's
|
||||
// earlier checks.
|
||||
const losses = [
|
||||
{ when: "artifacts/*", absent: ["artifacts/*/meta.json", "logs", "embeddings"] },
|
||||
{ when: "artifacts/*", payload: { prompt: "a person waves", duration: 2, constraints: [{ type: "root2d" }] }, absent: ["artifacts/*/constraints.json", "artifacts/*/meta.json"] },
|
||||
{ when: "artifacts/*/meta.json", absent: ["logs", "embeddings"] },
|
||||
{ when: "logs", absent: ["embeddings"] },
|
||||
{ when: "embeddings", absent: ["logs/worker.log"] },
|
||||
{ when: "embeddings/stub-healthy", absent: ["artifacts/*/request.json", "jobs/*"] },
|
||||
];
|
||||
for (const { when, absent, payload } of losses) {
|
||||
test(`a request is stopped at its next write when the lease goes at ${when}${payload ? " (constraints in the body)" : ""}`, async () => {
|
||||
const daemon = await idunnDaemon({ leased: true, loseWhenExists: when });
|
||||
try {
|
||||
await until(() => existsSync(path.join(daemon.stateRoot, "vili.service.cc")), "the service store", 10);
|
||||
const answer = await generate(daemon, payload);
|
||||
assert.equal(answer.status, 503);
|
||||
assert.equal(answer.body.code, "write-lease-lost");
|
||||
for (const gone of absent) {
|
||||
const parts = gone.split("/");
|
||||
const found = (base, rest) => (rest.length === 0 ? existsSync(base)
|
||||
: rest[0] === "*" ? existsSync(base) && readdirSync(base).some((entry) => found(path.join(base, entry), rest.slice(1)))
|
||||
: found(path.join(base, rest[0]), rest.slice(1)));
|
||||
assert.equal(found(daemon.stateRoot, parts), false, `${gone} must not exist`);
|
||||
}
|
||||
if (when === "embeddings") assert.deepEqual(daemon.spawns(), []);
|
||||
} finally {
|
||||
await daemon.stop();
|
||||
}
|
||||
});
|
||||
}
|
||||
|
||||
// The daemon's own copy of a worker's stdout into worker.log is a write like any other.
|
||||
test("a worker's output line is not logged once the lease is gone", async () => {
|
||||
const daemon = await idunnDaemon({ leased: true, loseWhenExists: "embeddings/stub-spawns.log" });
|
||||
try {
|
||||
await until(() => existsSync(path.join(daemon.stateRoot, "vili.service.cc")), "the service store", 10);
|
||||
const answer = await generate(daemon);
|
||||
assert.equal(answer.status, 503);
|
||||
assert.equal(answer.body.code, "write-lease-lost");
|
||||
const log = readFileSync(path.join(daemon.stateRoot, "logs", "worker.log"), "utf8");
|
||||
assert.ok(!log.includes("stub-stdout-line"), "the daemon must not copy the worker's stdout after the loss");
|
||||
} finally {
|
||||
await daemon.stop();
|
||||
}
|
||||
});
|
||||
|
||||
// The lease goes while the first persist is under way: the daemon survives it, stays degraded, and the rest of the
|
||||
// startup writes are not made.
|
||||
test("a lease lost while the state root is opened leaves the daemon up, degraded and writing nothing more", async () => {
|
||||
const daemon = await idunnDaemon({ leased: true, loseWhenExists: "vili.service.cc" });
|
||||
try {
|
||||
await until(() => daemon.health().includes("degraded") || daemon.proc.exitCode !== null, "the loss to be seen", 15);
|
||||
assert.equal(daemon.proc.exitCode, null, "the daemon must not die of it");
|
||||
assert.deepEqual(daemon.health(), ["degraded"]);
|
||||
const store = readFileSync(path.join(daemon.stateRoot, "vili.service.cc"));
|
||||
assert.ok(store.includes("operator_state") && !store.includes("command_boundary"), "the second document is not written");
|
||||
const answer = await generate(daemon);
|
||||
assert.equal(answer.status, 503);
|
||||
assert.equal(answer.body.code, "write-lease-lost");
|
||||
} finally {
|
||||
await daemon.stop();
|
||||
}
|
||||
});
|
||||
|
||||
test("a request arriving right after the lease is lost writes and spawns nothing", async () => {
|
||||
const daemon = await idunnDaemon({ leased: true });
|
||||
try {
|
||||
await until(() => existsSync(path.join(daemon.stateRoot, "vili.service.cc")), "the service store", 10);
|
||||
rmSync(daemon.lease);
|
||||
const answer = await generate(daemon);
|
||||
assert.equal(answer.status, 503);
|
||||
assert.equal(answer.body.code, "write-lease-lost");
|
||||
assert.deepEqual(daemon.spawns(), []);
|
||||
const artifacts = path.join(daemon.stateRoot, "artifacts");
|
||||
assert.deepEqual(existsSync(artifacts) ? readdirSync(artifacts) : [], []);
|
||||
assert.deepEqual(readdirSync(path.join(daemon.stateRoot, "jobs")), []);
|
||||
} finally {
|
||||
await daemon.stop();
|
||||
}
|
||||
});
|
||||
|
||||
test("a worker respawned under the lease is killed when the lease is lost", async () => {
|
||||
const daemon = await idunnDaemon({ leased: true });
|
||||
try {
|
||||
await until(() => existsSync(path.join(daemon.stateRoot, "vili.service.cc")), "the service store", 10);
|
||||
assert.equal((await generate(daemon)).status, 200);
|
||||
const [first] = daemon.spawns();
|
||||
process.kill(-first, "SIGKILL");
|
||||
await until(async () => (await backendOf(daemon)).state === "stopped", "the daemon to see the worker gone");
|
||||
assert.equal((await generate(daemon)).status, 200);
|
||||
const second = daemon.spawns().find((pid) => pid !== first);
|
||||
assert.ok(second && isAlive(second));
|
||||
rmSync(daemon.lease);
|
||||
await until(() => !isAlive(second), "the respawned worker to die", 20);
|
||||
} finally {
|
||||
await daemon.stop();
|
||||
}
|
||||
});
|
||||
|
||||
// Idunn stops an incumbent (SIGTERM to the unit) before it revokes the lease. What Vili owns in that sequence: the
|
||||
// worker tree is dead when the process ends, it exits 0, and nothing is written from the signal on.
|
||||
test("on SIGTERM the daemon stops its worker tree, writes nothing more, and exits 0", async () => {
|
||||
const daemon = await idunnDaemon({ leased: true });
|
||||
try {
|
||||
await until(() => existsSync(path.join(daemon.stateRoot, "vili.service.cc")), "the service store", 10);
|
||||
assert.equal((await generate(daemon, { prompt: "x", model: "stubborn" })).status, 200);
|
||||
const [worker] = daemon.spawns();
|
||||
const [grandchild] = await until(() => { const found = daemon.grandchildren(); return found.length ? found : null; }, "the armed grandchild");
|
||||
const store = path.join(daemon.stateRoot, "vili.service.cc");
|
||||
const before = statSync(store).mtimeMs;
|
||||
const jobs = readdirSync(path.join(daemon.stateRoot, "jobs"));
|
||||
daemon.proc.kill("SIGTERM");
|
||||
await until(() => daemon.proc.exitCode !== null || daemon.proc.signalCode !== null, "the daemon to exit", 30);
|
||||
assert.equal(daemon.proc.exitCode, 0);
|
||||
assert.ok(!isAlive(worker) && !isAlive(grandchild), "the worker tree is dead when the daemon is gone");
|
||||
assert.equal(statSync(store).mtimeMs, before, "the 5 s surface tick must not write during the stop");
|
||||
assert.deepEqual(readdirSync(path.join(daemon.stateRoot, "jobs")), jobs);
|
||||
assert.ok(existsSync(daemon.lease), "the daemon never touches the lease Idunn revokes");
|
||||
} finally {
|
||||
await daemon.stop();
|
||||
}
|
||||
});
|
||||
|
||||
test("a job that finishes after the lease is lost is not recorded", async () => {
|
||||
const daemon = await idunnDaemon({ leased: true });
|
||||
try {
|
||||
await until(() => existsSync(path.join(daemon.stateRoot, "vili.service.cc")), "the service store", 10);
|
||||
const slow = generate(daemon, { prompt: "x", model: "slow-generate" });
|
||||
slow.catch(() => {}); // a failing assertion below must not leave this request pending
|
||||
await until(() => daemon.spawns().length, "the worker");
|
||||
await until(() => request(daemon.port, "GET", "/health").then((r) => r.body?.state?.backend?.state === "ready"), "the worker to be ready");
|
||||
rmSync(daemon.lease);
|
||||
const answer = await Promise.race([slow, new Promise((resolve) => setTimeout(() => resolve({ status: "no answer in 30 s" }), 30000))]);
|
||||
assert.equal(answer.status, 503);
|
||||
assert.equal(answer.body.code, "write-lease-lost");
|
||||
assert.deepEqual(readdirSync(path.join(daemon.stateRoot, "jobs")), []);
|
||||
} finally {
|
||||
await daemon.stop();
|
||||
}
|
||||
});
|
||||
|
||||
test("the route challenge is answered ahead of the cross-site refusal, and an Origin is still refused", async () => {
|
||||
const daemon = await idunnDaemon();
|
||||
try {
|
||||
// the challenge is how Idunn learns the incarnation is warming, so it is answered before the lease
|
||||
const answered = await msgpackPost(daemon, snapshotRequest("m-42"));
|
||||
assert.equal(answered.status, 200);
|
||||
assert.equal(answered.headers["content-type"], "application/msgpack");
|
||||
const message = decode(answered.raw);
|
||||
assert.equal(message.schemaVersion, "cultnet.snapshot_response_raw.v0");
|
||||
assert.equal(message.messageId, "m-42");
|
||||
assert.equal(message.documents.length, 1);
|
||||
assert.equal((await msgpackPost(daemon, snapshotRequest(), { origin: "https://evil.example" })).status, 403);
|
||||
assert.equal((await msgpackPost(daemon, Buffer.from("not msgpack at all"))).status, 400);
|
||||
assert.equal((await msgpackPost(daemon, Buffer.from(encode({ schemaVersion: "cultnet.snapshot_response_raw.v0", messageId: "x", documents: [] })))).status, 400);
|
||||
assert.equal((await msgpackPost(daemon, Buffer.alloc(70 * 1024))).status, 400);
|
||||
const text = await request(daemon.port, "POST", "/motion/generate", { body: "{}", headers: { "content-type": "text/plain" } });
|
||||
assert.equal(text.status, 403);
|
||||
} finally {
|
||||
await daemon.stop();
|
||||
}
|
||||
});
|
||||
|
||||
test("a signer that cannot answer gives 503 and echoes nothing", async () => {
|
||||
const daemon = await idunnDaemon({ extraEnv: { FAKE_IDUNN_ANSWER_THROWS: "1" } });
|
||||
try {
|
||||
const answered = await msgpackPost(daemon, snapshotRequest(SECRET));
|
||||
assert.equal(answered.status, 503);
|
||||
assert.ok(!answered.raw.toString("utf8").includes(SECRET));
|
||||
assert.ok(!daemon.stderr().includes(SECRET));
|
||||
} finally {
|
||||
await daemon.stop();
|
||||
}
|
||||
});
|
||||
|
||||
test("the hand-run daemon has no route challenge", async () => {
|
||||
const daemon = await startDaemon();
|
||||
try {
|
||||
assert.equal((await msgpackPost(daemon, snapshotRequest())).status, 404);
|
||||
assert.equal((await msgpackPost(daemon, snapshotRequest(), { origin: "https://evil.example" })).status, 404);
|
||||
} finally {
|
||||
await daemon.stop();
|
||||
}
|
||||
});
|
||||
|
||||
// ---- the Idunn entry's own rules: its argument contract, and what it derives for the daemon
|
||||
|
||||
const entryPath = path.join(here, "idunn-entry.mjs");
|
||||
const runEntry = (args, env = {}) => new Promise((resolve) => {
|
||||
const proc = spawn(process.execPath, [entryPath, ...args], { env: { PATH: process.env.PATH, ...env }, stdio: ["ignore", "pipe", "pipe"] });
|
||||
let output = "";
|
||||
proc.stdout.on("data", (chunk) => { output += chunk; });
|
||||
proc.stderr.on("data", (chunk) => { output += chunk; });
|
||||
proc.on("exit", (code) => resolve({ code, output }));
|
||||
});
|
||||
|
||||
test("the entry takes exactly --state-root <absolute> --kimodo-commit <40 hex>, and echoes none of a bad value", async () => {
|
||||
const commit = "6bb58488037dd65360ff0c5d1692b403a23309f7";
|
||||
const good = ["--state-root", "/var/lib/gamecult/vili", "--kimodo-commit", commit];
|
||||
const bads = [
|
||||
[],
|
||||
[...good, "--extra", "x"],
|
||||
["--kimodo-commit", commit, "--state-root", "/var/lib/gamecult/vili"],
|
||||
["--state-root", "canary-relative-path", "--kimodo-commit", commit],
|
||||
["--state-root", "/var/lib/gamecult/vili", "--kimodo-commit", "canary-not-a-commit"],
|
||||
["--state-root", "/var/lib/gamecult/vili", "--kimodo-commit", commit.toUpperCase()],
|
||||
];
|
||||
for (const args of bads) {
|
||||
const answer = await runEntry(args, { GAMECULT_IDUNN_CANDIDATE_BIND: "127.0.0.1:18861" });
|
||||
assert.equal(answer.code, 2, JSON.stringify(args));
|
||||
assert.ok(!answer.output.includes("canary"), `the refusal repeated a value of ${JSON.stringify(args)}`);
|
||||
}
|
||||
for (const badBind of ["canary-not-a-bind", "127.0.0.1:65536", "127.0.0.1:99999", "127.0.0.1:123456", "127.0.0.1:"]) {
|
||||
const bind = await runEntry(good, { GAMECULT_IDUNN_CANDIDATE_BIND: badBind });
|
||||
assert.equal(bind.code, 2, badBind);
|
||||
assert.match(bind.output, /CANDIDATE_BIND is not host:port/);
|
||||
assert.ok(!bind.output.includes("canary"));
|
||||
assert.ok(!bind.output.includes("65536") && !bind.output.includes("99999"));
|
||||
}
|
||||
});
|
||||
|
||||
// A startup failure that kills the unit names its cause from what the failing code itself says (its error code, or
|
||||
// the fixed text of a plain Error), and never repeats a value: an fs error's own message carries the path.
|
||||
test("a failed startup names the cause and repeats no value", async () => {
|
||||
const good = ["--state-root", "/var/lib/gamecult/vili", "--kimodo-commit", "6bb58488037dd65360ff0c5d1692b403a23309f7"];
|
||||
const bind = { GAMECULT_IDUNN_CANDIDATE_BIND: "127.0.0.1:18861" };
|
||||
const unset = await runEntry(good, bind);
|
||||
assert.equal(unset.code, 1);
|
||||
assert.match(unset.output, /GAMECULT_IDUNN_RUNTIME_BUNDLE is required/);
|
||||
const missing = await runEntry(good, { ...bind, GAMECULT_IDUNN_RUNTIME_BUNDLE: "/canary-bundle-dir" });
|
||||
assert.equal(missing.code, 1);
|
||||
assert.match(missing.output, /ENOENT/);
|
||||
assert.ok(!missing.output.includes("canary"), "the failure repeated a value");
|
||||
});
|
||||
|
||||
test("the entry derives the daemon's host, port, interpreter and environment identity from the release", async () => {
|
||||
const { createHash } = await import("node:crypto");
|
||||
const { daemonEnvironment } = await import("./idunn-entry.mjs");
|
||||
const release = mkdtempSync(path.join(os.tmpdir(), "vili-entry-"));
|
||||
try {
|
||||
mkdirSync(path.join(release, "vili", "scripts"), { recursive: true });
|
||||
mkdirSync(path.join(release, "vili", "deployment", "idunn"), { recursive: true });
|
||||
writeFileSync(path.join(release, "vili", "deployment", "idunn", "requirements-build.lock"), "build lock bytes\n");
|
||||
writeFileSync(path.join(release, "vili", "deployment", "idunn", "requirements.lock"), "runtime lock bytes\n");
|
||||
const entryDir = path.join(release, "vili", "scripts");
|
||||
assert.equal(daemonEnvironment("127.0.0.1:65535", entryDir).VILI_DAEMON_PORT, "65535", "the top of the port range is accepted");
|
||||
for (const bind of ["127.0.0.1:18861", "http://127.0.0.1:18861"]) {
|
||||
const derived = daemonEnvironment(bind, entryDir);
|
||||
assert.equal(derived.VILI_DAEMON_HOST, "127.0.0.1");
|
||||
assert.equal(derived.VILI_DAEMON_PORT, "18861");
|
||||
assert.equal(derived.GAMECULT_IDUNN_CANDIDATE_BIND, "http://127.0.0.1:18861", "cultnet-ts compares the bind to the Expected route's endpoint, which carries the scheme Idunn's bare host:port lacks");
|
||||
assert.equal(derived.VILI_PYTHON, path.join(release, "python", "bin", "python3.12"));
|
||||
assert.equal(derived.VILI_PYTHON_ENV, `locks-sha256:${createHash("sha256").update("build lock bytes\nruntime lock bytes\n").digest("hex")}`);
|
||||
}
|
||||
} finally {
|
||||
rmSync(release, { recursive: true, force: true });
|
||||
}
|
||||
});
|
||||
|
||||
// The lease is asked while a generate awaits its worker, not only when the service documents are rewritten. The
|
||||
// bound is one WATCH_MS ask plus the SIGKILL; the test allows a second. The loss is made just after a rewrite of the
|
||||
// service documents (whose own gates have asked), the longest wait for an implementation that asks only then.
|
||||
test("a worker still writing when the lease is lost is dead within a second", async () => {
|
||||
const daemon = await idunnDaemon({ leased: true });
|
||||
try {
|
||||
const store = path.join(daemon.stateRoot, "vili.service.cc");
|
||||
await until(() => existsSync(store), "the service store", 10);
|
||||
const pending = generate(daemon, { prompt: "x", model: "writer" });
|
||||
pending.catch(() => {});
|
||||
const writes = () => {
|
||||
const dir = path.join(daemon.stateRoot, "artifacts");
|
||||
for (const entry of existsSync(dir) ? readdirSync(dir) : []) {
|
||||
const file = path.join(dir, entry, "motion.writes");
|
||||
if (existsSync(file)) return file;
|
||||
}
|
||||
return null;
|
||||
};
|
||||
const file = await until(writes, "the worker to write");
|
||||
const [worker] = daemon.spawns();
|
||||
const rewritten = statSync(store).mtimeMs;
|
||||
await until(() => statSync(store).mtimeMs !== rewritten, "the service documents to be rewritten", 8);
|
||||
await new Promise((resolve) => setTimeout(resolve, 150)); // the rewrite is done: its own gates have asked
|
||||
const lostAt = Date.now();
|
||||
rmSync(daemon.lease);
|
||||
await until(() => !isAlive(worker), "the worker to die", 10);
|
||||
assert.ok(Date.now() - lostAt < 1000, `the worker lived ${Date.now() - lostAt} ms after the loss`);
|
||||
const stamps = readFileSync(file, "utf8").trim().split("\n").map(Number);
|
||||
assert.ok(Math.max(...stamps) - lostAt < 1000, "the worker wrote more than a second after the loss");
|
||||
const answer = await Promise.race([pending, new Promise((resolve) => setTimeout(() => resolve({ status: "no answer" }), 10000))]);
|
||||
assert.equal(answer.status, 503);
|
||||
assert.equal(answer.body.code, "write-lease-lost");
|
||||
assert.deepEqual(readdirSync(path.join(daemon.stateRoot, "jobs")), []);
|
||||
} finally {
|
||||
await daemon.stop();
|
||||
}
|
||||
});
|
||||
|
||||
// A request in flight and one arriving during the stop. The worker takes 1.5 s to stop, so the stop is long enough
|
||||
// for the second request to arrive in it. From the signal on nothing is written and nothing is spawned.
|
||||
for (const kind of ["fake Idunn entry", "hand-run"]) {
|
||||
test(`SIGTERM (${kind}): nothing is written or spawned from the signal on, and the worker tree is dead at exit 0`, async () => {
|
||||
const daemon = kind === "hand-run" ? await startDaemon() : await idunnDaemon({ leased: true });
|
||||
try {
|
||||
await until(() => existsSync(path.join(daemon.stateRoot, "vili.service.cc")), "the service store", 10);
|
||||
const inFlight = generate(daemon, { prompt: "x", model: "slow-to-stop" });
|
||||
inFlight.catch(() => {});
|
||||
await until(() => daemon.spawns().length, "the worker");
|
||||
await until(() => request(daemon.port, "GET", "/health").then((r) => r.body?.state?.backend?.state === "ready"), "the worker to be ready");
|
||||
const store = path.join(daemon.stateRoot, "vili.service.cc");
|
||||
const storeBefore = statSync(store).mtimeMs;
|
||||
const artifacts = readdirSync(path.join(daemon.stateRoot, "artifacts"));
|
||||
daemon.proc.kill("SIGTERM");
|
||||
await new Promise((resolve) => setTimeout(resolve, 300));
|
||||
const second = await generate(daemon).catch((error) => ({ status: error.code }));
|
||||
assert.notEqual(second.status, 200, "a request arriving during the stop is not served");
|
||||
await until(() => daemon.proc.exitCode !== null || daemon.proc.signalCode !== null, "the daemon to exit", 30);
|
||||
assert.equal(daemon.proc.exitCode, 0);
|
||||
assert.equal(daemon.spawns().length, 1, "no worker is spawned after the signal");
|
||||
assert.ok(daemon.spawns().every((pid) => !isAlive(pid)), "the worker tree is dead at exit");
|
||||
assert.deepEqual(readdirSync(path.join(daemon.stateRoot, "artifacts")), artifacts, "no artifact is created after the signal");
|
||||
assert.deepEqual(readdirSync(path.join(daemon.stateRoot, "jobs")), [], "no job is recorded after the signal");
|
||||
assert.equal(statSync(store).mtimeMs, storeBefore);
|
||||
assert.ok(!readFileSync(path.join(daemon.stateRoot, "logs", "worker.log"), "utf8").includes("stopping-line"), "the worker's output while it stops is not copied");
|
||||
} finally {
|
||||
await daemon.stop();
|
||||
}
|
||||
});
|
||||
}
|
||||
|
||||
// One notion of root identity: a root is the same root under either spelling, for the lock and for reclaim.
|
||||
for (const [crashedOn, restartedOn] of [["link", "real"], ["real", "link"]]) {
|
||||
test(`a daemon restarted on the ${restartedOn} spelling of its root reclaims the worker a crash on the ${crashedOn} spelling left`, async () => {
|
||||
const real = mkdtempSync(path.join(os.tmpdir(), "vili-test-"));
|
||||
const link = `${real}-link`;
|
||||
symlinkSync(real, link);
|
||||
const spelling = { real, link };
|
||||
const first = await startDaemon({ root: spelling[crashedOn] });
|
||||
let second = null;
|
||||
let stray = null;
|
||||
try {
|
||||
assert.equal((await generate(first)).status, 200);
|
||||
[stray] = first.spawns();
|
||||
first.proc.kill("SIGKILL");
|
||||
await until(() => first.proc.exitCode !== null || first.proc.signalCode !== null, "the first daemon to die");
|
||||
assert.ok(isAlive(stray), "the probe must reproduce the orphan before the fix can be measured");
|
||||
second = await startDaemon({ root: spelling[restartedOn] });
|
||||
await until(() => !isAlive(stray), "the predecessor's worker to be reclaimed");
|
||||
assert.equal((await generate(second)).status, 200);
|
||||
assert.equal(aliveWorkers(second).length, 1, "one live worker on the one root");
|
||||
} finally {
|
||||
if (stray) { try { process.kill(-stray, "SIGKILL"); } catch { /* gone */ } }
|
||||
if (second) await second.stop();
|
||||
rmSync(link, { force: true });
|
||||
rmSync(real, { recursive: true, force: true });
|
||||
}
|
||||
});
|
||||
}
|
||||
|
||||
// The group is the worker's, and its leader may be the first to go. Any member carrying the root's marker proves it.
|
||||
test("a restarted daemon kills a worker group whose leader is gone and a member lives", async () => {
|
||||
const root = mkdtempSync(path.join(os.tmpdir(), "vili-test-"));
|
||||
const first = await startDaemon({ root });
|
||||
const stray = generate(first, { prompt: "x", model: "encoder" }).catch(() => null);
|
||||
const [grandchild] = await until(() => { const found = first.grandchildren(); return found.length ? found : null; }, "the encoder child");
|
||||
const [worker] = first.spawns();
|
||||
first.proc.kill("SIGKILL");
|
||||
await until(() => first.proc.exitCode !== null || first.proc.signalCode !== null, "the first daemon to die");
|
||||
process.kill(worker, "SIGKILL"); // the leader only
|
||||
await until(() => !isAlive(worker), "the leader to die");
|
||||
await stray;
|
||||
assert.ok(isAlive(grandchild), "the probe must leave a member alive in the leaderless group");
|
||||
const second = await startDaemon({ root });
|
||||
try {
|
||||
await until(() => !isAlive(grandchild), "the leaderless group's member to be killed");
|
||||
assert.equal(existsSync(path.join(root, "worker.pid")), false);
|
||||
} finally {
|
||||
try { process.kill(-worker, "SIGKILL"); } catch { /* gone */ }
|
||||
await second.stop();
|
||||
}
|
||||
});
|
||||
|
||||
// The copy of worker output is gated at every line, not at the first: here the lease goes after the first line
|
||||
// was logged, by the worker's first healthy answer, and the line it prints then must not reach the log.
|
||||
test("a worker's output line after the first is not logged once the lease is gone", async () => {
|
||||
const daemon = await idunnDaemon({ leased: true, loseWhenExists: "embeddings/stub-healthy" });
|
||||
try {
|
||||
await until(() => existsSync(path.join(daemon.stateRoot, "vili.service.cc")), "the service store", 10);
|
||||
const answer = await generate(daemon, { prompt: "x", model: "chatty" });
|
||||
assert.equal(answer.status, 503);
|
||||
assert.equal(answer.body.code, "write-lease-lost");
|
||||
await new Promise((resolve) => setTimeout(resolve, 500));
|
||||
const log = readFileSync(path.join(daemon.stateRoot, "logs", "worker.log"), "utf8");
|
||||
assert.ok(log.includes("stub-stdout-line"), "the first line was logged under the lease");
|
||||
assert.ok(!log.includes("after-healthy-line"), "a line after the loss must not be copied");
|
||||
} finally {
|
||||
await daemon.stop();
|
||||
}
|
||||
});
|
||||
|
|
|
|||
|
|
@ -55,6 +55,9 @@ class Identity(unittest.TestCase):
|
|||
self.assertNotEqual(key, te.embedding_key("id", "text b"))
|
||||
self.assertNotEqual(te.embedding_key("id", "a" * 300 + "x"), te.embedding_key("id", "a" * 300 + "y"))
|
||||
|
||||
def test_key_is_case_sensitive(self):
|
||||
self.assertNotEqual(te.embedding_key("id", "A person waves."), te.embedding_key("id", "a person waves."))
|
||||
|
||||
def test_chat_header_is_the_llama3_user_turn_pooled_after_the_separator(self):
|
||||
header = "<|start_header_id|>user<|end_header_id|>" + chr(10) * 2
|
||||
self.assertEqual(te.chat_header("hello "), header + "!@#$%^&*()hello<|eot_id|>")
|
||||
|
|
@ -269,6 +272,68 @@ class ChildBoundary(unittest.TestCase):
|
|||
self.assertEqual(len(answer["vectors"]), 2)
|
||||
|
||||
|
||||
class ChildHeader(unittest.TestCase):
|
||||
def test_every_text_is_tokenized_with_the_chat_header(self):
|
||||
import importlib.util
|
||||
import torch
|
||||
spec = importlib.util.spec_from_file_location("kimodo_text_encode", Path(te.__file__).with_name("kimodo-text-encode.py"))
|
||||
child = importlib.util.module_from_spec(spec)
|
||||
spec.loader.exec_module(child)
|
||||
|
||||
class Recorder:
|
||||
def __init__(self):
|
||||
self.seen = []
|
||||
|
||||
def tokenize(self, batch):
|
||||
self.seen.extend(batch)
|
||||
return batch
|
||||
|
||||
def forward(self, features):
|
||||
return torch.zeros(1, te.EMBEDDING_DIM)
|
||||
|
||||
model = Recorder()
|
||||
vectors, seconds = child.encode_texts(model, ["a person waves", " padded "])
|
||||
self.assertEqual(model.seen, [te.chat_header("a person waves"), te.chat_header(" padded ")])
|
||||
for header in model.seen:
|
||||
self.assertTrue(header.startswith("<|start_header_id|>user<|end_header_id|>" + chr(10) * 2))
|
||||
self.assertEqual([len(v) for v in vectors], [te.EMBEDDING_DIM * 4] * 2)
|
||||
self.assertEqual(len(seconds), 2)
|
||||
|
||||
def test_a_vector_of_the_wrong_width_is_refused(self):
|
||||
import importlib.util
|
||||
import torch
|
||||
spec = importlib.util.spec_from_file_location("kimodo_text_encode", Path(te.__file__).with_name("kimodo-text-encode.py"))
|
||||
child = importlib.util.module_from_spec(spec)
|
||||
spec.loader.exec_module(child)
|
||||
|
||||
class Narrow:
|
||||
def tokenize(self, batch):
|
||||
return batch
|
||||
|
||||
def forward(self, features):
|
||||
return torch.zeros(1, te.EMBEDDING_DIM - 1)
|
||||
|
||||
with self.assertRaises(AssertionError):
|
||||
child.encode_texts(Narrow(), ["a person waves"])
|
||||
|
||||
|
||||
# sha256 of the two Kimodo sources the encoder mirrors, at vendor/kimodo 6bb58488 (embedding-identity-complete).
|
||||
# A Kimodo bump that changes either file must re-read the mirrored loader before these move.
|
||||
KIMODO_LOADER_SHA256 = {
|
||||
"model/load_model.py": "008092cd0b7adef1faea9fa9075718a61bea8ca9feaf9acf1b1d6caacaa0380a",
|
||||
"model/llm2vec/llm2vec.py": "dc6e8b7d38086b4f456e013ab22cd4cc2cd72eb7fd2c13069f037f42aec80857",
|
||||
}
|
||||
|
||||
|
||||
class KimodoPin(unittest.TestCase):
|
||||
def test_the_mirrored_loader_sources_are_the_pinned_ones(self):
|
||||
import hashlib
|
||||
import kimodo
|
||||
root = Path(kimodo.__file__).resolve().parent
|
||||
for relative, expected in KIMODO_LOADER_SHA256.items():
|
||||
self.assertEqual(hashlib.sha256((root / relative).read_bytes()).hexdigest(), expected, relative)
|
||||
|
||||
|
||||
REFERENCE_MODELS = os.environ.get("VILI_REFERENCE_MODELS")
|
||||
ROCK = "A person shakes their fist up and down three times, then thrusts a closed fist forward like a rock."
|
||||
REFERENCE_CODE = """
|
||||
|
|
|
|||
|
|
@ -1,11 +1,13 @@
|
|||
#!/usr/bin/env node
|
||||
import http from "node:http";
|
||||
import os from "node:os";
|
||||
import { closeSync, openSync, readdirSync, readFileSync, unlinkSync, writeFileSync, writeSync } from "node:fs";
|
||||
import { createHash } from "node:crypto";
|
||||
import { closeSync, openSync, readdirSync, readFileSync, realpathSync, unlinkSync, writeFileSync, writeSync } from "node:fs";
|
||||
import { mkdir, writeFile } from "node:fs/promises";
|
||||
import path from "node:path";
|
||||
import readline from "node:readline";
|
||||
import { fileURLToPath } from "node:url";
|
||||
import net from "node:net";
|
||||
import { fileURLToPath, pathToFileURL } from "node:url";
|
||||
import { createRequire } from "node:module";
|
||||
import { spawn } from "node:child_process";
|
||||
|
||||
|
|
@ -16,7 +18,7 @@ const {
|
|||
SingleFileMessagePackBackingStore,
|
||||
defineDocumentRegistry,
|
||||
defineDocumentType,
|
||||
} = require("cultcache-ts");
|
||||
} = require("@gamecult/cultcache-ts");
|
||||
const projectRoot = path.resolve(__dirname, "..");
|
||||
const providerId = "vili.animation";
|
||||
const viliStoreFileName = "vili.service.cc";
|
||||
|
|
@ -73,6 +75,17 @@ const viliDocumentRegistry = defineDocumentRegistry(
|
|||
|
||||
const MAX_THREADS = 16;
|
||||
|
||||
// The state root's one identity: symlinks resolved as far as the path exists. The hand-run lock's name, reclaim's
|
||||
// marker and the worker's environment all derive from args.stateRoot, so two spellings of one root are one root.
|
||||
function canonicalPath(target) {
|
||||
try {
|
||||
return realpathSync(target);
|
||||
} catch {
|
||||
const parent = path.dirname(target);
|
||||
return parent === target ? target : path.join(canonicalPath(parent), path.basename(target));
|
||||
}
|
||||
}
|
||||
|
||||
function parseArgs(argv) {
|
||||
const env = process.env;
|
||||
const args = {
|
||||
|
|
@ -85,7 +98,6 @@ function parseArgs(argv) {
|
|||
threads: env.VILI_THREADS || "8",
|
||||
kimodoCommit: env.VILI_KIMODO_COMMIT || "",
|
||||
pythonEnv: env.VILI_PYTHON_ENV || "",
|
||||
health: false,
|
||||
};
|
||||
|
||||
for (let index = 0; index < argv.length; index += 1) {
|
||||
|
|
@ -99,13 +111,13 @@ function parseArgs(argv) {
|
|||
else if (arg === "--threads") args.threads = argv[++index];
|
||||
else if (arg === "--kimodo-commit") args.kimodoCommit = argv[++index];
|
||||
else if (arg === "--python-env") args.pythonEnv = argv[++index];
|
||||
else if (arg === "--health") args.health = true;
|
||||
}
|
||||
if (!/^[0-9]{1,2}$/.test(String(args.threads)) || Number(args.threads) < 1 || Number(args.threads) > MAX_THREADS) {
|
||||
console.error(`threads must be an integer from 1 to ${MAX_THREADS}`);
|
||||
process.exit(2);
|
||||
}
|
||||
args.threads = Number(args.threads);
|
||||
args.stateRoot = canonicalPath(args.stateRoot);
|
||||
if (args.modelsRoot) args.modelsRoot = path.resolve(args.modelsRoot);
|
||||
return args;
|
||||
}
|
||||
|
|
@ -245,13 +257,17 @@ function transportProfile() {
|
|||
}
|
||||
|
||||
async function persistSurfaces() {
|
||||
guard();
|
||||
await mkdir(args.stateRoot, { recursive: true });
|
||||
const cache = CultCache.builder()
|
||||
.withRegistry(viliDocumentRegistry)
|
||||
.withGenericStore(new SingleFileMessagePackBackingStore(path.join(args.stateRoot, viliStoreFileName)))
|
||||
.build();
|
||||
guard();
|
||||
await cache.put(operatorStateDefinition, "vili", operatorState());
|
||||
guard();
|
||||
await cache.put(commandBoundaryDefinition, "vili", commandBoundary());
|
||||
guard();
|
||||
await cache.put(transportProfileDefinition, "vili", transportProfile());
|
||||
}
|
||||
|
||||
|
|
@ -294,11 +310,9 @@ function waitForExit(proc, timeoutMs) {
|
|||
}
|
||||
|
||||
// The worker leads its own process group (its pid is the group id), so the encoder child it spawns is in the group
|
||||
// too. The daemon owns the whole group: it is killed when the worker exits for any reason, on a requested stop, and
|
||||
// at the next start if a crashed daemon left it behind (reclaimStrayWorker). Under systemd, KillMode=control-group
|
||||
// is the same ownership from outside the process.
|
||||
const workerPidFile = () => path.join(args.stateRoot, "worker.pid");
|
||||
|
||||
// too. The daemon owns the whole group: it is killed when the worker exits for any reason and on a requested stop.
|
||||
// A crashed hand-run daemon's group is reclaimed by the next hand-run daemon (handRunRuntime). Under Idunn the
|
||||
// unit's cgroup is the same ownership from outside the process.
|
||||
function killGroup(pgid, signal) {
|
||||
try {
|
||||
process.kill(-pgid, signal);
|
||||
|
|
@ -307,61 +321,109 @@ function killGroup(pgid, signal) {
|
|||
}
|
||||
}
|
||||
|
||||
function forgetWorkerPid(pid) {
|
||||
try {
|
||||
if (readFileSync(workerPidFile(), "utf8").trim() === String(pid)) unlinkSync(workerPidFile());
|
||||
} catch {
|
||||
// no pid file, or another worker's
|
||||
}
|
||||
}
|
||||
// The hand-run way of owning a state root (tests, development): a Unix socket lock, the worker's pid file and the
|
||||
// /proc reclaim. Idunn's runtime (scripts/idunn-entry.mjs) replaces all three with its process write lease.
|
||||
export function handRunRuntime(stateRoot) {
|
||||
const pidFile = path.join(stateRoot, "worker.pid");
|
||||
let lock = null;
|
||||
|
||||
// A worker left by a crashed daemon: the pid file names its group, and the group is ours only if some member
|
||||
// still carries this state root's embeddings directory in its environment (the pid may have been reused).
|
||||
function reclaimStrayWorker() {
|
||||
let pgid;
|
||||
try {
|
||||
pgid = Number(readFileSync(workerPidFile(), "utf8").trim());
|
||||
} catch {
|
||||
return;
|
||||
}
|
||||
const marker = `VILI_EMBEDDINGS_DIR=${path.join(args.stateRoot, "embeddings")}`;
|
||||
let ours = false;
|
||||
try {
|
||||
for (const entry of readdirSync("/proc")) {
|
||||
if (!/^\d+$/.test(entry)) continue;
|
||||
try {
|
||||
const stat = readFileSync(`/proc/${entry}/stat`, "utf8");
|
||||
const fields = stat.slice(stat.lastIndexOf(")") + 2).split(" ");
|
||||
if (Number(fields[2]) !== pgid) continue;
|
||||
if (readFileSync(`/proc/${entry}/environ`, "utf8").split("\0").includes(marker)) ours = true;
|
||||
} catch {
|
||||
// the process exited while we looked
|
||||
}
|
||||
// The lock is a listening socket in Linux's abstract namespace, named by the state root, which is canonical (its real path, see canonicalPath). A bind is
|
||||
// atomic and exclusive and the kernel drops it with its owner, so there is no lock file, no stale lock to probe
|
||||
// or take over, and no takeover to race: a bind either succeeds (this daemon owns the root) or it does not.
|
||||
const listen = () => new Promise((resolve, reject) => {
|
||||
const name = createHash("sha256").update(stateRoot).digest("hex");
|
||||
const server = net.createServer((socket) => socket.destroy());
|
||||
server.once("error", reject);
|
||||
server.listen(`\0vili-state-root:${name}`, () => {
|
||||
server.removeListener("error", reject);
|
||||
lock = server;
|
||||
resolve();
|
||||
});
|
||||
});
|
||||
|
||||
// A worker left by a crashed daemon: the pid file names its group, and the group is ours only if some member
|
||||
// still carries this state root's embeddings directory in its environment (the pid may have been reused).
|
||||
function reclaimStrayWorker() {
|
||||
let pgid;
|
||||
try {
|
||||
pgid = Number(readFileSync(pidFile, "utf8").trim());
|
||||
} catch {
|
||||
return;
|
||||
}
|
||||
const marker = `VILI_EMBEDDINGS_DIR=${path.join(stateRoot, "embeddings")}`;
|
||||
let ours = false;
|
||||
try {
|
||||
for (const entry of readdirSync("/proc")) {
|
||||
if (!/^\d+$/.test(entry)) continue;
|
||||
try {
|
||||
const stat = readFileSync(`/proc/${entry}/stat`, "utf8");
|
||||
const fields = stat.slice(stat.lastIndexOf(")") + 2).split(" ");
|
||||
if (Number(fields[2]) !== pgid) continue;
|
||||
if (readFileSync(`/proc/${entry}/environ`, "utf8").split("\0").includes(marker)) ours = true;
|
||||
} catch {
|
||||
// the process exited while we looked
|
||||
}
|
||||
}
|
||||
} catch {
|
||||
// no /proc: nothing can be verified, so nothing is killed
|
||||
}
|
||||
if (ours) killGroup(pgid, "SIGKILL");
|
||||
try {
|
||||
unlinkSync(pidFile);
|
||||
} catch {
|
||||
// already gone
|
||||
}
|
||||
} catch {
|
||||
// no /proc: nothing can be verified, so nothing is killed
|
||||
}
|
||||
if (ours) killGroup(pgid, "SIGKILL");
|
||||
try {
|
||||
unlinkSync(workerPidFile());
|
||||
} catch {
|
||||
// already gone
|
||||
}
|
||||
|
||||
return {
|
||||
async acquire() {
|
||||
await mkdir(stateRoot, { recursive: true });
|
||||
try {
|
||||
await listen();
|
||||
} catch (error) {
|
||||
if (error?.code !== "EADDRINUSE") throw error;
|
||||
console.error("the state root is held by a running daemon");
|
||||
process.exit(2);
|
||||
}
|
||||
reclaimStrayWorker(); // synchronous: no request is served before a predecessor's worker is gone
|
||||
},
|
||||
held: () => lock !== null,
|
||||
refusal: () => "write-lease-pending",
|
||||
watch() {},
|
||||
recordWorker(pid) {
|
||||
writeFileSync(pidFile, `${pid}\n`);
|
||||
},
|
||||
forgetWorker(pid) {
|
||||
try {
|
||||
if (readFileSync(pidFile, "utf8").trim() === String(pid)) unlinkSync(pidFile);
|
||||
} catch {
|
||||
// no pid file, or another worker's
|
||||
}
|
||||
},
|
||||
release() {
|
||||
if (lock) lock.close();
|
||||
lock = null;
|
||||
},
|
||||
};
|
||||
}
|
||||
|
||||
async function stopWorker() {
|
||||
// A requested stop gives the worker its grace (SIGTERM, then SIGKILL); a lost writer gets SIGKILL at once.
|
||||
async function stopWorker(signals = ["SIGTERM", "SIGKILL"]) {
|
||||
const proc = worker.proc;
|
||||
if (!proc || proc.pid === undefined) return;
|
||||
worker.state = "stopped";
|
||||
for (const signal of ["SIGTERM", "SIGKILL"]) {
|
||||
for (const signal of signals) {
|
||||
killGroup(proc.pid, signal);
|
||||
if (await waitForExit(proc, signal === "SIGTERM" ? 10000 : 5000)) break;
|
||||
}
|
||||
}
|
||||
|
||||
async function startWorker(model, timeoutMs) {
|
||||
guard();
|
||||
await mkdir(path.join(args.stateRoot, "logs"), { recursive: true });
|
||||
guard();
|
||||
await mkdir(path.join(args.stateRoot, "embeddings"), { recursive: true });
|
||||
guard(); // synchronous with the log open and the spawn below: no await between them
|
||||
const logFd = openSync(path.join(args.stateRoot, "logs", "worker.log"), "a");
|
||||
const argv = [args.workerEntry, "--host", "127.0.0.1", "--port", "0"];
|
||||
if (model) argv.push("--model", model);
|
||||
|
|
@ -374,10 +436,10 @@ async function startWorker(model, timeoutMs) {
|
|||
});
|
||||
closeSync(logFd);
|
||||
if (proc.pid !== undefined) {
|
||||
writeFileSync(workerPidFile(), `${proc.pid}\n`);
|
||||
runtime.recordWorker(proc.pid);
|
||||
proc.once("exit", () => {
|
||||
killGroup(proc.pid, "SIGKILL"); // whatever the worker left running, an encoder child included
|
||||
forgetWorkerPid(proc.pid);
|
||||
runtime.forgetWorker(proc.pid);
|
||||
});
|
||||
}
|
||||
Object.assign(worker, { state: "starting", proc, pid: proc.pid ?? null, port: null, servedModels: [], lastError: null });
|
||||
|
|
@ -418,7 +480,7 @@ async function startWorker(model, timeoutMs) {
|
|||
} catch {
|
||||
// not JSON: an ordinary log line
|
||||
}
|
||||
writeSync(sink, `${line}\n`);
|
||||
if (mayWrite()) writeSync(sink, `${line}\n`);
|
||||
});
|
||||
lines.on("close", () => closeSync(sink));
|
||||
}),
|
||||
|
|
@ -463,6 +525,16 @@ async function readRequestBody(request) {
|
|||
}
|
||||
|
||||
async function generateMotion(request, response) {
|
||||
try {
|
||||
await generate(request, response);
|
||||
} catch (error) {
|
||||
if (!(error instanceof LeaseLost)) throw error;
|
||||
json(response, 503, { ok: false, code: stopping ? "stopping" : runtime.refusal() });
|
||||
}
|
||||
}
|
||||
|
||||
async function generate(request, response) {
|
||||
guard();
|
||||
if (workerUnconfigured()) {
|
||||
json(response, 503, { ok: false, code: "worker-unconfigured" });
|
||||
return;
|
||||
|
|
@ -492,6 +564,7 @@ async function generateMotion(request, response) {
|
|||
|
||||
const jobId = `vili-${Date.now()}`;
|
||||
const artifactDir = path.join(args.stateRoot, "artifacts", jobId);
|
||||
guard();
|
||||
await mkdir(artifactDir, { recursive: true });
|
||||
const duration = firstDefined(body.durationSeconds, body.duration, 4);
|
||||
const bodyMeta = typeof body.meta === "object" && body.meta ? body.meta : {};
|
||||
|
|
@ -528,6 +601,7 @@ async function generateMotion(request, response) {
|
|||
return undefined;
|
||||
})();
|
||||
if (body.constraints !== undefined && typeof body.constraints !== "string") {
|
||||
guard();
|
||||
await writeFile(path.join(artifactDir, "constraints.json"), JSON.stringify(body.constraints, null, 2));
|
||||
}
|
||||
const requestRecord = {
|
||||
|
|
@ -551,6 +625,7 @@ async function generateMotion(request, response) {
|
|||
saveExampleDir: boolValue(firstDefined(body.saveExampleDir, body.save_example_dir)),
|
||||
inputFolder: firstDefined(body.inputFolder, body.input_folder),
|
||||
};
|
||||
guard();
|
||||
await writeFile(path.join(artifactDir, "meta.json"), JSON.stringify(meta, null, 2));
|
||||
|
||||
let result;
|
||||
|
|
@ -580,6 +655,7 @@ async function generateMotion(request, response) {
|
|||
saveExampleDir: boolValue(firstDefined(body.saveExampleDir, body.save_example_dir)),
|
||||
output: path.join(artifactDir, "motion"),
|
||||
};
|
||||
guard();
|
||||
await writeFile(path.join(artifactDir, "request.json"), JSON.stringify(workerRequest, null, 2));
|
||||
const answer = await fetch(`http://127.0.0.1:${worker.port}/generate`, {
|
||||
method: "POST",
|
||||
|
|
@ -590,9 +666,11 @@ async function generateMotion(request, response) {
|
|||
const workerBody = await answer.json();
|
||||
result = { ok: answer.ok && workerBody?.ok !== false, error: answer.ok ? null : `the worker answered ${answer.status}`, body: workerBody };
|
||||
} catch (error) {
|
||||
if (error instanceof LeaseLost) throw error;
|
||||
result = { ok: false, error: `the worker request failed (${error?.name || "Error"})`, body: null };
|
||||
}
|
||||
|
||||
guard(); // the writer may have been lost while the job ran
|
||||
const job = {
|
||||
schema: "gamecult.vili.motion_job.v0",
|
||||
jobId,
|
||||
|
|
@ -612,8 +690,48 @@ async function generateMotion(request, response) {
|
|||
json(response, result.ok ? 200 : 502, job);
|
||||
}
|
||||
|
||||
const SNAPSHOT_BODY_MAX = 64 * 1024;
|
||||
|
||||
// Idunn's route challenge: a msgpack POST from the host, answered from local authority. A browser page can send
|
||||
// it too, so an Origin is refused exactly as on every other POST. Only the Idunn runtime answers; the hand-run
|
||||
// daemon has no such route.
|
||||
async function answerRouteChallenge(request, response) {
|
||||
if (!runtime.answerSnapshot) {
|
||||
text(response, 404, "not found\n");
|
||||
return;
|
||||
}
|
||||
if (request.headers.origin !== undefined) {
|
||||
json(response, 403, { ok: false, code: "cross-site" });
|
||||
return;
|
||||
}
|
||||
const chunks = [];
|
||||
let size = 0;
|
||||
for await (const chunk of request) {
|
||||
size += chunk.length;
|
||||
if (size > SNAPSHOT_BODY_MAX) {
|
||||
json(response, 400, { ok: false, code: "invalid-snapshot-request" });
|
||||
return;
|
||||
}
|
||||
chunks.push(chunk);
|
||||
}
|
||||
let answer;
|
||||
try {
|
||||
answer = runtime.answerSnapshot(Buffer.concat(chunks));
|
||||
} catch (error) {
|
||||
if (error?.code === "invalid-snapshot-request") json(response, 400, { ok: false, code: error.code });
|
||||
else json(response, 503, { ok: false, code: "runtime-presence-unavailable" });
|
||||
return;
|
||||
}
|
||||
response.writeHead(200, { "content-type": "application/msgpack", "cache-control": "no-store" });
|
||||
response.end(answer);
|
||||
}
|
||||
|
||||
async function handleRequest(request, response) {
|
||||
const url = new URL(request.url, `http://${request.headers.host || `${args.host}:${args.port}`}`);
|
||||
if (url.pathname === "/cultnet/snapshot" && request.method === "POST") {
|
||||
await answerRouteChallenge(request, response);
|
||||
return;
|
||||
}
|
||||
if (request.method === "POST" && refuseCrossSite(request, response)) return;
|
||||
|
||||
if (url.pathname === "/" || url.pathname === "/health") {
|
||||
|
|
@ -627,23 +745,33 @@ async function handleRequest(request, response) {
|
|||
}
|
||||
}
|
||||
|
||||
async function healthCheck() {
|
||||
const host = args.host === "0.0.0.0" ? "127.0.0.1" : args.host;
|
||||
const response = await fetch(`http://${host}:${args.port}/health`);
|
||||
if (!response.ok) throw new Error(`health returned ${response.status}`);
|
||||
const body = await response.json();
|
||||
if (!body.ok) throw new Error("health returned ok=false");
|
||||
let runtime;
|
||||
let stopping = false;
|
||||
|
||||
class LeaseLost extends Error {}
|
||||
|
||||
// The one write authority: the runtime holds the state root (the lease, or the lock) and the daemon is not stopping.
|
||||
// held() is read at the moment of every ask, so a loss is seen by the first gate to ask.
|
||||
function mayWrite() {
|
||||
return !stopping && runtime.held();
|
||||
}
|
||||
|
||||
if (args.health) {
|
||||
healthCheck().then(() => process.exit(0), (error) => {
|
||||
console.error(error.message);
|
||||
process.exit(1);
|
||||
});
|
||||
} else {
|
||||
await mkdir(path.join(args.stateRoot, "jobs"), { recursive: true });
|
||||
await persistSurfaces();
|
||||
// The one write gate. Every write to the state root and every spawn calls it, immediately before it happens (not once
|
||||
// per request), so a request admitted before the lease went, or before SIGTERM, still stops at its next write.
|
||||
function guard() {
|
||||
if (!mayWrite()) throw new LeaseLost("the writer was lost");
|
||||
}
|
||||
|
||||
// How often the lease is asked while nothing else asks, and so how long a worker can outlive a loss (plus the
|
||||
// SIGKILL): a generate can await its worker for 30 minutes. The service documents are rewritten every PERSIST_EVERY asks.
|
||||
const WATCH_MS = 250;
|
||||
const PERSIST_EVERY = 20;
|
||||
|
||||
// The one daemon. `rt` owns the state root: the hand-run runtime (socket lock) or Idunn's (process write lease).
|
||||
// The server listens first, so Idunn's challenge can be answered while the lease is still pending; nothing is
|
||||
// written to the state root until the runtime holds it.
|
||||
export async function serve(rt) {
|
||||
runtime = rt;
|
||||
const server = http.createServer((request, response) => {
|
||||
handleRequest(request, response).catch((error) => {
|
||||
// The error's message can carry the request; name only its class.
|
||||
|
|
@ -652,20 +780,55 @@ if (args.health) {
|
|||
else response.end();
|
||||
});
|
||||
});
|
||||
await new Promise((resolve) => server.listen(args.port, args.host, resolve));
|
||||
console.log(`Vili listening on http://${args.host}:${server.address().port}`);
|
||||
|
||||
server.listen(args.port, args.host, () => {
|
||||
reclaimStrayWorker(); // synchronous: no request is served before a predecessor's worker is gone
|
||||
console.log(`Vili listening on http://${args.host}:${server.address().port}`);
|
||||
});
|
||||
|
||||
const surfaceTimer = setInterval(() => persistSurfaces().catch((error) => console.error(`persist failed: ${error?.name || "Error"}`)), 5000);
|
||||
|
||||
let tick = null;
|
||||
// From the signal on nothing is accepted, written or spawned (guard() refuses what is already in flight); then the
|
||||
// worker tree is stopped, and only then does the process end.
|
||||
const shutdown = async () => {
|
||||
clearInterval(surfaceTimer);
|
||||
await stopWorker();
|
||||
if (stopping) return;
|
||||
stopping = true;
|
||||
server.close();
|
||||
server.closeIdleConnections();
|
||||
clearInterval(tick);
|
||||
await stopWorker();
|
||||
runtime.release();
|
||||
process.exit(0);
|
||||
};
|
||||
process.on("SIGTERM", shutdown);
|
||||
process.on("SIGINT", shutdown);
|
||||
|
||||
await runtime.acquire();
|
||||
let lost = false;
|
||||
const onLost = () => {
|
||||
if (lost) return;
|
||||
lost = true;
|
||||
clearInterval(tick);
|
||||
stopWorker(["SIGKILL"]).finally(() => runtime.reportHealth?.("degraded"));
|
||||
};
|
||||
runtime.watch(onLost); // held() reports a loss through this the moment any gate sees it
|
||||
try {
|
||||
guard();
|
||||
await mkdir(path.join(args.stateRoot, "jobs"), { recursive: true });
|
||||
await persistSurfaces();
|
||||
} catch (error) {
|
||||
if (error instanceof LeaseLost) return; // lost while opening the state root: onLost has run, and nothing is active
|
||||
throw error;
|
||||
}
|
||||
let asks = 0;
|
||||
tick = setInterval(() => {
|
||||
runtime.watch(onLost);
|
||||
if (lost || ++asks % PERSIST_EVERY !== 0) return;
|
||||
persistSurfaces().catch((error) => {
|
||||
if (!(error instanceof LeaseLost)) console.error(`persist failed: ${error?.name || "Error"}`);
|
||||
});
|
||||
}, WATCH_MS);
|
||||
if (lost) return;
|
||||
runtime.reportHealth?.("active");
|
||||
console.log("Vili holds the state root");
|
||||
}
|
||||
|
||||
if (process.argv[1] && import.meta.url === pathToFileURL(process.argv[1]).href) {
|
||||
await serve(handRunRuntime(args.stateRoot));
|
||||
}
|
||||
|
|
|
|||
1
vendor/SOMA-X
vendored
Submodule
1
vendor/SOMA-X
vendored
Submodule
|
|
@ -0,0 +1 @@
|
|||
Subproject commit 56a3d957a549c66425b500fddf7f3183bc6fbdff
|
||||
1
vendor/kimodo
vendored
Submodule
1
vendor/kimodo
vendored
Submodule
|
|
@ -0,0 +1 @@
|
|||
Subproject commit 6bb58488037dd65360ff0c5d1692b403a23309f7
|
||||
Loading…
Reference in a new issue