From 4a4c8ea5dd52ae4ba3f6dd181d29f9bf2b4adcb4 Mon Sep 17 00:00:00 2001 From: Meta Vens Date: Fri, 23 Oct 2026 02:20:44 +0200 Subject: [PATCH] Research: kimodo.cpp probe (Q8 GGML encoder, MNTP adapter missing from Vili's vectors) Measured on Yggdrasil CPU; Raven GPU timing deferred (household using the GPU). Co-Authored-By: Claude Opus 5.5 --- docs/research/kimodo-cpp-probe-2026-10.md | 161 ++++++++++++++++++++++ 1 file changed, 161 insertions(+) create mode 100644 docs/research/kimodo-cpp-probe-2026-10.md diff --git a/docs/research/kimodo-cpp-probe-2026-10.md b/docs/research/kimodo-cpp-probe-2026-10.md new file mode 100644 index 0000000..f4b749a --- /dev/null +++ b/docs/research/kimodo-cpp-probe-2026-10.md @@ -0,0 +1,161 @@ +# kimodo.cpp probe, 2026-10-23 + +Directed by `motion-pipeline:ruling:remap-against-community-tooling`. Shaped by +`motion-pipeline:ruling:own-implementation-borrow-best`: kimodo.cpp is weighed as a +source of parts for Vili, not as a replacement worker. Measured by +imagination-kimodo-cpp, session sp-reentry-2026-10-21. + +## Headline + +1. **Vili's text vectors lack the LLM2Vec MNTP adapter.** This also holds for + upstream Kimodo at 6bb5848 and at origin/main e1c219c. Under transformers 5.1.0, + Kimodo's `LLM2Vec.from_pretrained("McGill-NLP/...-mntp")` reports the MNTP LoRA + keys as UNEXPECTED (`base_model.model.layers.*`) and MISSING (`layers.*`, newly + initialized). The merge that follows therefore merges a freshly initialized + adapter. Two torch assemblies were run on the same machine with the same + versions (torch 2.5.1 CPU, transformers 5.1.0, peft 0.18.1, bf16): + - **A**, Kimodo's own `LLM2VecEncoder` (the path Vili's own loader mirrors), + reproduces Vili's stored vectors at cosine **0.999972** (fist) and **0.999968** + (wave). + - **B** loads the Llama base, merges the MNTP adapter explicitly with + `PeftModel` and then adds the supervised adapter. This is LLM2Vec's design and + kimodo.cpp's reference assembly. It reproduces kimodo.cpp BF16 at **0.999946** + and **0.999960**. + - A and B agree only at cosine **0.906 to 0.936**. + + This answers the probe asked for by `motion-pipeline:follow_up:encoder-adapter-load-report`. + Upstream has not changed `kimodo/model/llm2vec`, `load_model.py` or the + transformers pin since 6bb5848. The probe cannot tell which encoder NVIDIA's + denoiser was trained on: A, or a correctly loaded B from an older transformers + version. +2. **kimodo.cpp's encoder is numerically B.** Over Vili's 8 cached prompts, its Q8_0 + output has cosine **0.9075 to 0.9354** with Vili's bf16 GPU vectors, while its BF16 + output has **0.9059 to 0.9361**. The whole gap is the + MNTP difference above, not quantization: Q8_0 against kimodo.cpp BF16 is + **0.99967 minimum, 0.99978 mean**, and Q8_0 against torch assembly B is **0.99972** + (fist) and **0.99984** (wave). Upstream's own sweep reports 0.999824. +3. **Clips cannot be compared against Vili's reference clips for the same seed.** This is a + finding, not a gap in the run. kimodo.cpp draws its initial noise with + `std::mt19937_64` and `std::normal_distribution`. Kimodo uses torch's generator + after `torch.manual_seed` and `torch.cuda.manual_seed_all`, on CUDA. Beyond that, + kimodo.cpp does not run Kimodo's post-processing (Vili runs it by default), it + returns 30 SOMA joints rather than the 77-joint posed skeleton Vili stores, and + its CLI fixes both CFG weights at 2.0. Its fixture tools + (`kmd-sample-embedding`, `kmd-sample-fixture`) accept injected embedding and + noise. A like-for-like clip comparison would need Kimodo's CUDA noise captured + on Raven's GPU and post-processing disabled on both sides. That was not done, + because Raven's GPU was in use (below). + +## Clip deltas inside kimodo.cpp (same noise bytes, embedding swapped) + +Setup: SOMA-RP v1.1 F32 denoiser, 60 frames (2 s at 30 fps), 20 DDIM steps, CFG +2/2. The noise came from numpy `default_rng(seed)`. Positions come from forward +kinematics over kimodo.cpp's `soma30` offsets. Bone-length spread was at most +3e-8 m, and a rerun with the same embedding was 0.0000 mm from the first run, +so the sampler is deterministic. + +| prompt | seed | Q8_0 vs kimodo.cpp BF16, mean / max mm | Vili bf16 vector vs kimodo.cpp Q8_0, mean / max mm | +|---|---|---|---| +| fist | 7 | 2.87 / 62.30 | 84.72 / 927.40 | +| fist | 42 | 2.45 / 16.46 | 43.94 / 986.76 | +| shrug | 7 | 2.06 / 17.39 | 28.42 / 145.01 | +| shrug | 42 | 3.00 / 17.83 | 16.46 / 157.28 | +| wave | 7 | 0.54 / 10.87 | 14.06 / 391.16 | +| wave | 42 | 0.94 / 20.84 | 27.70 / 553.69 | + +How to read the columns: + +- **Q8 against BF16** is about the size of Vili's own GPU-bf16 against CPU drift + (fist: mean 4.63, max 54.28 mm, `motion-pipeline:verdict:cut-vili-encoder-placement.s1`). +- **MNTP against no MNTP** produces a different motion. The fist clip's root ends up + close to a metre away. + +## Resources + +None of these runs touched Raven's GPU. At the time of the probe Raven's GPU showed +35 % utilization and 11.6 GiB in use, so **GPU (Vulkan) timing is undone**. It is +deferred to a run when the household tray toggle allows GPU use. + +All runs were on Yggdrasil (AMD EPYC-Genoa VM, 16 vCPU, 62 GiB) in Docker +containers capped at 8 CPUs and 24 GiB (torch: 40 GiB), with `KIMODO_THREADS=8`. +The host was shared with other agents' verification containers, so the timings are +noisy. + +| what | wall | peak RSS | +|---|---|---| +| kimodo.cpp Q8_0 encode, cold process (load and 1st encode) | 6.8 to 14 s (27 s on the very first, cold page cache) | 1.97 to 2.12 GiB | +| kimodo.cpp Q8_0 encode, warm 2nd encode in-process | 3.2 to 7.8 s | the same | +| kimodo.cpp BF16 encode, cold process (warm in-process encode 8.2 to 12.5 s) | 16.6 to 38 s | 3.45 to 3.60 GiB | +| kimodo.cpp 2 s clip from embedding (`kmd-sample-embedding`, 20 steps, includes the 1.1 GB denoiser load) | 26 to 75 s, typically about 30 s | 1.13 GiB | +| kimodo.cpp text to 2 s clip end to end (`kmd-generate`, Q8_0) | 42.9 s | 2.05 GiB | +| torch assembly A or B, bf16 CPU, model load / encode | 41 s or 38 s / 0.9 to 2.1 s | about 16 GiB of weights (peak not sampled) | + +For comparison, today on Raven: a cache hit costs about 2 s for a 2 s clip, and a +miss costs a 61 s encode (`cut-vili-encoder-placement.s1`). Raven has 16 GiB of host +RAM, which is why its CPU bf16 encode took 428 s; Yggdrasil's 1 to 2 s CPU bf16 +encode is a RAM fact, not a CPU fact. kimodo.cpp's Q8 encoder reads its weights +layer by layer and needs about 2 GiB resident, so it would fit Raven on CPU. + +## Feature coverage (kimodo.cpp ed8a931, 2026-10-19) + +- Checkpoints: SOMA-RP and SOMA-SEED **v1.1** only, plus G1 and a locally converted + SMPL-X. There is **no v1.2** (`motion-pipeline:follow_up:kimodo-v1-2`). +- Skeleton: the 30 joints SOMA predicts. 77-joint expansion is "not implemented yet". +- Output: raw local rotations and root positions as f32, and skeleton-only + `animation.glb` from the Go demo. There is **no BVH or FBX from the binaries** + (`scripts/export_bvh.py` is a Python helper), and no skinned GLB. +- Constraints: "General constraint input ... not implemented". The C API + carries a `constraint_cfg_weight` but no constraint data. Vili's + `load_constraints_lst` path has no equivalent. Multi-prompt sequences with + transitions are supported. +- Post-processing: Kimodo's motion correction and foot handling do not run in the + inference path. +- Backends: CPU and Vulkan. Windows MSVC builds via the fork TheLocalLab/kimodo.cpp-windows, + merged upstream 2026-08-31. + +## Maintenance signal and licence + +- **Port.** github.com/localai-org/kimodo.cpp, Apache-2.0, 34 commits from + 2026-08-22 to 2026-10-19. Richard Palethorpe wrote 30 of them, with 3 from + Deepesh70 and 1 from mudler-agent. It is published by the LocalAI team, and the + README carries their attribution. It has 929 stars. The last functional commits + were 2026-09-17 (performance work and the monolithic quantizations); since then + there has been one README commit. GGML is a pinned submodule. The code is built + for parity work: reference capture harness, layer, stage and fixture parity + tests, fuzzers and sanitizer presets. +- **Installer defect seen.** `scripts/download_gguf_weights.sh` exits with + "missing or wrong-sized file: generated/llm2vec-text-bundle/tokenizer.gguf", + because MANIFEST.json still lists the legacy split bundle. The monolith files + were verified by hand against the manifest's sha256 values: + - Q8_0 `b26d0e74...4c5c` (8,142,058,752 bytes) + - BF16 `d9a60017...b20f` (15,177,675,936 bytes) + - tokenizer `81614aca...607e` +- **Quantized weights.** + - HF `LocalAI-io/Llama-3-Kimodo-GGML` was created 2026-08-23, last modified + 2026-09-17, has 161,096 downloads, and is licensed "other". It was quantized by + LocalAI-io with their own `kmd-quantize-text` (Q8_0 for the large matrices; + RMSNorm in BF16 and the supervised LoRA in F32, unquantized). + - It was converted from exactly the revisions in Vili's encoder identity: + `meta-llama/Meta-Llama-3-8B-Instruct@8afb486c`, `McGill-NLP/...-mntp@31474e39` + and `...-mntp-supervised@baa8ebf0`. Its manifest records them. + - It carries the Meta Llama 3 Community License, and the LLM2Vec adapters are MIT. + - The SOMA-RP v1.1 GGUF (`kimodo-soma-rp-v1.1-f32.gguf`, sha256 `3bf1229f...8ee4`) + is a one-to-one conversion under the NVIDIA Open Model License. + +## Method and custody + +- **Reference vectors.** The 8 reference vectors came from a read-only copy of + Raven's `E:\Projects\Vili\.vili\embeddings\text-embeddings.cc`, decoded with + cultcache-py and Vili's `gamecult.vili.text_embedding.v0` definition. Every + vectorSha256 checked. kimodo.cpp was fed each record's stored, already + normalized text. +- **Torch assemblies.** These used the McGill adapters and the Llama config, + copied from Raven's WSL Hugging Face cache, and the four Llama safetensors + shards from the ungated mirror `NousResearch/Meta-Llama-3-8B-Instruct`. Those + shards are byte-identical to Raven's blobs: the sha256 values `d8cf9c4d`, + `8d4782b4`, `3acdd690` and `67e9ad31` match. No Hugging Face token left Raven. +- **Nothing on Raven was touched** beyond those reads: not Vili's daemon, its + worker container, or the GPU. +- **Yggdrasil scratch** is `~/scratch/kimodo-cpp-probe` (39 GiB: weights, + checkouts, outputs). The probe's containers are all removed. Delete the scratch + directory once nobody needs to rerun the probe.