From fd321fdacd8f1c4d0230c0f8940bac8ce5af89db Mon Sep 17 00:00:00 2001 From: Meta Vens Date: Mon, 26 Oct 2026 13:53:34 +0100 Subject: [PATCH 001/300] Research: prior art for a materialized read image in Huginn (serves eureka-substrate:follow_up:huginn-materialized-image) Co-Authored-By: Claude Opus 5.5 --- docs/research/huginn-materialized-image.md | 109 +++++++++++++++++++++ 1 file changed, 109 insertions(+) create mode 100644 docs/research/huginn-materialized-image.md diff --git a/docs/research/huginn-materialized-image.md b/docs/research/huginn-materialized-image.md new file mode 100644 index 0000000..5ad4bef --- /dev/null +++ b/docs/research/huginn-materialized-image.md @@ -0,0 +1,109 @@ +# Prior art: a materialized read image advanced by a single writer, with snapshots for cursors + +Serves follow-up `eureka-substrate:follow_up:huginn-materialized-image` under operator ruling `eureka-substrate:ruling:readback-gate-and-materialized-image`. Campaign `eureka-substrate`, session `self-2026-10-25-eureka`. Written by Eyes (`eyes-materialized-image`) on 2026-10-26. Facts carry evidence pointers; inference is marked "(inferred)". No recommendation is made in the body; the last section lists what each option would delete and cost, for the design fork. + +Method note: web sources were read through a summarizing fetch tool, not verbatim. Claims about third-party systems are as summarized from the cited page; where a page did not say something, that is stated. Nothing was measured in Huginn; the 200 ms / 11k figure is the brief's. + +## 1. What Huginn does today (code facts) + +Paths are under `F:\Projects\Huginn\crates\huginn-mind\src\` unless noted. + +**Storage and the in-memory image** +- `Mind::open` takes the redb store's exclusive per-path lock for the Mind's lifetime (`mind.rs:130-147`; refusal `MindAlreadyOwned`). One process, one writer. +- `open_checked` pulls all envelopes, runs the gates, then `attach` (`mind.rs:183-191`). `Mind.image` is a `Vec` (`mind.rs:102`), the raw encoded envelopes (msgpack payload bytes), in identity order. `Mind::envelopes()` hands it out (`mind.rs:214`). +- After every commit, `refresh()` runs `cache.pull_all_backing_stores()` then `image = snapshot(&cache)` (`mind.rs:274-278`), called from `receipt::commit` (`receipt.rs:246-286`, the call at `:271`). The redb backing store's `pull_all` opens a redb `begin_read()` and reads the whole table (`F:\Projects\CultLib\packages\cultcache-rs\src\lib.rs:2349-2350`). So the write path is also linear in the mind: every admitted batch re-reads every envelope from redb. +- Commit is a compare-and-swap batch: strong reads as expectation, writes plus the receipt as replacements (`receipt.rs:253-270`). Receipts are ordinary envelopes of type `huginn.mind_commit_receipt.v2` (`receipt.rs:28, 97`) with a dense ordinal `1..=N`; `head()` decodes every receipt, sorts ordinals, and refuses unless they are exactly `1..=N` (`receipt.rs:204-213`). + +**The read path (`query.rs`)** +- `Mind::view` (`query.rs:456-460`): `receipt::head` (decodes all receipts), then `Reader::at(self, head)`, then a find and `view_of`. +- `Mind::query` (`query.rs:469-487`): validates, takes `as_of` from the cursor (`Cursor::parse`, refused if beyond head, `:473-484`) or head, then `Reader::at(self, as_of)?.rows()?`, then `select`. Members/ranked paths build a Reader the same way (`:532-533`, `:589-590`). +- `Reader::at` (`query.rs:380-391`): `Docs::from_image(mind.envelopes())` decodes every envelope of a known kind into `Held { kind, key, document }` (`docs.rs:27-54`); `AdmissionIndex::build` calls `head` again and decodes every receipt again, building a `BTreeMap<(type, key), AdmissionFacts>` over all writes, refusing if two receipts write one identity (`query.rs:315-340`); then it keeps `Held` whose receipt `ordinal <= as_of` and drops the rest. "Built once per call and thrown away with it" (`query.rs:365-367`). +- `held()` is a linear `iter().find` (`query.rs:393-395`); `Docs::find`/`in_image` likewise (`docs.rs:65-75`); `Mind::raw_envelope` is a linear scan over the envelope Vec (`mind.rs:261-263`). +- Status is derived per document: `view_of` calls `Docs::closing_resolution` (`query.rs:397-405`), which scans all resolutions and, per candidate, calls `in_force(Resolution, key)`, which itself is `closing_resolution_unless(...).is_none()` (`docs.rs:165-183, 204-206`). `rows()` does this for every held document plus a `base()` walk (`query.rs:420-439`). Cost shape (inferred from the code, not profiled): decode O(N), receipt decode O(R) twice, rows O(N x Rz) where Rz is the number of resolutions, recursion depth bounded by the Q19 cap at two (`query.rs:414-419`). The 200 ms may be dominated by the status scan rather than decode; that is a hypothesis to profile (inferred). +- Admission builds its own `Docs::from_image(self.envelopes())` per batch (`admission.rs:127`; also `index.rs:128`), so admission pays the full decode as well. + +**Invariants the image must keep (from doc comments)** +- A snapshot at `as_of` is the documents whose writing receipt has `ordinal <= as_of`, with every derivation (in force, closing resolution, base, citations) computed over that image alone (`query.rs:374-379`). +- Every held document must have its receipt at any `as_of`; an orphan or a double write refuses at any snapshot (`query.rs:377-379, 351`). +- `writes` only: a document later cited as a strong read keeps the receipt that wrote it (`query.rs:311-314`). A10: one write per identity, which means documents are written once and never overwritten in place; the only "change" is a later resolution document (inferred from `query.rs:311-314`; a persona put is the stated exception, `receipt.rs:259-260`). +- Cursors carry `as_of` and are minted with a per-process random key (`mind.rs:190`, `CursorKey::random()`), so a cursor does not outlive the process (inferred: a restart makes old cursors unverifiable). + +Consequence worth stating plainly (inferred): because identities are written once and receipts are dense and monotone, the set of documents visible at `as_of` is a prefix of the receipt chain. A document's own introduction ordinal is immutable. Only derived status (closed by a resolution) varies with `as_of`, and it varies monotonically: a resolution with ordinal r closes its subject for every `as_of >= r`. This is the structure the prior art below is being asked to exploit. + +## 2. MVCC read views + +**PostgreSQL.** Under Repeatable Read the snapshot is taken at the first non-transaction-control statement and every later statement in the transaction sees the same committed data (PostgreSQL docs, "Transaction Isolation", https://www.postgresql.org/docs/current/transaction-iso.html). A snapshot can be exported with `pg_export_snapshot()` and imported by another session via `SET TRANSACTION SNAPSHOT`; the exporting transaction must stay open for the id to remain importable (https://www.postgresql.org/docs/current/functions-admin.html). Failure mode: an old row version must be kept while any transaction may still see it, so open or idle-in-transaction sessions, prepared transactions and replication slots hold back cleanup and cause bloat (https://www.postgresql.org/docs/current/routine-vacuuming.html; the xmin-horizon mechanism is named in the page's diagnostic queries and "outside the page" note, i.e. partly the fetch tool's general knowledge). +- Relevance: PostgreSQL's cursor-like long snapshot is a held transaction. The cost of serving an old snapshot is paid in retained versions, by the writer's storage, for as long as the snapshot is held. A Huginn cursor has no connection and may be resumed minutes later; holding an engine transaction per outstanding cursor is the Postgres exported-snapshot model and carries its bloat risk (inferred). + +**LMDB.** Single writer, readers take no locks; read transactions pin old page versions, so "the database can grow quickly" under long-lived readers, and stale readers from aborted processes cause the same; docs advise avoiding long-lived transactions, checking stale readers with `mdb_reader_check`, and note the reader table has a fixed slot count (default 126) (lmdb.h comments, https://raw.githubusercontent.com/LMDB/lmdb/mdb.master/libraries/liblmdb/lmdb.h). `MDB_NOTLS` lets a read transaction cross threads. +- Relevance: same cost class as redb below. + +**redb (already in Huginn).** MVCC on a copy-on-write B+tree. A read transaction takes a private copy of the root and registers itself, so pages that root references are not freed while it runs; freed pages are queued and reused only after readers that could reference them finish (redb `docs/design.md`, https://github.com/cberner/redb/blob/master/docs/design.md; also the 1.0 post, https://www.redb.org/post/2023/06/16/1-0-stable-release/, which says readers and writers do not block each other). `ReadTransaction` is `Send + Sync`, closes when it and all objects referencing it drop (https://docs.rs/redb/latest/redb/struct.ReadTransaction.html). Savepoints register as read transactions to preserve a snapshot and cost allocator-state copy of about 64 kB per GB of values at 4 kB pages; persistent savepoints survive restart, must be deallocated explicitly, and keep their pages from reclamation (design.md; the space-growth statement is the fetch tool's inference). +- What a redb read transaction gives for free: a consistent, lock-free view of the whole table as of the last commit before it began, for the lifetime of the handle, with no copy. Huginn's `pull_all` already uses it, but only to copy every envelope into a Vec and drop the transaction (`cultcache-rs lib.rs:2349-2350`). Nothing in the pages read gives a way to open a read transaction at an arbitrary older commit; older views exist only as held read transactions or savepoints (the pages describe no "open at txid" API; absence in the fetched text, not proof). +- What it does not give: decoded documents, a status index, or an ordinal filter. It would remove the in-process image copy and its refresh, not the decode and derivation work (inferred). +- Failure mode for cursors: one pinned transaction per live cursor-snapshot prevents page reuse, so the redb file grows while cursors are held, and there is no natural expiry for a cursor that an agent abandons (inferred from the reuse rule). +- Note a structural mismatch (inferred): redb's snapshots are per commit of the engine; Huginn's `as_of` is per receipt. Because each admitted batch is one compare-and-swap commit with its receipt in the same write transaction (`receipt.rs:268-270`), engine commits and receipt ordinals correspond 1:1, but redb does not expose the mapping and cannot be asked for "the view as of ordinal 12,345" after the fact. + +## 3. Immutable database values and persistent maps + +**Datomic.** Information accumulates and is never removed; databases are immutable values, so compute instances need no coordination for query and caches hold only immutable data (https://docs.datomic.com/whatis/architecture.html). Queries merge persistent index trees with an in-memory representation of recent change; new index trees are built occasionally by background indexing jobs (same page; the pages fetched did not use the terms novelty/memory index there). The operations page defines the memory index as transaction data logged but not yet in a disk index, triggers an indexing job at `memory-index-threshold` (default 32m), lets the transactor continue until `memory-index-max` (default 512m), and applies radical backpressure if novelty exceeds max before indexing finishes (https://docs.datomic.com/operation/capacity.html). +- `as-of` is a filter over the same indexes, not a stored copy; filtering "is not free"; entity-plus-time queries jump via EAVT then drop newer datoms; time-only questions would need a scan of the whole database (https://docs.datomic.com/reference/filters.html). The current db from a connection does no filtering, so "now" pays no history penalty. +- Relevance (inferred): this is the closest match to Huginn's data model: append-only, writes never overwritten, and `as_of` is a filter on a monotone position (Datomic `t`, Huginn receipt ordinal). The Datomic answer to "older snapshot" is not to keep versions; it is to keep every datom with its `t` and filter. Applied here: keep each document once, tagged with its introduction ordinal, and keep each resolution tagged with its ordinal; an `as_of` read filters by tag. A Datomic filter pays per touched datom, not per mind, because it goes through an index. +- Failure modes: the novelty/indexing split exists because the in-memory representation must be bounded; backpressure is the cost of the writer outrunning the indexer. Huginn's single-writer, low-rate admission is far from that regime (inferred). + +**Persistent (structural-sharing) maps in Rust.** +- `imbl`: clone is a refcount increment on the root; nodes are copied on write only when shared; `HashMap` is "almost neck and neck" with std, `OrdMap` about 2-3x slower than std on basic operations, `Vector` loses to `Vec` on memory and indexing; sole-owner in-place mutation is about an order of magnitude faster than immutable operations; default pointer is `Arc` (thread safe), `Rc` and `triomphe` options exist (https://docs.rs/imbl/latest/imbl/). +- `rpds`: `Vector`, `HashTrieMap`, `RedBlackTreeMap`; thread safety is opt-in via `_sync` constructors using `Arc`, which is "significantly slower to clone and drop than Rc"; mutable methods are faster when old versions are not needed (https://github.com/orium/rpds). +- Relevance (inferred): a persistent map gives one owner advancing a head version per admission and cheap retention of any older version by holding its root. Memory cost of N retained versions is the changed paths, not N copies. Hazards: per-version retention has no expiry unless something drops the root (a cursor table), and `OrdMap`/trie ops cost a constant factor over a plain map for reads. Because Huginn's history is a prefix chain, each version differs from its predecessor by a few documents, so path-copy cost per admission is small (inferred); but the same prefix property means a version could be reconstructed by filtering rather than retained (see Datomic). +- Handing a version to readers: `arc-swap` provides an atomically swappable `Arc` for read-mostly state, one writer publishing, many lock-free readers each keeping the version it loaded (https://docs.rs/arc-swap/latest/arc_swap/). The page notes the crate is niche, and that 1.9 strengthened memory orderings with probable cost; it did not discuss guard hold times. + +## 4. Event-sourced projections with a checkpoint + +**Marten (PostgreSQL).** An async daemon processes events in order; a high-water mark (HWM) advances contiguously and stops at gaps; each projection shard records its own progression; rebuilds via `projections --rebuild` or `RebuildProjectionAsync` (https://martendb.io/events/projections/async-daemon.html). Failure modes listed there: non-transactional PostgreSQL sequences burn numbers on rollback, creating gaps that stall the HWM; long-running or idle-in-transaction sessions can stall it; forcing a skip (`SkipStaleGapsDespiteLiveTransactionsAfter`, `AdvanceHighWaterMarkToLatestAsync`) can permanently skip events that commit later; slow pages beyond `StopAndDrainTimeout` get cancelled and reprocessed each shutdown; poison events go to a dead-letter table; migrating an inline projection to async needs downtime. +- Relevance (inferred): the gap hazard comes from multiple concurrent appenders with non-transactional sequence numbers. Huginn has one writer and a dense ordinal checked as `1..=N` in `head` (`receipt.rs:204-213`), so the HWM and gap machinery have no counterpart; the projection's checkpoint is simply "head ordinal applied". + +**EventStoreDB / Kurrent, Akka Projection.** The Kurrent projections page URL fetched returned 404 on the second path; the Akka overview page says only that an offset per event lets a projection resume after restart, with offsets kept in Cassandra, JDBC, R2DBC or Slick, and did not state delivery guarantees or rebuild procedure. Neither was verified further here. Not evidence for or against. + +**Fowler, "Event Sourcing".** State can be derived by replaying the log from a blank state, to any point; replay is "a slow process, particularly if there are many events"; snapshots of application state speed restarts and can be built in parallel; "Complete Rebuild" discards derived state and replays (https://martinfowler.com/eaaDev/EventSourcing.html). +- Relevance (inferred): Huginn is already its own event log (the receipts) and its own store (the envelopes). The projection here is derived from the same file that holds its checkpoint. On a crash or version change the image rebuilds from the envelopes (what `Reader::at` does today per read), so "rebuild on version change" is the existing code run once at open instead of per read. A persisted projection (a second store) would add a checkpoint that can disagree with the file (inferred: a second owner of the head ordinal). + +## 5. Incremental view maintenance for derived status + +**DBSP (Budiu, McSherry, Ryzhyk, Tannen).** A language for computation over streams, with an algorithm that incrementalizes arbitrary programs, covering relational queries, aggregation and recursion (https://arxiv.org/abs/2203.16684, abstract only; the PDF fetch returned unreadable binary, so claims about cost bounds were not verified; the fetch tool's general-knowledge recollection is that distinct and aggregation need integrated state that grows with input). +**Differential dataflow / Materialize arrangements.** Arrangements are shared indexed state serving many queries; logical compaction (`set_logical_compaction(frontier)`) declares that times before the frontier will no longer be queried, letting the trace merge them, after which reads of older times are not guaranteed; holding trace handles prevents compaction (https://timelydataflow.github.io/differential-dataflow/chapter_5/chapter_5_3.html). +- Relevance (inferred): Huginn's "status" is a two-level lookup (resolution subject -> closing resolution, `docs.rs:171-183`), not a recursive query. A general IVM engine is machinery well beyond that derivation. The transferable lesson is the compaction frontier: an indexed state that serves several times needs an explicit declaration of the oldest time still readable, otherwise it never shrinks. For Huginn the analogue is a cursor TTL. The simple incremental alternative the evidence supports is recompute-per-key-on-write: when a resolution with subject S is admitted, record `closed_by[S] = (resolution key, ordinal)`; `view_of` is then one map lookup. Status at `as_of` is `closed_by[S]` if its ordinal <= as_of (inferred, from the monotone-closure property in section 1; A8 and the Q19 cap admit at most one closing resolution per document, `docs.rs:168-170`, so no multi-valued history is needed for that case; resolutions that are themselves resolved make `in_force(Resolution)` depend on the same lookup, depth capped at two). + +## 6. What each option would mean here + +Common frame: single owner of the image (the `Mind`); an admission advances it after the redb commit; `Mind::view/query` read it. Memory figures are order-of-magnitude (inferred) with decoded document size unknown; assume 1-3 KB decoded per document: 11k docs about 10-35 MB; 1M docs about 1-3 GB. The current per-read build allocates and frees that on every call; the options differ in how much stays resident. + +| Option | Owner / advance | Old `as_of` served by | Resident memory | Deletes from Huginn | Main risk | +|---|---|---|---|---|---| +| A. Hold redb read transactions as the snapshot (per cursor) | Image stays derived per read from a held transaction; no resident image | Keep the transaction open per live cursor | Engine pages for pinned versions, grows with write churn while cursors live | `refresh()` re-pull and the `image` Vec could go (inferred); not decode or status scan | Unbounded cursor lifetime pins file growth (LMDB/redb rule); does not fix the 200 ms, which is decode plus derivation (inferred) | +| B. One resident decoded image at head, advanced per admission; old `as_of` rebuilt on demand | `Mind` owns `Image { docs by (kind,key) -> Held+ordinal, closed_by map, receipts }`; admission appends the batch's Held and closure entries after the redb commit; head reads clone nothing | Rebuild from envelopes for `as_of < head` only (today's `Reader::at` path, kept for the rare cursor) | Full decoded mind: 10-35 MB at 11k, 1-3 GB at 1M | per-read `from_image`, `AdmissionIndex::build`, double `head` receipt decode for head reads; admission's own `from_image` (`admission.rs:127`); the linear `held`/`raw_envelope` scans become map lookups | Two code paths (head fast, old slow) must agree; rebuild cost for old cursors stays O(N) | +| C. Resident image tagged with ordinals; `as_of` is a filter (Datomic-style) | Same as B, but every Held and every closure entry keeps its ordinal; an `as_of` read filters by ordinal tag | The same image: visible iff `ordinal <= as_of`; status iff `closed_by.ordinal <= as_of` | Same as B plus an ordinal per document and closure | Everything B deletes, plus the second (rebuild) path: one image, one read path, `Reader::at(as_of)` becomes a view parameter | A filtered scan for `query` is still O(N) per page unless ordered by ordinal; correctness relies on the prefix property (documents written once) | +| D. Persistent-map versions (imbl/rpds), one root retained per live cursor | Writer publishes a new root per admission (e.g. via `ArcSwap`); cursors hold a root | The retained root | Head image plus changed paths per retained version | Same as B; plus the rebuild path | Constant-factor read cost (OrdMap about 2-3x std); roots need an expiry; new dependency; solves a problem C does not have if history is a prefix (inferred) | +| E. Persisted projection with checkpoint (event-sourcing style) | A second store holding the image and a checkpoint ordinal, advanced by an applier | Replay from checkpoint or rebuild | Same resident image plus disk | Nothing; adds a second owner of head | Checkpoint can disagree with the file; rebuild-on-version-change machinery (Marten/Fowler) for no gain over rebuilding at open (inferred) | +| F. General IVM (DBSP/differential) | An IVM runtime maintains `status` | Compaction frontier | Arrangements | The status scan | Machinery far exceeding a two-level subject lookup; unverified cost bounds here | + +What B/C replace in the status derivation (inferred): `closing_resolution` / `in_force` scans (`docs.rs:171-206`) with a `closed_by` map written at admission time. The admission rule that already computes "which resolution closes this subject" for A8/Q19 (`docs.rs:165-183` is shared by admission and views, per its doc comment at `docs.rs:202-203`) would be the place that writes the entry, keeping one derivation. + +## 7. Open facts the fork will want measured (not measured here) + +- Where the 200 ms goes: envelope decode (`docs.rs:44-54`), receipt decode twice (`receipt.rs:204`, `query.rs:325`), or the status scan in `rows()` (`query.rs:431-439` over `docs.rs:171-183`). The brief's linear-in-mind figure does not distinguish these. +- Decoded size per document at 11k; the memory table above is an estimate. +- Whether `Selection` evaluation (`select`, `query.rs:486`) is itself linear per page, which an image would not fix. +- Cursor lifetime in practice (key is per process, `mind.rs:190`) and how many outstanding `as_of` values older than head exist at once; this decides whether "rebuild on demand for old as_of only" is rare enough. +- Whether the per-commit `refresh()` (`mind.rs:274-278`) is worth keeping once an image is advanced from the committed batch; it also serves the lost-CAS diff (`receipt.rs:272-283`). + +## Sources + +- Huginn: `F:\Projects\Huginn\crates\huginn-mind\src\{query,docs,mind,receipt,admission,index}.rs`; `F:\Projects\CultLib\packages\cultcache-rs\src\lib.rs` (lines 1975, 2349-2350). +- redb: https://github.com/cberner/redb/blob/master/docs/design.md ; https://www.redb.org/post/2023/06/16/1-0-stable-release/ ; https://docs.rs/redb/latest/redb/struct.ReadTransaction.html +- LMDB: https://raw.githubusercontent.com/LMDB/lmdb/mdb.master/libraries/liblmdb/lmdb.h +- PostgreSQL: https://www.postgresql.org/docs/current/transaction-iso.html ; https://www.postgresql.org/docs/current/functions-admin.html ; https://www.postgresql.org/docs/current/routine-vacuuming.html +- Datomic: https://docs.datomic.com/whatis/architecture.html ; https://docs.datomic.com/operation/capacity.html ; https://docs.datomic.com/reference/filters.html +- imbl: https://docs.rs/imbl/latest/imbl/ ; rpds: https://github.com/orium/rpds ; arc-swap: https://docs.rs/arc-swap/latest/arc_swap/ +- Marten: https://martendb.io/events/projections/async-daemon.html ; Akka Projection: https://doc.akka.io/libraries/akka-projection/current/overview.html ; Fowler: https://martinfowler.com/eaaDev/EventSourcing.html +- DBSP: https://arxiv.org/abs/2203.16684 ; differential dataflow: https://timelydataflow.github.io/differential-dataflow/chapter_5/chapter_5_3.html +- Not reached: Datomic "Time in Datomic" and indexing pages (404), Kurrent projections page (404/redirect), LMDB docs HTML (TLS error), DBSP PDF (binary). From 1e319f75f44654fc9bed7eab32572a3c14cc1fea Mon Sep 17 00:00:00 2001 From: Meta Vens Date: Mon, 26 Oct 2026 14:00:15 +0100 Subject: [PATCH 002/300] eureka-substrate map: upgrade read-back gate (probe timings, one gate in huginn-mind, cut sequence) Co-Authored-By: Claude Opus 5.5 --- docs/eureka-substrate-map.md | 71 ++++++++++++++++++++++++++++++++++++ 1 file changed, 71 insertions(+) diff --git a/docs/eureka-substrate-map.md b/docs/eureka-substrate-map.md index e9c61c4..d76270d 100644 --- a/docs/eureka-substrate-map.md +++ b/docs/eureka-substrate-map.md @@ -718,3 +718,74 @@ payload and metadata. Only the search request changes. score. It waits on 1 being deployed. 5. `gap-persona-state-not-indexed` stays with the persona-state upgrade (`huginn-upgrade-daemon-first`). + +## Upgrade read-back gate (Imagination, 2026-10-26) + +Ruling `eureka-substrate:ruling:readback-gate-and-materialized-image` part (1) +and follow-up `eureka-substrate:follow_up:upgrade-readback-unbounded`. Pinned: +Huginn `origin/main` `dd07485b4e69e6743825fec2f19825761ea446a6`, gamecult-ops +`origin/main` `b9e0109773eccea3215ed53c1af5ecccae37b543`. + +### Body facts (probe and source reads, 2026-10-26) + +- Probe, 2026-10-26 ~12:52 UTC, Yggdrasil, `ygg-verify.sh` `RUN_ON_HOST=1` + (image `eureka-verify-rust:b99d30ec441e`, 4 CPUs, 6 GiB, nice 10). A scratch + clone at `dd07485` plus one probe test in `query.rs`'s tests (scratch commit + `d93b400`, pushed only to the verify mirror) ran against a copy of the cold + copy `/var/backups/huginn-upgrade/dd07485b…/mind.redb` (sha256 + `814947b9…c5e0`, 245,370,880 bytes) inside the job's work dir; the copy was + deleted afterwards. 15,291 envelopes, 11,328 pipeline documents. + + | Step | debug | release | + | --- | --- | --- | + | `Mind::open` | 3.16 s | 0.58 s | + | decode, validate, re-key, re-prepare to stored bytes (all 11,328) | 1.14 s | 0.23 s | + | one `Mind::view` (one `Reader::at`) | 19.6 s | 1.05 s | + | one `Reader::at` | 20.4 s | 0.97 s | + | `held` + `view_of` for all 11,328 from that one `Reader` | 4.88 s | 0.39 s | + | whole test binary | 49.3 s | 3.35 s | + + So the deleted organ test's loop (one `Mind::view` per document) costs about + 11,328 x 20 s = 63 h in debug and 3.3 h in release: it never finishes, which + matches the 2 h hang. The one-snapshot gate costs open + byte check + + one `Reader::at` + the view loop: about 30 s debug, 2.5 s release at this + size. The runbook's deadline is 600 s. +- The view loop from one `Reader` is not strictly linear: `Reader::held` is a + linear find and `Docs::closing_resolution` scans every resolution, so it is + O(N x R) comparisons. It is the same cost `Reader::rows` pays on every query + today, measured at 4.9 s debug. The gate does not fix it; part (2) of the + ruling (the materialized image) owns it. +- `mind::tests::snapshot_admits_and_reads_back` (`mind.rs:789-878`) also + resolves six hard-coded idunn-watchdog findings and withdraws resolutions + whose withdrawal keys exceed 64 bytes: the 2026-10-01 key-bound migration, + done once. `eureka-pipeline` `tests::stored_documents_read_back` + (`lib.rs:2750-2822`) checks decode, validate, re-key and stored bytes, plus + resolvability loops and the same six findings ("still stuck"). + +### Rationale + +One gate, in `huginn-mind`. The leaf cannot hold it: `eureka-pipeline` has no +dependency on `huginn-mind` (the reverse holds), so it cannot see `view`. Two +gates overlapped on decode, validate and re-key; the leaf's byte check costs +1.1 s debug and moves into the organ's gate beside the view check. The leaf's +resolvability loops are pinned on synthetic keys by its bound tests (ruling +`key-bound`), and its six-finding list is history. + +The gate goes through the read path, not a copy of it: `Mind::view` is split +into `Reader::at` plus a new `Reader::view` (lookup, then `view_of`), and the +gate builds one `Reader` and calls `Reader::view` per document. A test that +re-spelled `held` and `view_of` would agree with its own copy. That split is +the only production edit; `Reader::at`, `held`, `view_of`, `rows`, `query` +and `rank` are unchanged, and the read path's cost stays with part (2). + +### Cut sequence + +1. readback-gate (Huginn): `cut-readback-gate.r1`. +2. ops-readback-deadline (gamecult-ops): `cut-ops-readback-deadline.r1`, + beside 1. Step 5 runs the one gate detached in container + `huginn-readback-$sha`, deadline 600 s, kill by name, old daemon restarted on + expiry or failure. It names 1's test, so an upgrade with it targets Huginn + main at or after 1's merge. +3. materialized-image (Huginn): a decoded, indexed image the daemon keeps and + advances per admission (ruling part 2). Mapped after prior art; not mapped + here. From 3f47de512264c50324759ad48559d91607fd01e2 Mon Sep 17 00:00:00 2001 From: Meta Vens Date: Wed, 28 Oct 2026 04:04:20 +0100 Subject: [PATCH 003/300] eureka-body map: connector-kit-lift probes (F26-F29), its cut line, path B refreshed Imagination imagination-kit-lift, session self-2026-10-28-eureka. Spec eureka-body:cut_spec:cut-connector-kit-lift.r1; questions connector-kit-relicense and codex-connector-fate. Co-Authored-By: Claude Opus 5.5 --- docs/eureka-body-map.md | 115 ++++++++++++++++++++++++++++++++-------- 1 file changed, 93 insertions(+), 22 deletions(-) diff --git a/docs/eureka-body-map.md b/docs/eureka-body-map.md index f47c118..f09716f 100644 --- a/docs/eureka-body-map.md +++ b/docs/eureka-body-map.md @@ -44,6 +44,13 @@ and "How a Self run is carried" is new. Specs: `cut-mind-self-run.r1`, `cut-mind-body-riders.r1`. Questions: `first-self-run-starter`, `self-run-permissions`. Probe outputs: `F:\Projects\eureka-handoff-aux-2026-10-22\imag-self-run\`. +Revised again on 2026-10-28 by Imagination (`imagination-kit-lift`, session +`self-2026-10-28-eureka`) at CodexConnector `fd7cec4`, Ghostlight `c43eb54` and ModelConnector +`1cf879f`, for cut `connector-kit-lift`. New facts F26-F29, new rationale "Where the kit lift's +cut line falls", path B refreshed. Spec: `cut-connector-kit-lift.r1`. Questions: +`connector-kit-relicense` (blocks the spec), `codex-connector-fate`. The baseline probe log was +a session scratchpad file and is not kept; its counts are in F26. + ## Pinned heads (fetched 2026-10-08 ~12:10 UTC) | Repo | Ref | Commit | Note | @@ -318,6 +325,48 @@ Each line: what was run, where, result. correctness holds under either delivery, and its operator verification probes both on Yggdrasil with her token. +- **F26 CodexConnector at origin/main `fd7cec4` builds and tests green** (2026-10-28, Yggdrasil, + ygg-verify rust image, `cargo build --features daemon --all-targets`, `cargo test --features + daemon`, `cargo test`): 46 lib + 1 bin tests with `daemon`, 11 lib tests without. `fd7cec4` is + F10's `3ddc12e` plus a README merge; F10's line numbers still hold. Files: `src/lib.rs` 1,958, + `daemon.rs` 1,180, `provider_backend.rs` 1,463, `idunn_health.rs` 1,396, `main.rs` 188, no + `vendor/`. CultLib pinned at `e171eca` (2026-09-04); 248 CultLib commits touch + `packages/cultcache-rs` or `packages/cultnet-rs` since, and the store API the connector uses + (`OwnedRedbMessagePackBackingStore`, `SingleFileMessagePackBackingStore`, `register_entry_type`, + `pull_all_backing_stores`, `get_all_with_keys`) still exists at CultLib `a4eae90`. +- **F27 What is provider-neutral and what is Codex** (source read, `fd7cec4`). Neutral: the contract + types and validation (`lib.rs:51-485`, `622-631`, `1351-1415`), the AES-256-GCM envelope and + four-byte big-endian framing (`633-710`, `824-952`), the one-request client (`711-833`), caller + admission and the CultCache replay service (`954-1313`), and the accept loop + (`daemon.rs:344-380`, `565-740`). The daemon calls its backend at exactly two points, + `readiness()` and `execute(&invocation) -> CodexTransportResult` (`daemon.rs:592-653`), so one + trait is the whole seam. Codex: `provider_backend.rs` (the pinned `codex app-server` child, + `auth.json`, Responses SSE parsing), the Responses lowering `canonical_responses_body` + (`lib.rs:487-620`), `CodexDaemonConfig` (`daemon.rs:47-171`, CultCache type + `gamecult.codex.connector_config.v1`, with `codex_executable` and `codex_home`), the managed + paths (`daemon.rs:453-536`), and the Idunn recipe (npm `@openai/codex` input). `idunn_health.rs` + is Idunn runtime presence with Codex-named constants (target `codex-connector`, capability + `gamecult.codex.subscription-inference`, env `CODEX_CONNECTOR_CULTNET_RUDP`). The request + itself carries OpenAI Responses fields (`reasoning_summary`, `service_tier`, + `previous_response_id`, `output_format_name`, `prompt_cache_key`). +- **F28 The wire is serde MessagePack, not CultNet** (source read). The envelope, invocation and + result are `rmp_serde` encodings of serde structs (`lib.rs:476-710`, client `748` and `765`, + daemon `637` and `652`); schema ids are strings compared by hand, not a CultNet schema. + CultCache holds only the replay record and the daemon config; CultNet is used only for Idunn + runtime presence and route observation. JSON: `parameters_json`, `output_schema_json` and tool + `arguments` are JSON strings checked by `require_json_object`; they are provider-facing (JSON + Schema and the provider's own tool-call arguments), a xenos boundary. Every other JSON use is + inside `provider_backend.rs` (HTTP body, SSE, app-server JSON-RPC, `auth.json`), also xenos. +- **F29 Consumers and license** (git grep at each origin/main). Ghostlight `c43eb54` pins + `codex-connector` at `68fe94b` as a library (`Cargo.toml:18`, no `daemon` feature) and imports + the contract and client in `crates/ghostlight/src/controllers.rs:34-37` and seven other files; + Epiphany's pins are archive (target invariant `nothing-runs-from-epiphany`). gamecult-ops + `idunn/yggdrasil/bindings/codex-connector.toml.in` names origin GitHub CodexConnector; Idunn + `docs/migration.md:49-55` says retire codex-connector, and `:163` calls its recipe stale. + CodexConnector's project code is PolyForm Noncommercial 1.0.0 (`LICENSE` section 3), all 33 + commits by Meta Vens; ModelConnector is AGPL-3.0-only at `1cf879f` (LICENSE and a 26-line + planned README). + ## The model page (step 0b) One row per persistent kind this body introduces or changes. Rows for the thirteen @@ -493,6 +542,22 @@ never trip (finding `breaker-counts-every-document`). Two attributions are avail Idunn's migration doc says Codex is dead (F9). The kit is lifted out of CodexConnector, and the Codex backend's existing tests are the only proof that the lift is behaviour neutral. Keeping the backend compiling costs nothing; deploying it is not proposed. +Superseded 2026-10-28 by "Where the kit lift's cut line falls" below: a wire parity test +against the v2 library proves neutrality without porting the backend. + +### Where the kit lift's cut line falls (cut `connector-kit-lift`) + +The lift moves only what F27 calls provider-neutral, behind one `ProviderBackend` trait, and +proves itself by a parity test against the v2 library at `fd7cec4` rather than by porting the +Codex backend. Three reasons. The Codex daemon has no consumer (F29, ruling +`no-return-to-openai-supplier`), so porting it buys nothing the parity test does not prove more +directly. Leaving it out makes the cut independent of question `codex-connector-fate`. And it keeps +B2 a pure move (verbatim code, `Codex` stripped from Rust names, wire bytes unchanged), so Soul can +falsify the move alone, and B3 owns every contract change: ids, the reasoning item, spend, and the +Responses-shaped request fields (F27). This supersedes "Why the kit keeps its Codex backend" above, +whose premise (the backend's tests are the only proof of neutrality) the parity test removes. +Idunn presence (`idunn_health.rs`) is not lifted with it: it is deployment, and is checked +against Idunn's current presence contract when a connector binary is first deployed (B4). ### How a Self run is carried (cut `mind-self-run`) @@ -553,32 +618,38 @@ operator's token, then A6 and the bring-up; A3, A4 and A1 run beside it): rule in the actuator. Waits on A2 (ruling `run-measure-in-body-store`). Not on the path to the first run: A2 runs every run fresh, which the ruling allows (a cold cache is fresh). -**B. To "Eureka speaks a Persona turn"** (critical path: B1, B2, B3, B4, B5, B6): +**B. To "Eureka speaks a Persona turn"** (critical path: B2, B3, B4, B5, B6; refreshed +2026-10-28, Imagination `imagination-kit-lift`): -1. Self decision for follow-up `body-campaign-repos`. The mechanism is ruled - (`eureka-substrate:ruling:campaign-repo-set-home`) and mapped - (`eureka-substrate:cut_spec:cut-campaign-repos.r1`, Huginn, and `cut-skill-campaign-repos.r1`, - Eureka; both unreported). Self must decide three things: dispatch those two cuts now and - upgrade the daemon; the connector kit repo's name and host (the follow-up proposes - `GameCult/ModelConnector`; no `body-repos-on-forge` ruling exists in the mind, so either the - doctrine default, GitHub `GameCult`, or a question); and admit `target:r2` with the repo set - plus `GameCult/Idunn` and the kit. That fills the 8-repo bound (6 + 2). -2. `connector-kit-lift` (kit repo): waits on B1 (ruling `connector-kit-home`). -3. `connector-contract-v3` with `connector-reasoning-item` and `connector-spend` (kit): wait on - B2; rulings `connector-opaque-state-item`, `connector-owner`; follow-up - `connector-reasoning-state-cheap-providers`. -4. `claude-connector` or `openai-compatible-connector` (kit): waits on B3, and on which - provider serves Persona turns first (rulings `no-return-to-openai-supplier`, follow-up - `connector-reasoning-state-cheap-providers`). Self raises that question when B3 is mapped. +1. Done 2026-10-28: `target:r2` carries the eight-repo set with GameCult/Idunn and + GameCult/ModelConnector (follow-up `body-campaign-repos` Fixed by Huginn `f8eaf50`); the kit + repo is GameCult/ModelConnector on the forge (`connector-kit-repo-modelconnector`). +2. `connector-kit-lift` (ModelConnector): spec `cut_spec:cut-connector-kit-lift.r1`. Blocked on + question `question:connector-kit-relicense` (F29: CodexConnector is PolyForm Noncommercial). + Waits on nothing else. + - Beside it, off the critical path: `codex-connector-park` or `codex-onto-kit` + (CodexConnector, gamecult-ops). Waits on question `question:codex-connector-fate` and on B2. +3. `connector-contract-v3` with `connector-reasoning-item` and `connector-spend` (ModelConnector): + waits on B2. Ids become `gamecult.model.*.v3`; the `ReasoningState` item (ruling + `connector-opaque-state-item`); `max_cost_usd` on the invocation, cost on the outcome, a + `Budget` refusal and a durable fail-closed ledger (ruling `connector-owner`); the + provider-neutral request shape, settling F27's Responses fields; whether the wire becomes + CultNet typed documents instead of serde MessagePack (F28, GameCult doctrine); `wire_parity_v2` + deleted. Follow-up `connector-reasoning-state-cheap-providers` is read when the item is mapped. +4. `claude-connector` (ModelConnector; ruling `persona-first-connector-claude`): waits on B3. Its + map raises how it authenticates and whose credential it holds against invariant + `token-out-of-her-reach` (rulings `connector-owner`, `self-run-token-reach`). It also carries + the connector's own config document, the CultLib bump from `e171eca` (F26), and Idunn presence + rebuilt from `idunn_health.rs` against Idunn's current contract, if Idunn deploys it. 5. `mind-persona` (Huginn): the Persona organ, one connector request per turn (`persona-turn-connector`), Persona state through Huginn (landed), the start-Self tool (A6), traces with provenance from day one (`collect-with-provenance-gate-at-training`). Waits on B4 and on where traces live (question `persona-trace-retention` was withdrawn to be re-asked with the trace store; re-ask it when B5 is mapped). -6. `persona-mouth` (Bifrost or gamecult-ops): where a turn is heard. Waits on B5 and on which - mouth first: Matrix (`chat-platform`, not yet deployed; follow-up - `identity-minting-moves-to-bifrost` precedes the chat rollout) or Discord - (`body-includes-discord`). Self raises it when B5 is mapped. +6. `persona-mouth` (Bifrost or gamecult-ops): Matrix (ruling `persona-mouth-matrix`; Discord + stays out, `gamecult-chat:ruling:discord-taints-training`). Waits on B5 and on the Matrix + deployment (`chat-platform`, `chat-stand-up-now`; follow-up + `identity-minting-moves-to-bifrost` precedes the chat rollout). **C. Beside both paths:** @@ -591,8 +662,8 @@ operator's token, then A6 and the bring-up; A3, A4 and A1 run beside it): and eureka-substrate Cut 12 leasing) and the swarm scheduler (C3). 3. `swarm-ctb-scheduler` (Huginn): rulings `mind-scheduling-ctb`, `swarm-dial-one-budget`, `swarm-scheduler-one-shot`, `turn-cost-measured`. Waits on A8 (its cost source). -4. `idunn-job-workload` (Idunn): the deletion line of the hand install. Waits on B1 (Idunn - joins the campaign); due before the first hosted member (`mind-units-templated-interim`). +4. `idunn-job-workload` (Idunn): the deletion line of the hand install. Idunn joined the + campaign in `target:r2`; due before the first hosted member (`mind-units-templated-interim`). 5. `wire-names-rename` (Huginn, Eureka): before the first hosted mind (`wire-names-rename-before-hosting`). Waits on nothing; schedule before C2. 6. `doctrine-repoint` (Eureka, F12), then `epiphany-archive` (last). From 725f5a5a751844483b4d8d45a5be3b2bbbe82fa6 Mon Sep 17 00:00:00 2001 From: Meta Vens Date: Wed, 28 Oct 2026 07:27:59 +0100 Subject: [PATCH 004/300] Research: connector contract v3 prior art and probes (reasoning state, spend, wire) Serves eureka-body cut connector-contract-v3 and questions connector-v3-wire, connector-v2-coexistence. Co-Authored-By: Claude Opus 5.5 --- .../connector-contract-v3-prior-art.md | 222 ++++++++++++++++++ 1 file changed, 222 insertions(+) create mode 100644 docs/research/connector-contract-v3-prior-art.md diff --git a/docs/research/connector-contract-v3-prior-art.md b/docs/research/connector-contract-v3-prior-art.md new file mode 100644 index 0000000..4056cd0 --- /dev/null +++ b/docs/research/connector-contract-v3-prior-art.md @@ -0,0 +1,222 @@ +# Connector contract v3: prior art and probes + +Serves: cut `eureka-body:cut_spec:cut-connector-contract-v3.r1` (map line B3, +GameCult/ModelConnector), questions `eureka-body:question:connector-v3-wire` and +`eureka-body:question:connector-v2-coexistence`, and the map line `connector-spend` +(B3b), which is mapped from section 2 when it is next. Also answers follow-up +`eureka-body:follow_up:connector-reasoning-state-cheap-providers` (section 1d) and +`v3-ciphertext-as-bytes` (section 4). + +Sections 1-3 are an Eyes facts pass (2026-10-28, session `self-2026-10-28-eureka`), fetched +from current primary docs; UNVERIFIED marks what it could not confirm. Section 4 is +Imagination's probes on Yggdrasil against ModelConnector `ec0519d` and CultLib `a4eae90`. +Section 5 is what Imagination drew from them for the cut. Recommendations live in the +questions; this page is their evidence. + + +Facts only, each with a pointer. Fetched 2026-10-28 from current docs. UNVERIFIED marks what I could not confirm. +Provider docs have moved hosts: docs.claude.com -> platform.claude.com/docs; platform.openai.com -> developers.openai.com/api/docs. +Model names in current Anthropic docs: Fable 5.1, Mythos 5.1, Opus 5.5, Sonnet 5.5, Haiku 5.5 (and older). + +## 1. Opaque / reasoning state a caller must replay + +### 1a. Anthropic Messages API +Pointers: T = https://platform.claude.com/docs/en/build-with-claude/thinking ; TS = .../thinking-troubleshooting ; P = .../preserved-thinking (linked from T, NOT fetched) + +- Thinking arrives as `thinking` content blocks (fields `thinking`, `signature`) and `redacted_thinking` blocks (field `data`, opaque, encrypted). [T "Thinking encryption", "Redacted thinking blocks"] +- `signature` holds the full encrypted thinking; API uses it to verify blocks were generated by Claude; opaque, "don't interpret or parse". Streaming: arrives as `signature_delta` in a `content_block_delta` just before `content_block_stop`. [T "Thinking encryption"] +- `display: "omitted"` (default on newer models) returns thinking blocks with empty `thinking` string and populated `signature`; such a block is complete and must be replayed; text put into the empty field is ignored not rejected. [TS "The thinking field is empty"; T "Preserving thinking blocks" Note] +- Replay rules [T "Thinking with tool use", "Preserving thinking blocks"]: + - Required: within a tool-use turn, pass thinking blocks back, complete and unmodified, with the `tool_use` block they accompanied. + - Recommended: across turns pass everything back. Allowed: outside tool use, omit prior turns' thinking. + - A tool-use loop is one assistant turn. Consecutive thinking blocks in the latest assistant message must match what the model generated: no reorder, edit, partial drop; includes `redacted_thinking`. + - Filtering by `block.type == "thinking"` silently drops `redacted_thinking` and breaks the protocol. [T "Redacted thinking blocks" Tip] +- Errors [TS]: + - Modified/dropped in latest assistant message: 400 `invalid_request_error`, message contains "`thinking` or `redacted_thinking` blocks in the latest assistant message cannot be modified". + - Signature present but fails verify (truncated/altered/empty): 400 "Invalid `signature` in `thinking` block" (ends there; no prefix clause). + - Fable 5.1, Opus 5.5, Sonnet 5.5, Haiku 5.5: replayed block valid only while `system`, `tools` and preceding messages are unchanged ("prefix check"): 400 "...The block is bound to a different conversation. Remove the block, or set `thinking.block_binding.prefix_mismatch_behavior` to "drop_block"." Enforced by default for accounts created on/after 2026-08-31; older accounts only when the request sets that field. Option needs beta header `thinking-binding-controls-2026-08-01`; response then carries an `input_transformations` array listing dropped blocks. [T "Preserved thinking"; TS] + - Mid-turn toggle of thinking: no error; API silently disables thinking for that request and may strip blocks. [T "Thinking with tool use"] + - Manual mode (`type:"enabled"`): final assistant turn must start with a thinking block; `type:"enabled"` is 400 on 4.7+ models. [extended-thinking page "Turn structure in manual mode"; TS table] +- Earlier-turn thinking stripped or kept: per model. Keep all prior turns: Opus 4.5+, Sonnet 4.6+, Haiku 5.5, Fable/Mythos 5/5.1. Keep last turn only: earlier Opus/Sonnet and Haiku through 4.5; API strips automatically on pass-back. Kept blocks bill as input. Override: `clear_thinking_20251015` context-editing strategy. [T "Thinking block preservation by model", "Preserving thinking blocks"] +- Portability: + - Across platforms: `signature` values are compatible across Claude API, Bedrock, Google Cloud (Google managed); exception: Sonnet 5.5 and Haiku 5.5 blocks work only in the producing account or a linked account. [T "Thinking encryption"; P#account-bound-thinking not fetched] + - Across models: "readable only by the model that produced it and certain other models"; API drops unreadable blocks without error and without billing. Fable 5.1 reads Opus 5 and (Claude API) Opus 5.5 blocks; Opus 5/5.5 do NOT read Fable 5.1 blocks; Fable 5.1 and Mythos 5.1 read each other only. Switching to them keeps reasoning; away from them drops it. [T "Thinking block preservation by model: Switching models", "Preserved thinking"] +- Prompt caching [T "Thinking and prompt caching"]: + - Thinking config and resolved effort are rendered into the prompt; changing mode, `budget_tokens` or effort invalidates message-level breakpoints (and may invalidate tool/system ones). + - Thinking blocks in a tool loop are cached automatically with tool results and count as input when read from cache. + - Last-turn-only models: after a non-tool-result user message, earlier thinking is stripped from context/cache. + - 1-hour cache suggested for long thinking sessions. +- Billing: thinking tokens billed as output; `usage.output_tokens_details.thinking_tokens` reports them (final `message_delta` when streaming). [extended-thinking page "Budget rules and tuning"] + +Stop reasons (https://platform.claude.com/docs/en/build-with-claude/handling-stop-reasons, first 100k chars read): +- `end_turn`, `max_tokens`, `stop_sequence`, `tool_use`, `pause_turn` (server-tool loop hit iteration limit, default 10; resend assistant content to continue), `refusal` (HTTP 200, `stop_details` names policy category, null for other reasons), `model_context_window_exceeded` (treat as truncated). +- Streaming: `stop_reason` is null in `message_start`, appears in `message_delta`. `max_tokens` can end in an incomplete `tool_use` block. + +Usage object (https://platform.claude.com/docs/en/build-with-claude/prompt-caching "Usage fields"): +- `input_tokens` = tokens after last cache breakpoint (uncached); `cache_creation_input_tokens` = 5m + 1h writes; `cache_read_input_tokens`; `cache_creation{ephemeral_5m_input_tokens, ephemeral_1h_input_tokens}`; `output_tokens`. total_input = cache_read + cache_creation + input_tokens. +- `output_tokens_details.thinking_tokens` (above). `server_tool_use` and `service_tier` fields: only seen via fetch-tool general-knowledge answer, API reference page was truncated before the schema: UNVERIFIED against docs. Token counts depend on tokenizer: Claude 4.7+ models (incl. Fable/Mythos 5/5.1) count ~30% more tokens for the same text. [token-counting page "Supported models", "Token counts on Claude Fable and Claude Mythos models"] +- Min cacheable prompt: 512 (Fable/Mythos/Opus 5.5/Opus 5/Sonnet 5.5/Haiku 5.5), 1024 (Opus 4.8, Sonnet 5/4.6/4.5), 2048 (Opus 4.7, Mythos Preview), 4096 (Opus 4.6/4.5, Haiku 4.5). Lookback 20 block positions. [prompt-caching page] + +### 1b. OpenAI Responses API +Pointer: https://developers.openai.com/api/docs/guides/reasoning (via fetch summary) +- Reasoning items appear in `output`; opaque, no reasoning text returned (optional `summary`). Example item id prefix `rs_`; page gives no other id rules. +- Stateless mode (`store:false` or ZDR org): reasoning items include `encrypted_content` by default; `include: ["reasoning.encrypted_content"]` still accepted as legacy/no-op. +- Replay: stored -> `previous_response_id`. Manual -> pass prior output items. Function calling: pass back reasoning items returned with the last function call plus outputs; with several calls in a row, pass back all reasoning, call and output items since last user message. `all_turns` with `store:false`: keep every output item (encrypted reasoning, assistant `phase`) and replay full history. `reasoning.context` in the response reports effective mode (`current_turn`/`all_turns`). +- Other models: reusable only within the same model family; switching families, API omits incompatible reasoning even with `all_turns` (GPT-5.6 Sol/Terra/Luna interchange, not GPT-5.5). +- Usage: `usage.output_tokens_details.reasoning_tokens`, `usage.input_tokens_details.cached_tokens`; reasoning billed as output. + +### 1c. Google Gemini +Official pages fetched were thin or truncated: +- https://ai.google.dev/gemini-api/docs/thought-signatures redirects to thinking guide; page that did render describes the Interactions API: signatures are "encrypted representations of the model's internal reasoning"; limited to `thought` steps and built-in tool steps; stateless mode "MUST always resend all `thought` blocks exactly as they were received"; stateful (`store:true`, `previous_interaction_id`) server handles it; on `generateContent` signatures "can be attached to any part, such as living inside `functionCall` parts or the final part of a response"; streaming `thought_signature` delta arrives in last delta before `step.stop`; a thought block may carry only a signature. +- https://ai.google.dev/gemini-api/docs/gemini-3 : stateless callers must include thought blocks with signatures (summary only). +- `generateContent` function-calling rules (first functionCall part in parallel calls, each step in sequential, 400 on missing, `skip_thought_signature_validator` dummy): UNVERIFIED in official docs (Vertex page docs.cloud.google.com/vertex-ai/generative-ai/docs/thought-signatures body was truncated in fetch). +- Observed third-party evidence (not official): error text "Unable to submit request because function call `X` in the N. content block is missing a `thought_signature`" and reports that only the first 1-2 parallel calls get signatures, https://discuss.ai.google.dev/t/gemini-3-flash-preview-inconsistent-thought-signature-generation-in-parallel-function-calls-causes-400-errors-and-potential-silent-data-loss/118936 ; thread quotes Google doc line "If you use the official SDKs and standard chat history, thought signatures are handled automatically". Text turns without a signature reportedly don't 400 (apiyi FAQ, https://docs.apiyi.com/en/faq/gemini-thought-signature-missing, third party). "Signature present but fails validation" is a distinct 400 reported when route/model/key changes mid-conversation (same FAQ, third party). + +### 1d. OpenAI-compatible providers (cheap-providers follow-up) +| Provider | Field | Form | Must replay in tool loop | Across turns | Pointer | +|---|---|---|---|---|---| +| DeepSeek (deepseek-flash, deepseek-v4-pro on page) | `reasoning_content` (sibling of content) | plaintext | Yes. With `tools`: omitting it -> documented 400. Each sub-request in a turn's loop must carry that turn's reasoning even if no tool call | With tools: pass back all previous turns' reasoning (concatenated into context). Without tools: ignored if sent, not carried | https://api-docs.deepseek.com/guides/thinking_mode | +| Moonshot Kimi | `reasoning_content` at level of `content` | plaintext | Within a task keep all and send each request; append complete assistant message unchanged. Page names no error for omission | Per model: kimi-k3.1-pro/k3 require it multi-turn; k2.7-code always keeps; k2.6 drops by default unless `thinking.keep:"all"` | https://platform.kimi.ai/docs/guide/use-kimi-k2-thinking-model (page covers k2.6/k2.7/k3, not "K2 thinking" naming) | +| Zhipu GLM (z.ai) | `reasoning_content`; request `thinking:{type, clear_thinking}` | plaintext; page silent on signing | Return with the assistant message that made tool calls; interleaved thinking on by default | "Preserved thinking": `clear_thinking:false` keeps earlier turns (default on for Coding Plan endpoint, off standard); when on, must return complete unmodified; reorder/edit "may degrade" and lowers cache hits. GLM-5.3 cannot disable thinking | https://docs.z.ai/guides/capabilities/thinking-mode | +| Qwen (Alibaba Model Studio) | `reasoning_content`; `enable_thinking`, `thinking_budget`, `preserve_thinking` | plaintext | Page doesn't say; links to function-calling docs | `preserve_thinking:true` (qwen3.8-max, 3.7-max/plus, 3.6-plus, some Kimi) makes model read historical reasoning, billed as input; omission not an error | https://www.alibabacloud.com/help/en/model-studio/deep-thinking | +| MiniMax (page = M3.1-Flash; M2 not on that page) | `reasoning_content` (OpenAI) / `thinking` blocks (Anthropic SDK) | plaintext | Append entire assistant message incl. `reasoning_content`; "essential" for best performance; no benchmark or error stated. Thinking is optional per turn | n/a | https://platform.minimax.io/docs/guides/text-m2-function-call . M2-specific `` tags / `reasoning_details` / `reasoning_split`: UNVERIFIED (not on fetched page) | +- None of these pages describe signatures or encryption; the DeepSeek 400 is the only documented hard failure. + +### 1e. Multi-provider libraries +- LiteLLM (https://docs.litellm.ai/docs/reasoning_content): `reasoning_content` string on message "returned across all providers"; `thinking_blocks` list (Anthropic only; `type`, `thinking`, `signature`) must be included in the assistant message with `tool_calls` when returning tool results; `provider_specific_fields` unspecified (Bedrock example `reasoningContentBlocks[].reasoningText{signature,text}`). Known failure: Anthropic 400 when `thinking_blocks` missing ("assistant message doesn't start with a thinking block"). OpenAI-compatible clients (LibreChat, Open WebUI, Vercel AI SDK) ignore `thinking_blocks` and lose them. Anthropic->DeepSeek switch needs `drop_params=True`. `litellm.modify_params=True` disables `thinking` for a turn when blocks are missing. +- OpenRouter (https://openrouter.ai/docs/guides/best-practices/reasoning-tokens): `reasoning_details` array at `choices[].message` / `choices[].delta`; common fields `id` (string|null), `format`, `index`; types `reasoning.summary` (`summary`), `reasoning.text` (`text`, optional `signature`), `reasoning.encrypted` (`data`; may stream as `[REDACTED]`). `format` values: unknown, openai-responses-v1, azure-openai-responses-v1, bedrock-openai-responses-v1, bedrock-xai-responses-v1, xai-responses-v1, meta-responses-v1, anthropic-claude-v1 (default), google-gemini-v1. Must preserve full consecutive sequence, no rearrange/modify. `reasoning` string field for plain-string models; `reasoning_content` alias. Doc claims switching OpenAI/Anthropic works without code change. +- Vercel AI SDK: `sendReasoning` (default true) controls whether reasoning is included in requests; Anthropic provider surfaces `providerMetadata.anthropic.inputTransformations`; `blockBinding.prefixMismatchBehavior` option; `providerOptions` escape hatch; between-tools thinking blocks "sent back unchanged on the next step" (https://ai-sdk.dev/providers/ai-sdk-providers/anthropic, https://ai-sdk.dev/docs/ai-sdk-core/reasoning). Thinking `signature`/`redactedData` parts: UNVERIFIED (not on fetched pages). +- Cross-provider replay failure evidence: LiteLLM page above (Anthropic<->DeepSeek, UI clients dropping fields); Gemini third-party reports of dropped `thoughtSignature` by OpenAI-compatible clients (https://discuss.ai.google.dev/t/openai-api-compatibility-broken-due-to-thought-signature-on-gemini-3-pro-preview/109823 , title only, body not fetched). + +## 2. Spend accounting and budget enforcement + +### 2a. Usage and money +- Anthropic usage: tokens only (fields in 1a); no money field seen in docs fetched. Cost must be computed from pricing page. UNVERIFIED that the full Message schema has no cost field (schema page truncated). +- OpenRouter: `usage.cost` = total charged in credits; `usage.cost_details.upstream_inference_cost` (BYOK only via generation endpoint); `usage.prompt_tokens_details.cached_tokens` and `.cache_write_tokens`; `usage.completion_tokens_details.reasoning_tokens`; `usage:{include:true}` deprecated/no effect, full usage always present; streaming usage in last SSE message. https://openrouter.ai/docs/guides/guides/usage-accounting +- OpenAI: tokens (1b). + +### 2b. LiteLLM +- Per-request cost in header `x-litellm-response-cost`; spend log field is `spend` (USD). The cost-tracking page does not name a `response_cost` field or `model_prices_and_context_window.json`; says it uses the "model cost map" and can sync from GitHub. https://docs.litellm.ai/docs/proxy/cost_tracking (file name `model_prices_and_context_window.json`: UNVERIFIED on page). Zero-priced usage on non-zero-rate model logs spend 0 with WARNING and counter `litellm_zero_cost_requests_total`. +- Budgets: `max_budget` on keys (`/key/generate`; `budget_limits` multiple windows), users, teams, team members, customers; global `litellm_settings.max_budget`; `budget_duration` ("30s/30m/30h/30d"; unset = never resets); reset check default every 10 min. https://docs.litellm.ai/docs/proxy/users +- Timing: with budget reservation (default on) LiteLLM estimates max cost from the request and reserves before calling; rejects pre-provider if reservation exceeds budget; after response, reservation is replaced with actual cost. Batch: reservation released, final cost recorded at completion. Zero-cost models allowed over budget. +- Errors vary by scope: `ExceededTokenBudget` (key), `budget_exceeded` code 400 or 429 (per-model), `ExceededBudget` with 401 (customer), 429 (agent sessions); `fail_closed_budget_enforcement: true` -> 503 when budget cannot be verified. +- No DB: "Budgets require a database". Global `max_budget` fails open (warning, requests continue); key/team/user budgets unavailable ("No connected db."). + +### 2c. OpenRouter limits +https://openrouter.ai/docs/api-reference/limits +- Per-key `limit`, `limit_reset`, `limit_remaining`. HTTP 402 with `error.metadata.limit_source`: `openrouter_key_limit`, `openrouter_credits`, `openrouter_in_flight_budget` (wait `Retry-After`). +- In-flight budget (certain prepaid accounts): request whose estimated cost doesn't fit is rejected pre-provider even with positive balance; estimate = input tokens + completion tokens allowed by `max_tokens` (fixed per-request cap if unset), excludes per-request fees/plugins/images; charged on finish. +- Negative balance causes errors incl. free models. + +### 2d. Pre-call reservation / bound patterns +- LiteLLM reservation (2b) and OpenRouter in-flight estimate (2c) both = estimated input + max_tokens-bounded output, hold, settle on actual. +- Anthropic token counting: `POST /v1/messages/count_tokens`, same inputs as Messages (system, tools, images, PDFs, thinking blocks); returns `{input_tokens}`; free; separate rate limit (5,000/10,000/20,000 RPM by tier Start/Build/Scale); result is an "estimate" and may include system-added tokens not billed; no caching logic; rejects server tools, MCP connector, url/file image/document sources; uses tokenizer of passed model. https://platform.claude.com/docs/en/build-with-claude/token-counting +- Portkey / Helicone / cloud billing ledgers: NOT searched (budget). UNVERIFIED. + +### 2e. Anthropic pricing (https://platform.claude.com/docs/en/about-claude/pricing, $/MTok: input / 5m write / 1h write / cache read / output) +- Fable 5.1 and Mythos 5.1: 10 / 12.50 / 20 / 0.25 / 50 (read = 0.025x) +- Opus 5.5: 4 / 5 / 8 / 0.20 / 20 (read = 0.05x) +- Opus 5, 4.8, 4.7, 4.6, 4.5: 5 / 6.25 / 10 / 0.50 / 25 +- Sonnet 5.5: 2 / 2.50 / 4 / 0.10 / 10 (read = 0.05x) +- Sonnet 5: 2 / 2.50 / 4 / 0.20 / 10 ; Sonnet 4.6, 4.5: 3 / 3.75 / 6 / 0.30 / 15 +- Haiku 5.5: prompts <=100k tokens 0.10 / 0.125 / 0.20 / 0.01 / 0.50 ; prompts >100k 0.50 / 0.625 / 1 / 0.05 / 2.50 (long-context tier keyed on prompt size) +- Haiku 4.5: 1 / 1.25 / 2 / 0.10 / 5 +- Multipliers: 5m write 1.25x, 1h write 2x, read 0.1x base unless noted above. `inference_geo:"us"` = 1.1x on all categories (4.6+; 400 on earlier models). Batch API discount: table exists on page (lines show batch input/output at about half), exact multiplier not extracted. Multipliers stack with batch and residency. +- Opus 5.5 / Sonnet 5.5 / Haiku 5.5 / Fable 5.1 listed as current at fetch time. + +## 3. Wire choice for a typed local RPC + +### 3a. MCP (spec revision 2026-07-28 is "latest" at fetch) +- JSON-RPC 2.0, UTF-8. Standard transports: stdio (newline-delimited) and Streamable HTTP (POST per message; JSON or request-scoped SSE reply). https://modelcontextprotocol.io/specification/latest/basic/transports +- Custom transports allowed; those over reliable byte streams (Unix sockets, TCP) "SHOULD reuse the stdio framing" (same page). +- Versioning: latest revision has NO initialize handshake; every request carries protocol version in `_meta.io.modelcontextprotocol/*` (HTTP also `MCP-Protocol-Version` header); unsupported -> error -32022 with `data{supported[], requested}`; client retries; `server/discover` MUST be implemented. Legacy (<= 2025-11-25) used `initialize` negotiation. Dual-era servers MAY serve both on one endpoint, selecting by how the client opens (modern `_meta` vs `initialize`). https://modelcontextprotocol.io/specification/latest/basic/lifecycle (rendered as "Versioning and Compatibility") +- Binary: base64 strings paired with `mimeType` (`ImageContent.data`, `AudioContent.data`, `BlobResourceContents.blob`). +- Error codes: -32700, -32600, -32601, -32602, -32603 (JSON-RPC); -32020 HEADER_MISMATCH, -32021 MISSING_REQUIRED_CLIENT_CAPABILITY, -32022 UNSUPPORTED_PROTOCOL_VERSION. https://modelcontextprotocol.io/specification/2026-07-28/schema + +### 3b. gRPC / protobuf (https://protobuf.dev/programming-guides/proto3/) +- Unknown fields preserved on parse and re-serialized (lost when going through JSON or field-by-field copy). Never reuse field numbers; reserve removed numbers/names. Adding fields, removing (no reuse), adding enum values are wire-safe; changing a field number or moving a field into an existing oneof is not. Unknown enum values preserved (open enums store int; closed enums, e.g. Java, special-case). +- `bytes`: arbitrary byte sequence (length limit garbled in fetch). +- `package` prevents name clashes; page does NOT frame packages as a versioning mechanism ("versioning by package names" claim: UNVERIFIED on this page). + +### 3c. MessagePack / rmp-serde +- msgpack spec: bin 8/16/32 = 0xc4/0xc5/0xc6 with 8/16/32-bit BE length; str family separate (0xd9-0xdb, fixstr 0xa0-0xbf); ext 0xc7-0xc9 and fixext 0xd4-0xd8, app ext types 0-127, -1 = Timestamp; array 0x90-0x9f/0xdc/0xdd; map 0x80-0x8f/0xde/0xdf. Spec recommends deserializers accept bin and str8 in minor releases, serializers distinguish in major. https://github.com/msgpack/msgpack/blob/master/spec.md +- rmp-serde: serde derive treats `Vec` as an array of values (about 50% overhead per docs); wrap with `serde_bytes` or use `Serializer::with_bytes` to emit bin. Re-exports `to_vec` and `to_vec_named` (the page does not explain the struct encoding difference; array-vs-map: UNVERIFIED here). Unknown-field handling not covered. https://docs.rs/rmp-serde/latest/rmp_serde/ +- Cap'n Proto: not searched. UNVERIFIED. + +### 3d. Keeping two wire versions alive +- MCP dual-era server (3a): one endpoint serves both generations; discriminator is how the first message opens (per-request `_meta` vs `initialize`); client probes `server/discover` (stdio) or attempts modern request and inspects 400 body (HTTP) and caches the result per server process/origin; a modern-only server SHOULD name its supported versions in the error it returns to `initialize`; compatibility matrix in the versioning page. +- Protobuf field-number discipline (3b). +- Version byte / frame tag, side-by-side schema ids, generic dual-stack guidance: no source fetched. UNVERIFIED. + +## Not done (budget) +Portkey, Helicone, Cap'n Proto, OpenAI pricing, Gemini usage fields (`usageMetadata.thoughtsTokenCount`, `cachedContentTokenCount`), Anthropic Messages API reference schema (page too large), P preserved-thinking page, Bedrock/Vertex-specific usage differences. + +## 4. Probes (Imagination, Yggdrasil ygg-verify rust image, 2026-10-28) + +Scratch clone of ModelConnector at `ec0519d` plus a probe-only commit (never pushed to the +forge). Baseline: `cargo test --workspace` and `--features model-connector/daemon` green, 77 +tests passed in all. + +- P2 Decoding the real v2 frame `tests/fixtures/wire_v2/envelope_invocation.mp` (the bytes + CodexConnector `fd7cec4` produced): 735-byte frame for 429 bytes of ciphertext (209 of them + >= 128), ratio 1.713. First byte `0x96`: a positional six-element array. It decrypts under + `parity-distinct-test-key` to caller `parity-caller`, request `request-1`, model `gpt-5.4`, + 4 input items, 1 tool, schema `gamecult.codex.transport_invocation.v2`. +- P2 The same envelope with the ciphertext as MessagePack `bin` (`serde_bytes`): 526 bytes, + ratio 1.226 (the rest is ids and nonce). A 1 MiB all-high ciphertext: v2 frame 2,097,251 + bytes, bin frame 1,048,675; the kit's `frame_bytes_for_payload(1 MiB)` = 2,097,750. +- P2 The v2 `TransportEnvelope` type decodes a bin-encoded ciphertext and compares equal + (rmp-serde reads bin into `Vec`). Encoding is the only change bin needs. +- P3 `ProviderRequest` positional 267 bytes (`0xdc` array16), named (`to_vec_named`) 590 + bytes (`0xde` map16). The v2 type decodes the named encoding and compares equal. +- P4a A struct with one added field: positional encoding fails to decode in the old type + ("array had incorrect length, expected 3"); named encoding decodes with the field ignored. + So positional MessagePack cannot add a field without breaking every reader, and named + encoding drops unknown fields silently unless the type denies them. +- P4b A v2 `InputItem` decoding a `ReasoningState` variant fails: "unknown variant". A v2 + peer cannot read a v3 transcript at all; there is no partial compatibility to preserve. +- P5 A v3 envelope schema id under v2 `decrypt_invocation`: `Err(Envelope)`, before any + decryption. A v2 daemon given a v3 envelope drops the connection; it never answers. +- PC CultNet carriage. `CultNetMessage::OperationRequest` (cultnet-rs at CultLib `a4eae90`) + carrying the sealed envelope (named, bin ciphertext, ids at 256 bytes) as its + `messagepack-base64` payload, encoded with `encode_cultnet_message_to_vec(.., + CultNetSchemaV0)` and decoded back, round-trips equal. Frame over ciphertext: 4.45 at 429 + bytes (fixed overhead), 1.354 at 64 KiB, 1.335 at 1 MiB, 1.334 at 8 MiB; encode 6.5 ms and + decode 1.0 ms at 1 MiB (release). The worst case is deterministic (base64 is exactly + 4*ceil(n/3)), where v2's integer array is up to 2.0x and depends on the bytes. + +Source reads behind the wire question (not probes): +- CultNet's `TcpFramed` transport protocol (`contracts.rs:112-119`) uses the same four-byte + big-endian length prefix as the kit (`framing.rs:6-15`; kit `envelope.rs:152-184`). +- `validate_message` requires `payload_encoding == "messagepack-base64"` on operation + messages (`contracts.rs:780`): CultNet has no binary operation payload today. +- CultNet's security options hold one connection key per server (`security.rs:51-75`); the + kit binds one key and epoch per caller and seals ids into the AES-GCM AAD + (`service.rs:19-65`, `envelope.rs:245-253`). +- Huginn already speaks this shape: `huginn-mind/src/envelope.rs` carries named MessagePack + as base64 in `cultnet.operation_request.v0`, checks `service_id`, `payload_schema` and the + operation against the decoded payload, and publishes its JSON Schema with `schemars`. + +## 5. What the cut takes from this + +- Reasoning item granularity follows the providers: Anthropic requires the latest assistant + turn's thinking blocks complete, unmodified and in place beside their `tool_use`; OpenRouter + carries `reasoning_details` per block with an index. So one `ReasoningState` item per + provider block, in transcript order, holding the connector's encoding of the whole block + (Anthropic: `thinking` text and `signature`, or `redacted_thinking` data, since a summarized + block must come back with both). +- Foreign state: every provider that tags reasoning by model drops what it cannot read across + completed turns (Anthropic drops unreadable blocks silently; OpenAI omits other families). + Only the in-flight tool exchange fails hard (Anthropic 400, DeepSeek 400). So the kit refuses + foreign state only after the last user text, and drops it before the backend sees it + elsewhere, counting the drop on the receipt (as Anthropic's `input_transformations` does). +- Anthropic's prefix check binds a block to the unchanged system, tools and earlier messages: + the caller's verbatim replay (ruling connector-opaque-state-item) is necessary, and the Claude + connector's lowering must be deterministic (B4). +- The cheap providers (1d) all keep plaintext `reasoning_content` and want it replayed inside a + tool loop; the item stays opaque to callers either way. +- Spend (B3b): LiteLLM and OpenRouter both bound a request before the call as estimated input + plus `max_tokens` times the output price, hold it, and settle to actual; LiteLLM without a + database fails open, which ruling connector-owner refuses. Anthropic returns tokens only, so + cost is the connector's arithmetic over a price table it owns. v3 therefore makes + `max_output_tokens` required, so the bound exists. From 3b64b03d223ac77d18793f2db0007e9639b28a6d Mon Sep 17 00:00:00 2001 From: Meta Vens Date: Wed, 28 Oct 2026 07:35:19 +0100 Subject: [PATCH 005/300] eureka-body map: connector-contract-v3 probes (F30-F31), its cut line, B3 split into B3a-c Spec cut-connector-contract-v3.r2; questions connector-v3-wire, connector-v2-coexistence; follow-up cultnet-binary-operation-payload. Research 725f5a5. Co-Authored-By: Claude Opus 5.5 --- docs/eureka-body-map.md | 88 ++++++++++++++++++++++++++++++++++------- 1 file changed, 73 insertions(+), 15 deletions(-) diff --git a/docs/eureka-body-map.md b/docs/eureka-body-map.md index f09716f..cf48665 100644 --- a/docs/eureka-body-map.md +++ b/docs/eureka-body-map.md @@ -51,6 +51,15 @@ cut line falls", path B refreshed. Spec: `cut-connector-kit-lift.r1`. Questions: `connector-kit-relicense` (blocks the spec), `codex-connector-fate`. The baseline probe log was a session scratchpad file and is not kept; its counts are in F26. +Revised again on 2026-10-28 by Imagination (`imagination-contract-v3`, session +`self-2026-10-28-eureka`) at ModelConnector `ec0519d` and CultLib `a4eae90`, for cut +`connector-contract-v3`. New facts F30-F31 (probes on Yggdrasil), new rationale "Where the +contract v3 cut line falls", B3 split into B3a-B3c. Spec: `cut-connector-contract-v3.r2`. +Questions: `connector-v3-wire`, `connector-v2-coexistence` (both raised in r1, which r2 +superseded only to split two context windows; they block r2). Follow-up: +`cultnet-binary-operation-payload`. Research: Huginn `725f5a5` +`docs/research/connector-contract-v3-prior-art.md`. + ## Pinned heads (fetched 2026-10-08 ~12:10 UTC) | Repo | Ref | Commit | Note | @@ -367,6 +376,26 @@ Each line: what was run, where, result. commits by Meta Vens; ModelConnector is AGPL-3.0-only at `1cf879f` (LICENSE and a 26-line planned README). +- **F30 The v2 wire decoded, and what a v3 change costs it** (2026-10-28, Yggdrasil, ygg-verify + rust image, scratch clone of ModelConnector `ec0519d` plus a probe-only commit; research + `725f5a5` section 4, probes P2-P5). Baseline 77 tests green with and without `daemon`. The real + v2 frame `envelope_invocation.mp` is 735 bytes for 429 bytes of ciphertext (ratio 1.713; a + positional six-array, `0x96`) and decrypts to Ghostlight-shaped content. Ciphertext as + MessagePack `bin` makes it 526 (1.226); at 1 MiB all-high, v2 2,097,251 vs bin 1,048,675. The + v2 type reads bin and named encodings unchanged. Positional encoding fails on any added field + ("array had incorrect length"); named encoding drops an unknown field silently; a v2 + `InputItem` refuses an unknown variant; a v3 envelope id is refused by v2 before decryption, + so a v2 daemon drops a v3 caller's connection and never answers. +- **F31 CultNet can carry the kit as an operation service** (same job, cultnet-rs at CultLib + `a4eae90`; source reads in research section 4). The sealed envelope as the + `messagepack-base64` payload of `cultnet.operation_request.v0` round-trips through + `encode_cultnet_message_to_vec`/`decode_cultnet_message_from_slice`: 1.354x at 64 KiB, 1.335x + at 1 MiB, 1.334x at 8 MiB (deterministic), 6.5 ms encode and 1.0 ms decode at 1 MiB, release. + CultNet's `TcpFramed` frame is the kit's 4-byte big-endian frame; operation payloads may only + be `messagepack-base64` (`contracts.rs:780`); CultNet security holds one key per server where + the kit binds one per caller with ids in the AAD. Huginn's `huginn.mind` is already this + shape (`huginn-mind/src/envelope.rs`). + ## The model page (step 0b) One row per persistent kind this body introduces or changes. Rows for the thirteen @@ -559,6 +588,33 @@ whose premise (the backend's tests are the only proof of neutrality) the parity Idunn presence (`idunn_health.rs`) is not lifted with it: it is deployment, and is checked against Idunn's current presence contract when a connector binary is first deployed (B4). +### Where the contract v3 cut line falls (cut `connector-contract-v3`) + +B3 as one cut (wire, request shape, reasoning item, spend) reads the whole kit and rewrites most +of it, past half of a Hands pass. It splits along owners, each falsifiable alone, all landing +before B4 so no consumer ever sees an intermediate v3: + +- B3a `connector-contract-v3` (spec r2): the wire and the documents. One codec owner (named + MessagePack, unknown fields refused), one carriage owner (a CultNet operation, if + `connector-v3-wire` goes as recommended), typed outcomes and refusals, the request made + provider-neutral, v2 deleted, schemas published from the types. It folds in + `v3-ciphertext-as-bytes` and the parts of `kit-contract-test-debt` in code it rewrites, and + takes the CultLib bump from B4 because cultnet-rs must be at the rev cultcache-rs is. +- B3b `connector-reasoning-item`, settled here so its spec is quick (research section 5): one + `ReasoningState {provider, model, state: bytes}` per provider block, in transcript order, as + both an `InputItem` and an event, the bytes the connector's encoding of the whole block; a kit + function that turns a result's events into transcript items, so no caller branches on provider; + `ProviderBackend` declares its provider tag; the service refuses foreign state after the last + user text (`ForeignReasoningState`) and drops it elsewhere before the backend sees it, counting + the drop on the receipt. A backend receives a call that cannot show it a foreign item. +- B3c `connector-spend`: price table in the connector's config, `max_cost` on the invocation, + a worst-case reservation (input estimate plus `max_output_tokens` times the output price) held + in the replay store and settled to actual, cost on the outcome, a `Budget` refusal, fail + closed when the ledger cannot be written (ruling `connector-owner`; LiteLLM and OpenRouter + both reserve this way). Its map raises how the connector's per-caller ceiling relates to her + run's `budget_usd` (invariant `run-is-the-grant`, ruling `turn-cost-measured`) and whether + amounts are integer micro-dollars or the mind's canonical decimal string. + ### How a Self run is carried (cut `mind-self-run`) The run document is the grant (`open_and_launch`); the unit takes only `--instance` @@ -618,29 +674,31 @@ operator's token, then A6 and the bring-up; A3, A4 and A1 run beside it): rule in the actuator. Waits on A2 (ruling `run-measure-in-body-store`). Not on the path to the first run: A2 runs every run fresh, which the ruling allows (a cold cache is fresh). -**B. To "Eureka speaks a Persona turn"** (critical path: B2, B3, B4, B5, B6; refreshed +**B. To "Eureka speaks a Persona turn"** (critical path: B2, B3a-c, B4, B5, B6; refreshed 2026-10-28, Imagination `imagination-kit-lift`): 1. Done 2026-10-28: `target:r2` carries the eight-repo set with GameCult/Idunn and GameCult/ModelConnector (follow-up `body-campaign-repos` Fixed by Huginn `f8eaf50`); the kit repo is GameCult/ModelConnector on the forge (`connector-kit-repo-modelconnector`). -2. `connector-kit-lift` (ModelConnector): spec `cut_spec:cut-connector-kit-lift.r1`. Blocked on - question `question:connector-kit-relicense` (F29: CodexConnector is PolyForm Noncommercial). - Waits on nothing else. - - Beside it, off the critical path: `codex-connector-park` or `codex-onto-kit` - (CodexConnector, gamecult-ops). Waits on question `question:codex-connector-fate` and on B2. -3. `connector-contract-v3` with `connector-reasoning-item` and `connector-spend` (ModelConnector): - waits on B2. Ids become `gamecult.model.*.v3`; the `ReasoningState` item (ruling - `connector-opaque-state-item`); `max_cost_usd` on the invocation, cost on the outcome, a - `Budget` refusal and a durable fail-closed ledger (ruling `connector-owner`); the - provider-neutral request shape, settling F27's Responses fields; whether the wire becomes - CultNet typed documents instead of serde MessagePack (F28, GameCult doctrine); `wire_parity_v2` - deleted. Follow-up `connector-reasoning-state-cheap-providers` is read when the item is mapped. +2. Landed 2026-10-28: `connector-kit-lift` (ModelConnector main `ec0519d`, spec + `cut_spec:cut-connector-kit-lift.r1`, relicensed by ruling `connector-kit-relicense-agpl`). + - Beside it, off the critical path: `codex-connector-park` (CodexConnector, gamecult-ops; + ruling `codex-connector-park`). +3. `connector-contract-v3` (ModelConnector), split three ways (rationale "Where the contract v3 + cut line falls"): + - a. `connector-contract-v3`: spec `cut_spec:cut-connector-contract-v3.r2`. Waits on B2 + (landed, `ec0519d`) and questions `question:connector-v3-wire` and + `question:connector-v2-coexistence`. + - b. `connector-reasoning-item` (ModelConnector): waits on B3a. Reads follow-up + `connector-reasoning-state-cheap-providers` (answered in research `725f5a5` section 1d). + - c. `connector-spend` (ModelConnector): waits on B3b (same files). 4. `claude-connector` (ModelConnector; ruling `persona-first-connector-claude`): waits on B3. Its map raises how it authenticates and whose credential it holds against invariant `token-out-of-her-reach` (rulings `connector-owner`, `self-run-token-reach`). It also carries - the connector's own config document, the CultLib bump from `e171eca` (F26), and Idunn presence - rebuilt from `idunn_health.rs` against Idunn's current contract, if Idunn deploys it. + the connector's own config document and Idunn presence rebuilt from `idunn_health.rs` against + Idunn's current contract, if Idunn deploys it (the CultLib bump moved to B3a). Its lowering + to the Messages API must be deterministic: Anthropic binds a replayed thinking block to the + unchanged system, tools and earlier messages (research `725f5a5` section 1a, prefix check). 5. `mind-persona` (Huginn): the Persona organ, one connector request per turn (`persona-turn-connector`), Persona state through Huginn (landed), the start-Self tool (A6), traces with provenance from day one (`collect-with-provenance-gate-at-training`). Waits on From bedfea44ecdb9f472a1b08101e3dcd007fda3247 Mon Sep 17 00:00:00 2001 From: Meta Vens Date: Wed, 28 Oct 2026 12:34:19 +0100 Subject: [PATCH 006/300] eureka-body map: connector-reasoning-item probe (F32), B3b placed by its spec Co-Authored-By: Claude Opus 5.5 --- docs/eureka-body-map.md | 33 +++++++++++++++++++++++++++++++-- 1 file changed, 31 insertions(+), 2 deletions(-) diff --git a/docs/eureka-body-map.md b/docs/eureka-body-map.md index cf48665..d7b84ba 100644 --- a/docs/eureka-body-map.md +++ b/docs/eureka-body-map.md @@ -60,6 +60,13 @@ superseded only to split two context windows; they block r2). Follow-up: `cultnet-binary-operation-payload`. Research: Huginn `725f5a5` `docs/research/connector-contract-v3-prior-art.md`. +Revised again on 2026-10-28 by Imagination (`imagination-reasoning-item`, session +`self-2026-10-28-eureka`) at ModelConnector `bf958fc` (B3a landed), for cut +`connector-reasoning-item`. New fact F32 (probe on Yggdrasil), the B3b bullet of "Where the +contract v3 cut line falls" made exact, path B line 3b given its spec. Spec: +`cut-connector-reasoning-item.r1`. No question: the ruling and research section 5 settle every +choice the cut makes. Research reused unchanged. + ## Pinned heads (fetched 2026-10-08 ~12:10 UTC) | Repo | Ref | Commit | Note | @@ -395,6 +402,21 @@ Each line: what was run, where, result. be `messagepack-base64` (`contracts.rs:780`); CultNet security holds one key per server where the kit binds one per caller with ids in the AAD. Huginn's `huginn.mind` is already this shape (`huginn-mind/src/envelope.rs`). +- **F32 The reasoning variants cost the v3 wire nothing it has pinned** (2026-10-28, Yggdrasil + ygg-verify rust image, scratch commit `af9fb74` on ModelConnector `bf958fc`, never pushed). + `InputItem::ReasoningState { provider, model, state }` and + `TransportEventPayload::ReasoningState { state }`, `state` under `serde_bytes` with the + envelope's `binary_schema`: all 83 unit tests and both `wire_golden_v3` tests pass with every + committed golden byte-identical; only `published_schemas` fails (the new variant, expected). + The item encodes as a one-key map whose `state` is MessagePack `bin` (`c4 03 ff 80 01` for three + bytes); a 1 MiB state encodes in 1,048,623 bytes (47 bytes of overhead); the schema shows + `state` as `format: binary`. So the receipt field `foreign_reasoning_dropped` and the new event + in the transported golden are the only wire changes B3b makes to existing fixtures. Source + reads at `bf958fc`: `ProviderBackend::execute` receives the caller's whole + `TransportInvocation` (`serve.rs:29`, via `ExecutionClaim::invocation`, `service.rs:115-119`), + and every backend writes its own `receipt.provider` (`fixtures.rs:63-83`, test backends + `serve.rs:945-962`, `service.rs:1074-1091`): today nothing stops a backend from seeing foreign + state or mis-tagging its own. ## The model page (step 0b) @@ -607,6 +629,11 @@ before B4 so no consumer ever sees an intermediate v3: `ProviderBackend` declares its provider tag; the service refuses foreign state after the last user text (`ForeignReasoningState`) and drops it elsewhere before the backend sees it, counting the drop on the receipt. A backend receives a call that cannot show it a foreign item. + Placed by spec `cut-connector-reasoning-item.r1` (F32): the event carries the bytes alone and + the kit stamps provider and model from the receipt when it builds transcript items, and the + receipt's provider and drop count are derived by the service (`ProviderCall::transported`, + gated in `complete`), so a backend can neither see foreign state nor mis-tag its own. Only a + `Completed` result yields transcript items: a partial thinking block is never replayed. - B3c `connector-spend`: price table in the connector's config, `max_cost` on the invocation, a worst-case reservation (input estimate plus `max_output_tokens` times the output price) held in the replay store and settled to actual, cost on the outcome, a `Budget` refusal, fail @@ -689,8 +716,10 @@ operator's token, then A6 and the bring-up; A3, A4 and A1 run beside it): - a. `connector-contract-v3`: spec `cut_spec:cut-connector-contract-v3.r2`. Waits on B2 (landed, `ec0519d`) and questions `question:connector-v3-wire` and `question:connector-v2-coexistence`. - - b. `connector-reasoning-item` (ModelConnector): waits on B3a. Reads follow-up - `connector-reasoning-state-cheap-providers` (answered in research `725f5a5` section 1d). + - b. `connector-reasoning-item` (ModelConnector): spec + `cut_spec:cut-connector-reasoning-item.r1` at `bf958fc`; B3a has landed, so it is ready to + dispatch. Follow-up `connector-reasoning-state-cheap-providers` is answered by research + `725f5a5` section 1d and needs no kit rule (every provider's state stays opaque bytes). - c. `connector-spend` (ModelConnector): waits on B3b (same files). 4. `claude-connector` (ModelConnector; ruling `persona-first-connector-claude`): waits on B3. Its map raises how it authenticates and whose credential it holds against invariant From 8c59b5a0ca8b6571c0946ae791313853eca62473 Mon Sep 17 00:00:00 2001 From: Meta Vens Date: Wed, 28 Oct 2026 13:40:07 +0100 Subject: [PATCH 007/300] eureka-body map: connector-spend probe (F33), B3c placed by its spec, B4 riders carried Co-Authored-By: Claude Opus 5.5 --- docs/eureka-body-map.md | 58 ++++++++++++++++++++++++++++++++++++++++- 1 file changed, 57 insertions(+), 1 deletion(-) diff --git a/docs/eureka-body-map.md b/docs/eureka-body-map.md index d7b84ba..3a552e0 100644 --- a/docs/eureka-body-map.md +++ b/docs/eureka-body-map.md @@ -67,6 +67,14 @@ contract v3 cut line falls" made exact, path B line 3b given its spec. Spec: `cut-connector-reasoning-item.r1`. No question: the ruling and research section 5 settle every choice the cut makes. Research reused unchanged. +Revised again on 2026-10-28 by Imagination (`imagination-connector-spend`, session +`self-2026-10-28-eureka`) at ModelConnector `1d51e71` (B3b landed), for cut `connector-spend`. +New fact F33 (probes on Yggdrasil), the B3c bullet of "Where the contract v3 cut line falls" +made exact, path B line 3c given its spec and B4's riders carried. Spec: +`cut-connector-spend.r1`. Question: `connector-spend-ceiling` (blocks the spec). Follow-up: +`connector-spend-readout`. Research reused unchanged (`725f5a5` section 2 is the ceiling +question's evidence). + ## Pinned heads (fetched 2026-10-08 ~12:10 UTC) | Repo | Ref | Commit | Note | @@ -417,6 +425,22 @@ Each line: what was run, where, result. and every backend writes its own `receipt.provider` (`fixtures.rs:63-83`, test backends `serve.rs:945-962`, `service.rs:1074-1091`): today nothing stops a backend from seeing foreign state or mis-tagging its own. +- **F33 The kit holds no spend today, and its store fails closed but locks out readers** + (2026-10-28, Yggdrasil ygg-verify rust image, scratch commit `79e14b8` on ModelConnector + `1d51e71`, never pushed; source reads at `1d51e71`). Baseline: 94 lib tests with `daemon`, 37 + without, plus 4 `published_schemas`, 4 `reasoning_state` and 3 `wire_golden_v3`, all green. + P1: a `CacheBackingStore` whose `push` fails makes `CultCache::put` return the error and leaves + nothing in memory (`get` is `None`), so a reservation written with `put` before execution is + either durable or refused. `begin` already returns `ServiceError::ReplayStore` before + `live.insert` and before the backend runs (`service.rs:367-369`), so the replay store is a + fail-closed ledger by construction. P2: a second `OwnedRedbMessagePackBackingStore::new` on a + held path fails "already has an active owner", in the same process and from a child process, + so no root stop hook can read the connector's store while the connector runs. Source reads: + `CallerConfig` and `CallerAdmission` carry no spend field (`serve.rs:105-114`, + `service.rs:19-28`); `Usage` is tokens only and the receipt has no cost (`contract.rs:193-204`, + `465-479`); `TransportOutcome::Failed` carries no usage (`524-527`); replay records are never + deleted or pruned (no `delete` on the store in `service.rs`); `TransportService::open` takes a + path, so a test cannot inject a failing store (`service.rs:181-182`). ## The model page (step 0b) @@ -641,6 +665,30 @@ before B4 so no consumer ever sees an intermediate v3: both reserve this way). Its map raises how the connector's per-caller ceiling relates to her run's `budget_usd` (invariant `run-is-the-grant`, ruling `turn-cost-measured`) and whether amounts are integer micro-dollars or the mind's canonical decimal string. + Placed by spec `cut-connector-spend.r1` (F33): + - The ledger is the connector's replay store, not her mind and not mind-body's control store. + The rulings settle it: `brake-dial-root-store-now` keeps the spend ledger with the wallet, + outside what she merges; its question names "the connector's max_cost_usd and ledger + (another user)" as an enforcement point outside her writable environment; `turn-cost-measured` + names the connector ledger as the Persona turn's cost source; `connector-owner` makes it + durable and fail-closed. `run-is-the-grant` binds her organs, and the connector is the + wallet's, not hers. The reservation lives in the replay record itself, so reserving and + claiming the request are one `put` (P1): there is no second record to drift. + - Amounts are integer micro-USD (`u64`), prices micro-USD per million tokens, cost rounded up + once per request. Not a fork: exact arithmetic needs integers or a decimal, cultcache-rs has + no decimal (follow-up `cultcache-rs-decimal-and-nofollow`), and a canonical decimal budget of + at most six places converts exactly at the Persona organ (B5). + - The input bound is the named-MessagePack length of the provider request plus a per-model + overhead from the price table: a byte-level BPE token is at least one byte, so text cannot + exceed it (unprobed against a real tokenizer; B4's rider). It is priced at the dearest input + rate the request's cache retention allows. + - Two refusals, not one `Budget`: `CostBound` (the worst case exceeds the invocation's own + `max_cost_micro_usd`, decided by the invocation alone, before the replay lookup) and + `SpendCeiling` (the caller's window cannot hold the reservation, after the replay lookup, so + a recorded answer is always replayed). OpenRouter tells its limit sources apart the same way. + - What bounds the caller's total is question `connector-spend-ceiling`: `max_cost` is written by + her own Persona organ, which she may rewrite (`she-owns-her-body`), so it is not a wallet + control on its own. ### How a Self run is carried (cut `mind-self-run`) @@ -720,7 +768,8 @@ operator's token, then A6 and the bring-up; A3, A4 and A1 run beside it): `cut_spec:cut-connector-reasoning-item.r1` at `bf958fc`; B3a has landed, so it is ready to dispatch. Follow-up `connector-reasoning-state-cheap-providers` is answered by research `725f5a5` section 1d and needs no kit rule (every provider's state stays opaque bytes). - - c. `connector-spend` (ModelConnector): waits on B3b (same files). + - c. `connector-spend` (ModelConnector): spec `cut_spec:cut-connector-spend.r1` at `1d51e71`. + B3b has landed; it waits on question `question:connector-spend-ceiling`. 4. `claude-connector` (ModelConnector; ruling `persona-first-connector-claude`): waits on B3. Its map raises how it authenticates and whose credential it holds against invariant `token-out-of-her-reach` (rulings `connector-owner`, `self-run-token-reach`). It also carries @@ -728,6 +777,13 @@ operator's token, then A6 and the bring-up; A3, A4 and A1 run beside it): Idunn's current contract, if Idunn deploys it (the CultLib bump moved to B3a). Its lowering to the Messages API must be deterministic: Anthropic binds a replayed thinking block to the unchanged system, tools and earlier messages (research `725f5a5` section 1a, prefix check). + Riders carried from B3c and the B3 Soul passes: the price table lives in that config document + (one `ModelPrice` per allowed model, a missing one refused at open); if B3c's byte bound + refuses real turns, the backend supplies a provider count instead (Anthropic `count_tokens`, + research section 2d); the turn-cost charge needs a spend readout because the store is held + exclusively (F33 P2, follow-up `connector-spend-readout`); the tool-exchange boundary is + confirmed against Anthropic's rules (follow-up `claude-tool-exchange-boundary`); and whatever + of follow-up `kit-contract-test-debt` B3a-c did not kill is killed in code B4 touches. 5. `mind-persona` (Huginn): the Persona organ, one connector request per turn (`persona-turn-connector`), Persona state through Huginn (landed), the start-Self tool (A6), traces with provenance from day one (`collect-with-provenance-gate-at-training`). Waits on From f2376231641f6472fec4bca6745778f9a7d85b0d Mon Sep 17 00:00:00 2001 From: Meta Vens Date: Wed, 28 Oct 2026 18:58:14 +0100 Subject: [PATCH 008/300] research: Claude connector prior art and probes (B4: credential terms, workspace limits, thinking across tool use, token counting) Co-Authored-By: Claude Opus 5.5 --- docs/research/claude-connector-prior-art.md | 189 ++++++++++++++++++++ 1 file changed, 189 insertions(+) create mode 100644 docs/research/claude-connector-prior-art.md diff --git a/docs/research/claude-connector-prior-art.md b/docs/research/claude-connector-prior-art.md new file mode 100644 index 0000000..e2e2151 --- /dev/null +++ b/docs/research/claude-connector-prior-art.md @@ -0,0 +1,189 @@ +# Claude connector: prior art and probes + +Serves: map line B4 `claude-connector` (GameCult/ModelConnector, ruling +`eureka-body:ruling:persona-first-connector-claude`), its cut spec +`eureka-body:cut_spec:cut-claude-backend.r1` (B4a), question +`eureka-body:question:claude-connector-credential`, and follow-ups +`eureka-body:follow_up:claude-tool-exchange-boundary` (section 4), +`claude-byte-bound-live-check` (section 5). Builds on Huginn `725f5a5` +`docs/research/connector-contract-v3-prior-art.md` (sections 1a, 2d, 2e), which is not repeated. + +Fetched 2026-10-28 by Imagination (`imagination-claude-connector`, session +`self-2026-10-28-eureka`) from current primary docs; section 8 is probes on Yggdrasil. UNVERIFIED +marks what the docs did not say. No Anthropic API call was made with any credential. + +## 1. Which credential a third-party program may use + +Source: https://code.claude.com/docs/en/legal-and-compliance ("Authentication and credential use"). + +- OAuth "is intended exclusively for purchasers of Claude Free, Pro, Max, Team, and Enterprise + subscription plans and is designed to support ordinary use of Claude Code and other native + Anthropic applications." +- Developers building products or services, the Agent SDK included, "should use API key + authentication through Claude Console or a supported cloud provider". Anthropic does not permit + third parties to route requests through Free, Pro or Max plan credentials, and developers may not + collect, store or intermediate Claude.ai credentials or session tokens. +- The restriction does not limit how a customer provisions its own API keys, for example in a + secrets manager or machine image for its own authorized users, provided the usage is billed to the + key owner and not resold or intermediated. +- An end user may still sign in to the unmodified Claude Code binary with their own subscription. + "Advertised usage limits for Pro and Max plans assume ordinary, individual usage of Claude Code + and the Agent SDK." Anthropic "may [enforce] without prior notice". + +Reading: a connector daemon that calls the Messages API with the operator's subscription OAuth token +is the routing the page forbids. Driving the unmodified `claude` binary with her subscription is the +permitted Claude Code path (the Self-run unit, ruling `self-run-token-reach`), but it is Claude +Code's harness, not a Messages request the connector lowers (section 6). + +## 2. What an API key can be capped by, provider-side + +Source: https://platform.claude.com/docs/en/api/rate-limits, https://platform.claude.com/docs/en/api/errors. + +- Two limit kinds: monthly spend limits and rate limits (RPM, ITPM, OTPM per model class), enforced + at the organization, with user-configurable limits per workspace. +- "You can't set spend limits on the default Workspace." Workspace limits are set per limiter type; + organization limits always apply too. The `anthropic-workspace-id` response header names the + workspace a key resolved to. +- Tier monthly spend caps: Start $500, Build $1,000, Scale $200,000. Reaching the tier cap returns + 429 `rate_limit_error` with `error.details.error_code: "enforced_spend_limit_reached"` and no + `retry-after`; retries fail until the next month. +- A spend limit you set returns 400 `invalid_request_error`, message starting "You have reached your + specified API usage limits" (or "...specified workspace API usage limits"). +- Error types: 400 invalid_request_error, 401 authentication_error, 402 billing_error, + 403 permission_error, 404, 409, 413 request_too_large (32 MB Messages limit, Cloudflare answers + before the API), 429 rate_limit_error, 500 api_error, 504 timeout_error, 529 overloaded_error. + Every response carries a `request-id` header. The docs do not say whether an errored request is + billed: UNVERIFIED. +- Rate limits are a token bucket; only uncached input (`input_tokens` + `cache_creation_input_tokens`) + counts toward ITPM on current models; `max_tokens` does not count toward OTPM. Headers + `anthropic-ratelimit-{requests,tokens,input-tokens,output-tokens}-{limit,remaining,reset}` and + `retry-after` (not on every 429). +- Long requests: use streaming for long-running requests; the SDK refuses non-streaming requests + expected to exceed 10 minutes; idle connections may be dropped by networks. + +## 3. How proxies hold provider credentials + +- LiteLLM proxy (https://docs.litellm.ai/docs/proxy/virtual_keys): callers hold LiteLLM virtual + keys (`/key/generate`), each limited to a model list and carrying budget and rate limits; the + provider key sits in the proxy's `config.yaml` `litellm_params.api_key` or an `os.environ/` + reference. The page does not say callers can ever see provider keys, and does not describe + at-rest encryption of provider keys (UNVERIFIED here; prior pass `6b496eb` + `autonomous-harness-prior-art.md` section 2 has its DB requirement for budgets). +- OpenRouter BYOK (https://openrouter.ai/docs/guides/overview/auth/byok): provider keys are + "securely encrypted" (method undocumented), several per provider, each scopable to models, + OpenRouter API keys or workspace members, ordered as prioritized or fallback, with OpenRouter's + own capacity as fallback unless disabled. +- Common shape: the caller never holds the provider credential; it holds a proxy-issued identity, + and the proxy maps that identity to the provider credential and to its own budget. The kit + already has this shape (per-caller connection key and windowed ceiling, rulings `connector-owner`, + `connector-spend-caller-window-ceiling`); what neither proxy documents is OS-level isolation of + the key from co-located processes, which is ours to supply (a unit-private credential file). + +## 4. Thinking across tool use: where the in-flight exchange starts + +Source: https://platform.claude.com/docs/en/build-with-claude/thinking ("Thinking with tool use", +"Preserving thinking blocks", "Thinking block preservation by model", "Thinking encryption", +"Controlling thinking display"); https://platform.claude.com/docs/en/agents-and-tools/tool-use/handle-tool-calls. + +- "A tool-use loop is one assistant turn." An assistant turn ends only when Claude completes its full + response, which may include several tool calls and results. Tool results appear as user messages + but are part of one continuous reasoning flow. +- Required: within a tool-use turn, pass thinking blocks back complete and unmodified. Within the + latest assistant message the consecutive thinking blocks must match what the model generated: no + reorder, edit or partial drop, `redacted_thinking` included; a modified block is a 400. +- In the user message carrying tool results, "the `tool_result` blocks must come FIRST in the + content array. Any text must come AFTER all tool results", and results must immediately follow + their `tool_use` message. Text after the results is legal with client tools only. +- Mid-turn toggle of thinking does not error: the API silently disables thinking for that request. + Adaptive mode does not require an assistant turn to start with a thinking block; manual mode does. +- Keep-all-prior-turns models: Opus 4.5 and later Opus, Sonnet 4.6 and later, Haiku 5.5, Fable and + Mythos 5/5.1; earlier models strip earlier-turn thinking. Kept blocks bill as input. A block is + readable only by its model and certain others; unreadable blocks are dropped without error. +- `signature` carries the full thinking encrypted; opaque, do not parse; compatible across Claude + API, Bedrock and Google Cloud, except Sonnet 5.5 and Haiku 5.5 blocks, which "work only in the + account that produced them or in an account linked to it". +- `display: "omitted"` is the default on Opus 5.5, Sonnet 5.5, Haiku 5.5, Fable 5.1 and others: + empty `thinking` field, full thinking still billed, replay unchanged. +- Prefix binding (from `725f5a5` 1a, confirmed on the errors page): on Fable 5.1, Opus 5.5, + Sonnet 5.5 and Haiku 5.5 a replayed block is accepted only while `system`, `tools` and the + preceding messages are unchanged. + +Answer to follow-up `claude-tool-exchange-boundary`: the kit's rule "in flight after the last user +text" is wrong for Claude whenever user text rides with tool results. In the kit transcript a +contiguous run of `UserText` and `ToolResult` items lowers to one user message; if that run holds a +`ToolResult`, the assistant turn before it is still in flight (the model has not yet answered those +results). So the in-flight exchange starts after the last user-side run that holds no `ToolResult`. +Probe P1 (section 8) shows the current kit drops, and executes past, foreign state in +`[user, foreign, call, result, user]`. + +## 5. Counting tokens for the reservation + +Source: https://platform.claude.com/docs/en/build-with-claude/token-counting; +https://platform.claude.com/docs/en/agents-and-tools/tool-use/define-tools ("Tool use system prompt"). + +- `POST /v1/messages/count_tokens` takes the Messages inputs and returns `{input_tokens}`. It is free, + has its own RPM limit (Start 5,000, Build 10,000, Scale 20,000), uses no caching logic, and is + "an **estimate**": "the actual number of input tokens ... might differ by a small amount", and + counts may include system-added tokens that are not billed. So it is neither an upper nor a lower + bound, and it is one more network call per request with its own failure mode. +- Claude 4.7 and later (Fable, Mythos) use a newer tokenizer: about 30% more tokens for the same text. + Still a byte-level tokenizer as far as the docs say; nothing documents a token shorter than one + byte of text: UNVERIFIED. +- With `tools`, the API constructs a system prompt from the tool definitions and its own fixed + instruction text. That text is input the caller did not send: a per-model constant the price + table's `overhead_tokens` must cover. +- Prior-turn thinking on keep-all models bills as input; what it bills is the decrypted thinking, + carried in the signature. Whether the signature's byte length is at least that thinking's token + count is not documented and cannot be probed without a live call: the one premise of the kit's + byte bound left open for Claude (follow-up `claude-byte-bound-live-check`). + +## 6. Facts the Messages lowering must respect + +- Tool names match `^[a-zA-Z0-9_-]{1,128}$`. `tool_choice`: `auto`, `any`, `tool`, `none`, plus + `disable_parallel_tool_use`. Opus 5.5, Sonnet 5.5, Fable 5.1 and Mythos 5.1 reject `any` and `tool` + with a 400 ("tool_choice: type \"tool\" and \"any\" are not supported for this model"); on models + that accept forced tool use, the forced call skips thinking. Changing `tool_choice` invalidates + cached message blocks. +- Claude 4.6 and later reject a prefilled final assistant message: the conversation must end with a + user message. +- Thinking: `thinking: {type: "adaptive"}` with `output_config.effort`; `type: "enabled"` is a 400 on + 4.7 and later; `type: "disabled"` is a 400 on Fable 5.1, Mythos 5.1, Fable 5, Mythos 5, Opus 5.5; + Sonnet 5.5 uses `between_tools` instead. Omitting `thinking` runs adaptive thinking. +- Structured outputs: `output_config.format: {type: "json_schema", schema}`; `minimum`, `maxLength` + and similar are unsupported (400); a `refusal` or `max_tokens` stop may yield non-conforming JSON. +- Usage fields are in `725f5a5` 1a: `input_tokens` (after the last cache breakpoint), + `cache_creation_input_tokens`, `cache_read_input_tokens`, `output_tokens`, + `output_tokens_details.thinking_tokens`. Stop reasons include `model_context_window_exceeded`, + `refusal` (HTTP 200, billed) and `pause_turn` (server tools only). + +## 7. Prices + +`725f5a5` 2e holds the current price table (per MTok: input, 5m write, 1h write, cache read, output; +Haiku 5.5's long-context tier above 100k prompt tokens). The kit's `Rates` and `LongContext` carry +every class it names; the connector's config document holds one `ModelPrice` per allowed model. + +## 8. Probes (Imagination, Yggdrasil ygg-verify rust image, 2026-10-28) + +Scratch commits `eb021f0` and `0c9a39d` on ModelConnector `e0c0832`, never pushed. + +- P1 boundary: a kit lib test admitting `[UserText, ReasoningState(other-provider), ToolCall, + ToolResult, UserText]` printed `executed, dropped=1`: the foreign item in an unanswered tool + exchange is dropped and the request runs. +- P2 verbatim schemas: `serde_json::value::RawValue` holding `{"b":1,"a":2}` serialized through the + `json!` macro came out `{"a":2,"b":1}` (the macro builds a `Value`, whose map sorts keys); through + a typed `#[derive(Serialize)]` struct field it came out `{"b":1,"a":2}`. A lowering that must keep + the caller's tool and output schemas byte-for-byte (prefix binding, section 4) serializes typed + structs with `RawValue` fields, never `json!`. +- P3 client: `ureq 3.5.0` (`default-features = false`, `rustls`) with `serde_json` `raw_value` + resolves and builds in the workspace (rust 1.95). Baseline at `e0c0832`: 138 kit lib tests with + `daemon`, all green. + +## 9. What the cut takes from this + +- The credential is an operator-owned Console API key in a dedicated workspace with its own spend + and rate limits (question `claude-connector-credential`, recommended), held only by the connector. +- The kit's in-flight boundary moves to the last user-side run with no tool result (section 4). +- The reservation keeps the kit's byte bound; the Claude price entries' `overhead_tokens` covers the + tool system prompt; the receipt's `overrun_micro_usd` is the live instrument (section 5). +- The lowering is a pure, deterministic function over typed structs with `RawValue` for caller JSON; + the response is read as a stream; failures that the docs do not say are free bill the reservation. From 6adc9d7fce8eb0aeeb021f89d693c4b054187475 Mon Sep 17 00:00:00 2001 From: Meta Vens Date: Wed, 28 Oct 2026 19:05:11 +0100 Subject: [PATCH 009/300] eureka-body map: claude-connector probes (F34), B4 cut line, B4a placed by its spec, B4b-c lines Co-Authored-By: Claude Opus 5.5 --- docs/eureka-body-map.md | 86 ++++++++++++++++++++++++++++++++++------- 1 file changed, 72 insertions(+), 14 deletions(-) diff --git a/docs/eureka-body-map.md b/docs/eureka-body-map.md index 3a552e0..c094fab 100644 --- a/docs/eureka-body-map.md +++ b/docs/eureka-body-map.md @@ -75,6 +75,13 @@ made exact, path B line 3c given its spec and B4's riders carried. Spec: `connector-spend-readout`. Research reused unchanged (`725f5a5` section 2 is the ceiling question's evidence). +Revised again on 2026-10-28 by Imagination (`imagination-claude-connector`, session +`self-2026-10-28-eureka`) at ModelConnector `e0c0832` (B3c landed), for map line B4. New fact F34 +(probes on Yggdrasil), rationale "Where the Claude connector cut line falls", path B line 4 split +into B4a-c. Spec: `cut-claude-backend.r1` (B4a). Question: `claude-connector-credential` (blocks +the spec). Follow-up: `claude-byte-bound-live-check`. Research: Huginn `f237623` +`docs/research/claude-connector-prior-art.md`. + ## Pinned heads (fetched 2026-10-08 ~12:10 UTC) | Repo | Ref | Commit | Note | @@ -441,6 +448,23 @@ Each line: what was run, where, result. `465-479`); `TransportOutcome::Failed` carries no usage (`524-527`); replay records are never deleted or pruned (no `delete` on the store in `service.rs`); `TransportService::open` takes a path, so a test cannot inject a failing store (`service.rs:181-182`). +- **F34 The kit's in-flight boundary misses Claude's turn rule; caller JSON must not pass through + `json!`** (2026-10-28, Yggdrasil ygg-verify rust image, scratch commits `eb021f0`, `0c9a39d` on + ModelConnector `e0c0832`, never pushed). P1: a kit lib test admitting `[UserText, + ReasoningState(other-provider), ToolCall, ToolResult, UserText]` printed `executed, dropped=1`, + so foreign state in a tool exchange the model has not yet answered is dropped, where Anthropic + counts a tool loop as one assistant turn (research `f237623` section 4). P2: a `RawValue` + `{"b":1,"a":2}` serialized through `json!` came out `{"a":2,"b":1}`; through a typed + `#[derive(Serialize)]` field it came out verbatim. P3: `ureq 3.5.0` (rustls, no default features) + with `serde_json` `raw_value` builds in the workspace on rust 1.95. Baseline: 138 kit lib tests + with `daemon`, green. Source reads at `e0c0832`: `ProviderBackend::execute` returns a + `TransportResult` synchronously on the serve loop's per-connection thread (`serve.rs:25-38`, + `243-260`); the backend's only result door is `ProviderCall::transported` (`service.rs:380`); + `wire_encode` is `pub(crate)` (`contract.rs:586`), so a connector crate encodes its own state + with `rmp_serde` directly; there is no connector binary, config document or Idunn presence in the + repo (`crates/` holds only `model-connector`); CodexConnector's `idunn_health.rs` is 1,396 lines + and its recipe binds a CultCache config document, a replay slot and a credential state slot + (`deployment/idunn/recipe.toml`). ## The model page (step 0b) @@ -690,6 +714,44 @@ before B4 so no consumer ever sees an intermediate v3: her own Persona organ, which she may rewrite (`she-owns-her-body`), so it is not a wallet control on its own. +### Where the Claude connector cut line falls (map line B4) + +B4 as one cut (backend, binary, config document, credential delivery, Idunn presence, spend +readout, retention, clock recovery) is three owners and well past one Hands pass: the presence port +alone is 1,396 lines in CodexConnector (F34). It splits so each part is falsifiable alone: + +- B4a `claude-backend` (spec `cut-claude-backend.r1`): the `ProviderBackend` for the Messages API + as a library crate. Lowering is a pure function over typed structs with caller JSON as `RawValue` + (F34 P2), because Opus 5.5, Sonnet 5.5, Haiku 5.5 and Fable 5.1 accept a replayed thinking block + only while system, tools and earlier messages are byte-unchanged. The response is read as a + stream (the docs steer long requests to streaming and a stream reports usage as it goes). One + execute is one HTTP request: no internal retry, so the replay store's "the provider runs once" + stays literal. Every provider failure bills the reservation (usage `None`), because the docs do + not say an errored request is free. The kit's in-flight boundary moves to the last user-side run + holding no tool result (F34 P1, research section 4), which answers follow-up + `claude-tool-exchange-boundary`. Token counting stays the kit's byte bound: `count_tokens` is an + estimate in both directions and a second network call, so it is not a bound either; the one open + premise (signature bytes against billed prior-turn thinking) is follow-up + `claude-byte-bound-live-check`, read from receipts' `overrun_micro_usd` on the first live turns. + The credential is question `claude-connector-credential`: Anthropic's terms forbid routing a + connector's requests through subscription OAuth, so the recommendation is an operator-owned + Console key in a dedicated workspace (the default workspace cannot carry a spend limit), readable + only by the connector's unit. +- B4b `claude-connector-binary`: the `claude-connector` binary, its config document (CultCache, + root-owned: listen address, `ServeLimits`, callers, the price table with one `ModelPrice` per + allowed model and `overhead_tokens` measured per `claude-byte-bound-live-check`, the credential + file path), credential delivery to the connector's own unit user, and deployment. Its map raises + whether Idunn deploys it now (presence rebuilt from `idunn_health.rs` against Idunn's current + contract) or it is hand-installed beside her units until `idunn-job-workload` + (`mind-units-templated-interim` is the precedent), and carries follow-up + `spend-window-clock-recovery` (the audited root verb belongs with the config document). +- B4c `connector-spend-readout`: the spend read operation beside `invoke` and replay-store + retention (follow-ups `connector-spend-readout-2`, `replay-store-retention`). Kit code, needed by + the turn-cost charge (A8, B5), not by the first live turn. + +B4a touches none of the code `kit-contract-test-debt` names, so that follow-up stays open for the +next cut in `contract.rs`, `client.rs` or `envelope.rs`. + ### How a Self run is carried (cut `mind-self-run`) The run document is the grant (`open_and_launch`); the unit takes only `--instance` @@ -770,20 +832,16 @@ operator's token, then A6 and the bring-up; A3, A4 and A1 run beside it): `725f5a5` section 1d and needs no kit rule (every provider's state stays opaque bytes). - c. `connector-spend` (ModelConnector): spec `cut_spec:cut-connector-spend.r1` at `1d51e71`. B3b has landed; it waits on question `question:connector-spend-ceiling`. -4. `claude-connector` (ModelConnector; ruling `persona-first-connector-claude`): waits on B3. Its - map raises how it authenticates and whose credential it holds against invariant - `token-out-of-her-reach` (rulings `connector-owner`, `self-run-token-reach`). It also carries - the connector's own config document and Idunn presence rebuilt from `idunn_health.rs` against - Idunn's current contract, if Idunn deploys it (the CultLib bump moved to B3a). Its lowering - to the Messages API must be deterministic: Anthropic binds a replayed thinking block to the - unchanged system, tools and earlier messages (research `725f5a5` section 1a, prefix check). - Riders carried from B3c and the B3 Soul passes: the price table lives in that config document - (one `ModelPrice` per allowed model, a missing one refused at open); if B3c's byte bound - refuses real turns, the backend supplies a provider count instead (Anthropic `count_tokens`, - research section 2d); the turn-cost charge needs a spend readout because the store is held - exclusively (F33 P2, follow-up `connector-spend-readout`); the tool-exchange boundary is - confirmed against Anthropic's rules (follow-up `claude-tool-exchange-boundary`); and whatever - of follow-up `kit-contract-test-debt` B3a-c did not kill is killed in code B4 touches. +4. `claude-connector` (ModelConnector; ruling `persona-first-connector-claude`), split three ways + (rationale "Where the Claude connector cut line falls"): + - a. `claude-backend`: spec `cut_spec:cut-claude-backend.r1` at `e0c0832`. B3 has landed; it + waits on question `question:claude-connector-credential`. + - b. `claude-connector-binary` (ModelConnector, gamecult-ops or Idunn): the binary, its config + document and price table, credential delivery, deployment and Idunn presence, follow-ups + `spend-window-clock-recovery` and `claude-byte-bound-live-check`. Waits on B4a and the + credential ruling; its map raises Idunn-now versus hand-install. + - c. `connector-spend-readout` (ModelConnector): follow-ups `connector-spend-readout-2`, + `replay-store-retention`. Waits on B4a; due before the turn-cost charge (A8, B5). 5. `mind-persona` (Huginn): the Persona organ, one connector request per turn (`persona-turn-connector`), Persona state through Huginn (landed), the start-Self tool (A6), traces with provenance from day one (`collect-with-provenance-gate-at-training`). Waits on From 4d77df80bcf39804d3dbaeef19d373d9ebe186d5 Mon Sep 17 00:00:00 2001 From: Meta Vens Date: Wed, 28 Oct 2026 19:05:29 +0100 Subject: [PATCH 010/300] eureka-body map: F34 cites execute_inference_frame at its real lines Co-Authored-By: Claude Opus 5.5 --- docs/eureka-body-map.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/eureka-body-map.md b/docs/eureka-body-map.md index c094fab..8e2f03f 100644 --- a/docs/eureka-body-map.md +++ b/docs/eureka-body-map.md @@ -459,7 +459,7 @@ Each line: what was run, where, result. with `serde_json` `raw_value` builds in the workspace on rust 1.95. Baseline: 138 kit lib tests with `daemon`, green. Source reads at `e0c0832`: `ProviderBackend::execute` returns a `TransportResult` synchronously on the serve loop's per-connection thread (`serve.rs:25-38`, - `243-260`); the backend's only result door is `ProviderCall::transported` (`service.rs:380`); + `240-262`); the backend's only result door is `ProviderCall::transported` (`service.rs:380`); `wire_encode` is `pub(crate)` (`contract.rs:586`), so a connector crate encodes its own state with `rmp_serde` directly; there is no connector binary, config document or Idunn presence in the repo (`crates/` holds only `model-connector`); CodexConnector's `idunn_health.rs` is 1,396 lines From 096b1423018b4a6e0444fe03493aa479d69f3ca7 Mon Sep 17 00:00:00 2001 From: Eureka Date: Thu, 29 Oct 2026 11:07:20 +0100 Subject: [PATCH 011/300] eureka-body map: B4a landed as claude-code-backend (F35), B4 line rewritten with B4d tool exchange Co-Authored-By: Claude Opus 5.5 --- docs/eureka-body-map.md | 117 +++++++++++++++++++++++++++------------- 1 file changed, 81 insertions(+), 36 deletions(-) diff --git a/docs/eureka-body-map.md b/docs/eureka-body-map.md index 8e2f03f..011f165 100644 --- a/docs/eureka-body-map.md +++ b/docs/eureka-body-map.md @@ -82,6 +82,15 @@ into B4a-c. Spec: `cut-claude-backend.r1` (B4a). Question: `claude-connector-cre the spec). Follow-up: `claude-byte-bound-live-check`. Research: Huginn `f237623` `docs/research/claude-connector-prior-art.md`. +Revised again on 2026-10-29 by Modeling (`modeling-claude-code-backend`, session +`self-2026-10-29-eureka`) at ModelConnector `200933f`, after B4a landed as `claude-code-backend` +(spec `cut-claude-code-backend.r1`; cut report `h2`, Soul verdict `s2`). Ruling +`claude-connector-via-claude-code` answered question `claude-connector-credential` and withdrew +spec `cut-claude-backend.r1`. New fact F35. The B4 rationale and path B line 4 are rewritten +(B4a landed, B4d added, the Console-key backend deferred), the spend-ledger row of the model +page changed and a Claude connector backend row was added. F34's repo-state sentence keeps its +tense. + ## Pinned heads (fetched 2026-10-08 ~12:10 UTC) | Repo | Ref | Commit | Note | @@ -465,6 +474,26 @@ Each line: what was run, where, result. repo (`crates/` holds only `model-connector`); CodexConnector's `idunn_health.rs` is 1,396 lines and its recipe binds a CultCache config document, a replay slot and a credential state slot (`deployment/idunn/recipe.toml`). +- **F35 B4a at ModelConnector `200933f` (landed; read from cut report `h2`, verdict `s2` and the + crate, 2026-10-29).** `crates/claude-connector` is a library (`backend.rs`, `config.rs`, + `handle.rs`, `launch.rs`, `process.rs`, `stream.rs`; deps `model-connector` with `daemon`, + `serde_json`, `rmp-serde`, `sha2`, `zeroize`, `libc` on unix; no HTTP client). There is still no + binary, config document or Idunn presence. `model-connector` and `schemas/` are byte-identical + to `e0c0832`. Gate: 198 tests green; `cargo mutants --in-diff` 155 mutants, 121 caught, 0 missed, + 2 timeouts (`process.rs` `StdChild::kill`). Live (claude 2.1.312, Starfire, haiku-5-5, four + calls): print mode emits `result.model_usage_since_last_result`, the turn's own usage; the + cumulative `modelUsage` of a resumed fork also counts the earlier turns, so it is never read; + `--system-prompt-file` replaces the system prompt as `--system-prompt` does. Argv sets + `--tools ""`, `--strict-mcp-config`, `--setting-sources ""`, `--permission-prompts none`, and + `--resume --fork-session` on every resume. The child env is cleared to `PATH`, `HOME`, + `CLAUDE_CONFIG_DIR`, `CLAUDE_CODE_MAX_OUTPUT_TOKENS` and the one const + `CLAUDE_CODE_OAUTH_TOKEN`. Not exercised live: `CLAUDE_CODE_MAX_OUTPUT_TOKENS`. Open scars from + Soul `s2`: `stderr-join-reachable` (the stderr-tail join is reachable but undefended in the + repo), `prompt-file-exclusive-undefended` (the exclusive create on a predictable temp name), + and `door-refusal-boundary` (code bills zero only for a 401 or 403 with explicitly empty usage; + ruling `claude-code-plan-quota-fuse-2`, admitted after `s2`, says 401, 403 and 429, so a 429 with + `{}` still bills the reservation until follow-up `claude-backend-hardening` lands). Follow-ups + `handle-unauthenticated`, `pin-checked-only-at-ready`. ## The model page (step 0b) @@ -480,7 +509,8 @@ pipeline kinds are in Huginn `docs/eureka-substrate-map.md`. | Brake | One document per instance in a root-owned CultCache store `/etc/gamecult/minds//control.cc`, type `eureka.control.brake.v1 { released: bool, set_at, set_by }`. | Replaced whole by the operator's CLI; never appended. Absent or undecodable reads as held. `mind-control brake hold` writes it held and then, in the same command, stops her running units: `systemctl stop` (blocking) over `mind-self@:*` and `mind-persona@:*`, through `SystemdLauncher::stop_all` (cut brake-stop; ruling `self-run-actuator`; F18-F20). | Operator only (ruling `brake-dial-root-store-now`); her units read it, and the file is not writable by any user she runs as. The hold is the one stopper of her units. `read_effective` is the one reader. Its `Released` answer carries a `Grant` (cadence and run cap, private fields, not `Clone`), which `open_and_launch` takes by value, so no run opens without a released reading (cut mind-launch r3). | | Burn-rate dial | Same store, `eureka.control.burn_rate.v1 { heat, base_cooldown_s, base_run_usd, set_at, set_by }`; `heat` in 0.05..=2.0. | Replaced whole by the CLI. Absent or out of bounds reads as heat 0, which launches nothing (fail closed, LiteLLM scar rider of ruling `connector-owner`). Schedules and her own rest arrive later as follow-up `burn-schedule-and-rest`. | Operator. Derived (never stored): wake cadence `base_cooldown_s / heat`, per-run cap `--max-budget-usd = base_run_usd × heat`, per-request `max_cost_usd`. | | Connector request and transcript items | Request: `(caller_runtime_id, request_id)`, the replay key the daemon already keeps (`replay.cc`). Items: UserText, AssistantText, ToolCall, ToolResult, plus `ReasoningState { provider, model, opaque }` (ruling `connector-opaque-state-item`). Contract ids become provider-neutral `gamecult.model.*.v3` (follow-up `connector-contract-untyped`); the connector serves Ghostlight too, so not `eureka.*`. | Invocation expires at `expires_at_unix_ms`; replay record per key; restart-era `Indeterminate`. The caller stores the transcript and replays it verbatim; foreign-tagged reasoning state inside an in-flight tool exchange is refused typed; completed-turn foreign state is settled per provider in the connector cut (follow-up `connector-reasoning-state-cheap-providers`). | The connector produces, reads or refuses reasoning bytes; callers never branch on provider. | -| Connector spend ledger | `gamecult.model.caller_spend.v1` keyed by caller runtime id and window start, in each daemon's state root. | Rolled per window; refusal `Budget` at the cap; durable, and a missing ledger refuses the request. | Each daemon writes its own; the root caller config sets the cap. | +| Connector spend ledger | The reservation lives in the replay record itself (cut connector-spend, F33): integer micro-USD, windowed per caller (ruling `connector-spend-caller-window-ceiling`). | Reserve before the provider runs, settle to actual; refusals `CostBound` (the invocation's own `max_cost_micro_usd`) and `SpendCeiling` (the caller's window); a ledger that cannot be written refuses the request. | The kit's `ProviderCall::charge` is the one billing owner; a backend reports usage or `None` (bills the reservation). The root caller config sets the ceiling. | +| Claude connector backend (B4a, landed `200933f`) | `ClaudeCodeBackend` in `crates/claude-connector`, provider tag `claude-code`: one `claude -p` child per call in its own process group, tool-less, system prompt in a private temp file; reasoning state is a named-MessagePack `ClaudeCodeHandle { session_id, instance, prefix_sha256 }`; usage is `model_usage_since_last_result` summed over models, in list-price units from B4b's price table (ruling `claude-code-plan-quota-fuse-2`). | A handle resumes (always `--fork-session`) only when it decodes, names this instance, its session file exists and its digest matches the request's instructions, model and earlier items; anything else re-seeds from the caller's transcript (ruling `claude-code-handle-reseeds`). The CLI's session store under `config_dir` is a cache; the caller's transcript is the truth. A refusal at the door with explicitly empty usage bills zero; unknown usage, a crash or a deadline bills the reservation. | The backend owns the argv, env, process and stream mapping. It never reads `total_cost_usd`, never forwards `ANTHROPIC_API_KEY`, never retries, never spawns twice per execute. Forbidden: `--bare`, `--fallback-model`, `--mcp-config` and `--dangerously-skip-permissions` (B4d changes the MCP line, by spec). Her organs hold none of it: the subscription token lives in the connector's own environment. | | Persona turn traces | **Empty: question `persona-trace-retention`.** Candidate: per turn, the request digest, the transcript items and the typed output, keyed by the turn's run. | Empty: how long they are kept, whether Discord-sourced content may enter a training export (F14). | Empty: who may export them. | | Persona state | `gamecult.persona_state.v0`, one per mind, `personaId` = instance (cut-huginn-persona-state r2, eureka-substrate). | Replaced whole under CAS with a receipt; she may rewrite any field (`identity-write-all-visible`). | The Persona organ is the only writer. | | Identity diffs | The pair of receipts of two consecutive Persona puts whose identity fields (publicName, publicDescription, presentation, values) differ. Derived, not stored. | Composed by the Persona organ at put time and sent as one delivery request through the crossing; a failed post is retried by Bifrost's delivery journal, not by her. | Persona organ composes; Bifrost posts under the permit. No rate limit. | @@ -718,38 +748,45 @@ before B4 so no consumer ever sees an intermediate v3: B4 as one cut (backend, binary, config document, credential delivery, Idunn presence, spend readout, retention, clock recovery) is three owners and well past one Hands pass: the presence port -alone is 1,396 lines in CodexConnector (F34). It splits so each part is falsifiable alone: +alone is 1,396 lines in CodexConnector (F34). Ruling `claude-connector-via-claude-code` +(2026-10-29, operator) answered the credential question: the first Claude backend drives the +`claude` binary on the operator's subscription, not the Messages API with a Console key. It +splits so each part is falsifiable alone: -- B4a `claude-backend` (spec `cut-claude-backend.r1`): the `ProviderBackend` for the Messages API - as a library crate. Lowering is a pure function over typed structs with caller JSON as `RawValue` - (F34 P2), because Opus 5.5, Sonnet 5.5, Haiku 5.5 and Fable 5.1 accept a replayed thinking block - only while system, tools and earlier messages are byte-unchanged. The response is read as a - stream (the docs steer long requests to streaming and a stream reports usage as it goes). One - execute is one HTTP request: no internal retry, so the replay store's "the provider runs once" - stays literal. Every provider failure bills the reservation (usage `None`), because the docs do - not say an errored request is free. The kit's in-flight boundary moves to the last user-side run - holding no tool result (F34 P1, research section 4), which answers follow-up - `claude-tool-exchange-boundary`. Token counting stays the kit's byte bound: `count_tokens` is an - estimate in both directions and a second network call, so it is not a bound either; the one open - premise (signature bytes against billed prior-turn thinking) is follow-up - `claude-byte-bound-live-check`, read from receipts' `overrun_micro_usd` on the first live turns. - The credential is question `claude-connector-credential`: Anthropic's terms forbid routing a - connector's requests through subscription OAuth, so the recommendation is an operator-owned - Console key in a dedicated workspace (the default workspace cannot carry a spend limit), readable - only by the connector's unit. -- B4b `claude-connector-binary`: the `claude-connector` binary, its config document (CultCache, - root-owned: listen address, `ServeLimits`, callers, the price table with one `ModelPrice` per - allowed model and `overhead_tokens` measured per `claude-byte-bound-live-check`, the credential - file path), credential delivery to the connector's own unit user, and deployment. Its map raises +- B4a `claude-code-backend` (landed, ModelConnector `200933f`, spec `cut-claude-code-backend.r1`, + F35): `ClaudeCodeBackend` as a library crate, tool-less turns, session-handle reasoning state + (ruling `claude-code-handle-reseeds`), usage from `model_usage_since_last_result` (ruling + `claude-code-plan-quota-fuse-2`). The kit is unchanged, so the in-flight boundary (follow-up + `claude-tool-exchange-boundary`) was not moved here; it belongs to B4d. +- B4b `claude-connector-binary` (next): the `claude-connector` binary, its config document + (CultCache, root-owned: listen address, `ServeLimits`, callers, the `ClaudeCodeConfig` fields, + the price table with one `ModelPrice` per allowed model in list-price units and + `overhead_tokens` for the harness preamble), credential delivery + (`CLAUDE_CODE_OAUTH_TOKEN` from a root-0600 EnvironmentFile to the connector's own unit user, + readable by no unit of hers), the pinned `claude` binary digest, and deployment. Its map raises whether Idunn deploys it now (presence rebuilt from `idunn_health.rs` against Idunn's current contract) or it is hand-installed beside her units until `idunn-job-workload` - (`mind-units-templated-interim` is the precedent), and carries follow-up - `spend-window-clock-recovery` (the audited root verb belongs with the config document). + (`mind-units-templated-interim` is the precedent). It carries follow-ups + `spend-window-clock-recovery` and `claude-byte-bound-live-check`; the first live tool-less turn + is its operator check, and confirms `CLAUDE_CODE_MAX_OUTPUT_TOKENS` under the pinned binary. - B4c `connector-spend-readout`: the spend read operation beside `invoke` and replay-store retention (follow-ups `connector-spend-readout-2`, `replay-store-retention`). Kit code, needed by the turn-cost charge (A8, B5), not by the first live turn. +- B4d `claude-tool-exchange`: ruling `claude-code-runs-her-tools` (operator, 2026-10-29) lets the + claude process run her tool loop over MCP, so one kit call becomes a whole agent loop, and + `self-run-actuator-2` restates the actuator ruling to match (the connector may host her Persona + turn's tool loop on this backend). The cut owes: the kit reserves for the whole call, not one + turn; the process carries `--max-turns` and a budget cap; her tools never inherit the + connector's environment, where the subscription token lives, so they run as MCP servers in her + own unit that claude reaches, or under a scrubbed environment (`token-out-of-her-reach` holds); + `--mcp-config` and `--tools` leave B4a's forbidden list for exactly that surface; the kit's + in-flight boundary and `partition_reasoning` change (F34 P1). It waits on B4b. +- Later, off the path: a Console-key Messages API backend (follow-up `claude-api-backend`, owner + Self when quota contention is observed), from the withdrawn spec `cut-claude-backend.r1`, whose + probes (F34) and anchors are its raw material. Its provider tag differs from `claude-code`, so + the kit treats each backend's reasoning state as foreign to the other. -B4a touches none of the code `kit-contract-test-debt` names, so that follow-up stays open for the +B4a touched none of the code `kit-contract-test-debt` names, so that follow-up stays open for the next cut in `contract.rs`, `client.rs` or `envelope.rs`. ### How a Self run is carried (cut `mind-self-run`) @@ -832,17 +869,25 @@ operator's token, then A6 and the bring-up; A3, A4 and A1 run beside it): `725f5a5` section 1d and needs no kit rule (every provider's state stays opaque bytes). - c. `connector-spend` (ModelConnector): spec `cut_spec:cut-connector-spend.r1` at `1d51e71`. B3b has landed; it waits on question `question:connector-spend-ceiling`. -4. `claude-connector` (ModelConnector; ruling `persona-first-connector-claude`), split three ways - (rationale "Where the Claude connector cut line falls"): - - a. `claude-backend`: spec `cut_spec:cut-claude-backend.r1` at `e0c0832`. B3 has landed; it - waits on question `question:claude-connector-credential`. - - b. `claude-connector-binary` (ModelConnector, gamecult-ops or Idunn): the binary, its config - document and price table, credential delivery, deployment and Idunn presence, follow-ups - `spend-window-clock-recovery` and `claude-byte-bound-live-check`. Waits on B4a and the - credential ruling; its map raises Idunn-now versus hand-install. +4. `claude-connector` (ModelConnector; rulings `persona-first-connector-claude`, + `claude-connector-via-claude-code`), split four ways (rationale "Where the Claude connector cut + line falls"): + - a. `claude-code-backend`: landed 2026-10-29 as ModelConnector `200933f` (spec + `cut_spec:cut-claude-code-backend.r1`, report `h2`, verdict `s2`). The withdrawn + `cut-claude-backend.r1` (Console-key Messages lowering) is raw material for the later + `claude-api-backend`. + - b. `claude-connector-binary` (next; ModelConnector, gamecult-ops or Idunn): the binary, its + CultCache config document and price table, credential delivery, deployment and Idunn + presence on Yggdrasil, follow-ups `spend-window-clock-recovery` and + `claude-byte-bound-live-check`; its map raises Idunn-now versus hand-install. - c. `connector-spend-readout` (ModelConnector): follow-ups `connector-spend-readout-2`, - `replay-store-retention`. Waits on B4a; due before the turn-cost charge (A8, B5). -5. `mind-persona` (Huginn): the Persona organ, one connector request per turn + `replay-store-retention`. Kit code, ready now that B4a has landed; due before the turn-cost + charge (A8, B5). + - d. `claude-tool-exchange` (ModelConnector, with her tool MCP servers): claude runs her tools + over MCP (ruling `claude-code-runs-her-tools`); her tools never inherit the connector's + token. Waits on B4b. + - Later: `claude-api-backend` (follow-up), a Console-key API backend. +5. `mind-persona` (Huginn): the Persona organ, one connector request per turn (a whole tool loop once B4d lands) (`persona-turn-connector`), Persona state through Huginn (landed), the start-Self tool (A6), traces with provenance from day one (`collect-with-provenance-gate-at-training`). Waits on B4 and on where traces live (question `persona-trace-retention` was withdrawn to be re-asked From 6963b755723513fae7e93e27ffabf85b70ed8fd5 Mon Sep 17 00:00:00 2001 From: Eureka Date: Thu, 29 Oct 2026 18:43:34 +0100 Subject: [PATCH 012/300] eureka-body map: swarm by 2026-10-30 (seven parallel cuts, credential lease, typed Persona reply) Co-Authored-By: Claude Opus 5.5 --- docs/eureka-body-map.md | 34 ++++++++++++++++++++++++++++++++++ 1 file changed, 34 insertions(+) diff --git a/docs/eureka-body-map.md b/docs/eureka-body-map.md index 011f165..3a715c1 100644 --- a/docs/eureka-body-map.md +++ b/docs/eureka-body-map.md @@ -811,6 +811,40 @@ in-force `operated_by Mind` Self run of its instance, which admission keeps uniq prefix passes `CEILING_TOKENS + FLUSH_ALLOWANCE_TOKENS`, it is interrupted once. A queued message that is never delivered costs the flush, never the stop. +### Swarm by 2026-10-30 (Imagination `imagination-swarm-tonight`, 2026-10-29) + +Ruling `self-runs-through-connector` (operator) makes this the top priority: Eureka hears and +speaks in Matrix, takes turns through the Claude connector on one token, starts her own Self runs +and reads the communal burn into #burn, under the operator's brake and dial. Units are +hand-managed on Yggdrasil tonight (follow-up `swarm-units-under-idunn`). + +Flow: a message in #swarm, #burn or #operator reaches the Bifrost mouth (`persona-mouth`, the only +holder of @eureka's Matrix token, ruling `persona-matrix-account-interim`), which queues a typed +delivery and rings `mind-persona@eureka`; the waker rings it on cadence too +(`persona-turn-drains-deliveries`). The unit's gate reads brake and dial; `mind-persona` builds a +turn (persona state, deliveries, her runs, the connector's spend and plan-quota readout), calls the +connector tool-less with a JSON Schema, and decodes one typed `PersonaReply` +(`persona-reply-typed-by-schema`): say items go back to the mouth, a `start_self_run` admits a brief +follow-up and calls `open_and_launch` (`self-run-brief-claimed-follow-up`). `mind-self@eureka` +leases the token from the connector on a pipe fd over a peer-uid unix socket and runs claude in her +own user and workspace (`self-run-credential-lease`); the connector reserves and settles in its +ledger. The brake hold stops both units; the mouth refuses sends while held. + +Authority: connector = token, admission, ledger, plan-quota observation; mind-control = brake and +dial; Bifrost = Matrix credential, ear, mouth; mind-persona = the turn and the decision to start a +Self run; mind-self-run = the Self run's process (argv, flush, interrupt). Demoted: A5's token +EnvironmentFile (never lands), A6 `mind-launch-self` (not built; the Persona opens runs, ruling +`first-self-run-starter`), the Discord posting actuator (deleted in `persona-mouth`). + +Cuts (specs `cut_spec:cut-