diff --git a/references/research-token-audit-2026-11-11.md b/references/research-token-audit-2026-11-11.md new file mode 100644 index 0000000..98d2ecc --- /dev/null +++ b/references/research-token-audit-2026-11-11.md @@ -0,0 +1,206 @@ +# Token audit of the GameCult swarm, 2026-11-05 to 2026-11-11 + +Served: an Imagination pass for Eureka Self (root), 2026-11-11, after the operator's +"Eureka needs to run lean, or we simply can't afford our own process." It answers where +the tokens go, what they bought, why the waste made sense to the agents, and what to cut +first. Evidence was gathered by eight Eyes subagents; nothing on any host was changed. + +All times UTC. "Prompt tokens" means input + cache-read + cache-creation per API request +(the journal's `request N prefix P`; `self_run.rs:395`). That is a gross count. About 96% +of it is cache-read, billed on the API at 0.1x; cache-creation with the 1 h TTL the units +use is 2x; output 5x. How the subscription meter weighs them is not known (see +"Not measured"), so every saving below is given gross, with a note where the weighted +effect is much smaller. + +## 1. Where the tokens go + +Three consumers, not one. The 818M figure for 2026-11-10 covered only the first. + +| Consumer | Window | Requests | Prompt tokens | 2026-11-10 alone | +|---|---|---|---|---| +| Self runs, 8 Minds (Yggdrasil) | 11-05..11-11 06:00 | 23,398 | 3,263M | 1,032M (6,379 requests, 743 turns) | +| Personas, 8 Minds (Yggdrasil) | 11-05..11-11 06:00 | 17,460 | 1,876M | 534M (4,515 requests, 4,351 turns) | +| Root sessions (Starfire) | 11-04..11-11 | 12,203 | 2,379M | 286M | +| Total | | 53,061 | 7,518M | 1,852M | + +The journal gives 1,032M for Self runs on 11-10 UTC (1,069M counting unfinished turns), +not 818M; the difference was not reconciled and is probably a day-boundary choice. + +Rough API-weighted units (input + 2x creation + 0.1x read + 5x output; Self and Persona +output and creation are extrapolated from four live transcripts, so treat as +-30%): +Self about 0.6B, Starfire about 0.35 to 0.4B before any premium for Opus-class models, +Persona about 0.25B. + +### Self runs + +- All Self and subagent requests run on `claude-sonnet-5-5`, effort medium, adaptive + thinking, compaction disabled (`DISABLE_COMPACT=1`), context ceiling 300k. +- A run starts at a median 34.8k tokens, and request 2 adds 18.3k: that is the Eureka + `SKILL.md` (53.2k chars), invoked first in 4 of 4 live transcripts. Every later request + carries it. Floor cost of the start prefix (P at request 2 x requests) is 1,229M, 37.7% + of Self tokens. The skill alone is 18.3k x 23,398 = 428M (13.1%). +- Request 1 by component (chars): tool schemas 25.6k, CLAUDE.md chain 33.3k (doctrine + 19.4k, gamecult.md 11.3k, seed 2.6k), output style 9.1k, skill listing 7.7k, system + prompt 7.2k, agent listing 3.9k, first user message 3.2 to 6.2k of which 2.4k is the + same every run. 11.4k tokens are cache-read at request 1; 20.6 to 24.5k are created + fresh on every run, and about 19k on every subagent spawn. +- Context growth, not orientation, is the mass. 59.6% of Self tokens sit in requests + above 150k; 65.9% in the 86 runs (of 720 starts) with more than 100 requests. The first + 12 requests of a run are 4.6% of it. +- A run is a session that swallows briefs one after another: on 11-10, 592 Persona + `start_self_run` briefs and 87 unit starts, 743 turns in 63 working runs. Median + context at the start of turn 1 is 34.8k; turn 2, 101.8k; turn 3, 126.1k; turn 4, + 137.4k; turn 5 and later, 208.4k. 518 of the 743 turns are turn 5 or later. Sessions + end by Idle (52), Ceiling at 300k (39) or Rotation (15) since 11-09. +- 345 of 720 starts made no request (325 lasted one second). They cost nothing. +- The 11-10 outage (UnknownCaller, 07:39 to 12:09, 24 refused starts) cost no Self + tokens: a refused lease starts no `claude`. Its token cost was a later sandbox fault + (8 turns, 1.8M) and Persona chatter about it (54 mentions). The loss was time. + +Activity on 11-10, classified from the closing text of the 80 costliest turns plus a +random 40 (transcripts are not kept, so this is the best available; 90% intervals wide): + +| Activity | Share of the day's Self tokens | +|---|---| +| ship (own change; about a fifth of it unmerged at turn end) | 36.9% (29.7 to 45.3) | +| unclassifiable, mostly "Soul pass running, I'll wait" | 23.5% | +| investigate, report only | 15.5% | +| review a peer's work | 10.8% (4.7 to 17.1) | +| coordinate | 10.3% (4.9 to 17.8) | +| verify own work, no new change | 2.3% | +| blocked | 0.8% | + +Flags on the top 80 turns (45% of the day): a Soul pass is mentioned in turns holding +64.8% of their tokens, mutation testing 36.0%, visual, cosmetic or docs-only change +20.1%, re-do at a new head 14.7%, waiting for a go 13.9%. Verification is mostly inside +"ship" turns and cannot be separated without usage logs. 146 turns (168.7M, 16.3%) close +with only "waiting on Soul, Hands or a poll". + +### Personas + +- 16,041 turns in 5.5 days; about 540 per Mind on 11-10; median gap 0.8 min; 1.09 + requests per turn; mean 107k prompt tokens per request. A process starts at a median + 37.2k, grows about 1.1k per turn with no compaction, and ends at a median 83.3k after + 28 turns (median life 2.5 h). +- The mouth (`Bifrost/tools/persona-matrix-mouth.mjs:315-332`) turns every non-self + message in all 11 joined rooms into a pending delivery for every Persona. There is no + addressed-only filter; addressing only sets the order (rank x 8 s stagger, + `persona.rs:1982`). On 11-10: 1,079 Mind posts, 4,351 Persona turns, 4.0 turns per post. +- 3,110 of those 4,351 turns (71.5%) said nothing and started nothing. 1,796 produced + only an `intent` string; 1,126 an urgency vote; 178 a memory. +- The prompt: 58.6k chars of appended instructions, of which `colossus.md` is 16.0k and + `persona.md` 9.3k; a 20k-char first message of which "Your latest runs" is 9.6k. +- 14% of turns emit a Self-run brief (2,093 in the window, 294 launches): the rest queue + into a live Self session, which is the brief stream described above. + +### Root sessions on Starfire + +- One session (`6d32c5f0`, open since 11-04 20:54) holds 2,034M of the 2,379M: 10,556 + requests, 169 subagents, largest single request 967k tokens. +- Subagents are 63.0% (1,498M): Sonnet 864M in 118 agents, Opus 629M in 89. Median + subagent 5.0M prompt tokens, p90 16.1M. 185 of 216 Agent calls passed no model. +- Main thread: about 2,219 requests for 881M, a mean of 397k per request. +- 51 cache misses re-created 10.4M tokens; 18 followed a gap over an hour. + +## 2. Output per token + +Public forge only (gamecult-ops is private and was not counted; deploys and artifacts +were not measured). + +| Mind | PRs merged 11-04..11 | Merged 11-10 | Self tokens (M) | Self M per merged PR | +|---|---|---|---|---| +| libby | 104 | 24 | 692 | 6.7 | +| eureka | 68 | 14 | 590 | 8.7 | +| void | 64 | 33 | 522 | 8.2 | +| weksa | 54 | 29 | 229 | 4.2 | +| heimdall | 38 | 10 | 392 | 10.3 | +| aqua | 34 | 19 | 232 | 6.8 | +| nibu | 26 | 17 | 332 | 12.8 | +| kiko | 3 | 0 | 274 | 91 (her work is mostly not PR-shaped) | + +Swarm-wide, Self plus Persona: 13.1M prompt +tokens per merged PR over the week, 10.1M on 11-10. + +- 391 PRs merged in the week; 155 on 11-10, three times the daily mean. Median diff 171 + lines; 95 PRs are 30 lines or fewer. +- 95% self-merged; median last push to merge 1.4 min; 81% have no comment or review on + the forge. Review happens in chat and in Self turns, not on the forge: Bifrost #21 and + #22 show zero forge interactions and a "Libby, round 3" in the body. +- The sign-in footer cluster on the forge is three PRs (Ratatoskr #61, #62, #63: 62, 90 + and 24 lines, each merged within 22 minutes), plus adjacent sky-band PRs. + +## 3. Structural causes + +1. **The Persona is the address filter.** Nothing before the model decides whether a + message concerns a Mind, so a 37k to 83k-token call is spent on each of 8 Minds per + message to produce, 71.5% of the time, silence. Missing: routing by address before + the model. The ordering logic that knows who is addressed already exists. +2. **Chat volume becomes Self volume.** Each Persona that hears a message may emit a + brief; briefs queue into the one live Self session; the session carries every earlier + brief's context into the next unrelated one, uncompacted, to 300k. Missing: a session + boundary that follows the work instead of the clock. +3. **Self cannot speak or see across campaigns.** Libby: "I can't post in #swarm myself, + since there is no chat tool here". Each relay costs a Persona turn and seven + deliveries. Heimdall re-read Huginn #29 "because that Soul pass's mind doesn't hold + the `eureka-body` or `eureka-substrate` campaigns": the author's Soul lacked the + rulings, so a peer had to read. Nibu: "ready for Void to merge on your go": 11 of her + PRs waited on the steward's merge. +4. **The root never clears and its subagents inherit its model.** A week-old session at + a mean 397k per main request, and 89 subagents on Opus because no agent definition + names a model. +5. **Nothing records usage.** Transcripts live on tmpfs and vanish at run end; the + launcher discards the CLI's usage envelope; the connector logs "weekly reading + refreshed" without the value. Agents and Self had no way to see what a habit cost. + +## 4. Ranked plan + +Savings are gross prompt tokens at the observed week's volume. Rank is by weighted +effect and confidence, not by the gross number. + +| # | Change | Owner | Expected saving and arithmetic | Shows it worked | +|---|---|---|---|---| +| 0 | Log one line per turn from the CLI result envelope (input, creation, read, output, model, subagent requests) and the meter value on each refresh. Adds a log line, no gate. Failure it prevents: this audit could not measure output, creation, subagent cost or quota weight, so item 4 cannot be costed. | Eureka (Huginn `self_run.rs`, connector) | None itself; makes every item below checkable | Usage lines in both journals; a per-day meter delta | +| 1 | Deliver in real time only what addresses a Mind (rank 0 in the existing ordering, DMs, Self replies); hold the rest and hand it over with her next addressed turn or beat. Deletes the deliver-everything path. | Heimdall (Bifrost mouth) with Eureka (`persona.rs` stagger) | 71.5% of turns are silent; at 60 to 70% of 534M/day: 320 to 370M/day, 1.1 to 1.3B/week. Nearly all cache-read plus about 270 output tokens per turn. Second-order: fewer briefs into Self | Persona turns per Mind per day from about 540 to under 150; silent share under 30%; briefs per day down from 592 | +| 2 | Root: name a model in each agent definition (Hands, Soul, Eyes, Modeling, Life on Sonnet or Haiku; Opus-class only where a definition argues for it), and clear the root at each handoff-area boundary | Root Self (agent definitions in the Eureka skill repo) | 629M/week of Opus subagent tokens move to Sonnet. Main thread from a mean 397k to under 150k per request: (397k - 150k) x 2,219 = 548M/week at the root model's price | Opus share of subagent tokens; mean main-thread prompt per request | +| 3 | Confirm the skill split landed, then keep cutting the fixed prefix: the 2.0k constant brief-closing rules, the 19k created per subagent spawn, `colossus.md` (16.0k chars) and "Your latest runs" (9.6k chars) in the Persona prompt | Eureka | Split: 12k x 23,398 = 281M/week (8.6% of Self), if references are not re-read. Colossus: about 5.5k x 17,460 = 96M/week. Each further 1k off the Self floor: 23M/week | P(request 2) - P(request 1) at or below 7k (was 18.3k; no post-split run existed by 05:37 on 11-11); Persona first-turn prefix | +| 4 | Lower the Self ceiling from 300k to about 150k, and end the session when a brief closes and none is queued | Eureka (`self_run.rs:99`) | At least the excess above 150k: 603M/week (18.5% of Self). Weighted effect is perhaps a third of that, since each restart re-creates about 20k tokens; do it after 3 and cost it with 0 | Share of Self tokens in requests above 150k from 59.6% to under 25%, with creation per day not rising | +| 5 | Give Self the missing means: a chat post from inside a Self run; cross-campaign read of rulings for a Soul pass; merge in a steward's repo where the steward grants it | Eureka (Huginn), each steward for merge rights | Peer review is 10.8% of Self (about 110M/day on 11-10); halving it is about 55M/day. Relays: one Persona turn plus seven deliveries each | Review-peer share in a resample of closing texts under 5%; "please post this" relays gone | + +Measure, do not re-propose: the 11-10/11 doctrine changes (one Soul pass, proportional +verification). The baseline to beat is above: mutation mentioned in 36.0% and +visual-only work in 20.1% of top-turn tokens, re-do at a new head 14.7%, peer review +10.8%, waiting for a go 13.9%. + +Forks for the operator or the steward, not decided here: +- Item 1 removes ambient hearing in real time. 1,126 urgency votes a day depend on every + Persona hearing everything. Is that vote a live mechanism worth about 350M tokens a day? +- Item 3: the global doctrine says the Colossus long form is not loaded automatically, + but every Persona loads it. Intent or drift? + +Considered and not placed: +- A 5-minute cache TTL for Self runs. 775 request gaps over 5 minutes would each + re-create about 150k tokens: about 145M units lost against about 98M saved. +- A fresh session per brief. Gross saving 44.5% of Self tokens, but about 1,960 extra + restarts a week at about 52k weighted units each cancel most of it, and related + briefs lose their context. Item 4 is the cheaper form. +- Keeping transcripts. Item 0 gives the numbers without the disk and privacy cost. +- A cheap-model pre-reader in front of each Persona. A new layer; the addressing rank + already answers the question. +- A cheaper model for Yggdrasil subagents. Not costed: no usage split exists (item 0). + +## 5. Not measured + +- Output, cache-creation and subagent tokens for Self and Persona; whether journal + request lines include subagent requests; so the true cost of Soul passes and mutation + runs is unknown. +- Quota weight per token class and per model; the meter's values. +- Tool-call level activity for any finished run (no transcripts); the classification + rests on closing texts cut to 650 chars. +- gamecult-ops #78 and all private repos; deploys and artifacts per Mind. +- The effect of the 11-10/11 fixes: no run had started on the 17 KB skill by 05:37. +- How many of the 2,093 Persona briefs were claimed, left waiting or dropped. +- The 818M versus 1,032M difference for 11-10. + +Evidence files (Starfire, session scratchpad, not retained): journal extracts per Mind, +Persona raw pulls, the turn table for 11-10 (`turns.csv`), forge PR rows (`rows.json`), +and the Starfire usage script.