Research: token audit of the swarm 2026-11-05..11 (Self 3.26B, Persona 1.88B, root 2.38B prompt tokens; Persona fan-out, session accumulation, root subagent models; ranked plan)
Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
This commit is contained in:
parent
1bbd8a574c
commit
b1b2d63f7b
1 changed files with 206 additions and 0 deletions
206
references/research-token-audit-2026-11-11.md
Normal file
206
references/research-token-audit-2026-11-11.md
Normal file
|
|
@ -0,0 +1,206 @@
|
|||
# Token audit of the GameCult swarm, 2026-11-05 to 2026-11-11
|
||||
|
||||
Served: an Imagination pass for Eureka Self (root), 2026-11-11, after the operator's
|
||||
"Eureka needs to run lean, or we simply can't afford our own process." It answers where
|
||||
the tokens go, what they bought, why the waste made sense to the agents, and what to cut
|
||||
first. Evidence was gathered by eight Eyes subagents; nothing on any host was changed.
|
||||
|
||||
All times UTC. "Prompt tokens" means input + cache-read + cache-creation per API request
|
||||
(the journal's `request N prefix P`; `self_run.rs:395`). That is a gross count. About 96%
|
||||
of it is cache-read, billed on the API at 0.1x; cache-creation with the 1 h TTL the units
|
||||
use is 2x; output 5x. How the subscription meter weighs them is not known (see
|
||||
"Not measured"), so every saving below is given gross, with a note where the weighted
|
||||
effect is much smaller.
|
||||
|
||||
## 1. Where the tokens go
|
||||
|
||||
Three consumers, not one. The 818M figure for 2026-11-10 covered only the first.
|
||||
|
||||
| Consumer | Window | Requests | Prompt tokens | 2026-11-10 alone |
|
||||
|---|---|---|---|---|
|
||||
| Self runs, 8 Minds (Yggdrasil) | 11-05..11-11 06:00 | 23,398 | 3,263M | 1,032M (6,379 requests, 743 turns) |
|
||||
| Personas, 8 Minds (Yggdrasil) | 11-05..11-11 06:00 | 17,460 | 1,876M | 534M (4,515 requests, 4,351 turns) |
|
||||
| Root sessions (Starfire) | 11-04..11-11 | 12,203 | 2,379M | 286M |
|
||||
| Total | | 53,061 | 7,518M | 1,852M |
|
||||
|
||||
The journal gives 1,032M for Self runs on 11-10 UTC (1,069M counting unfinished turns),
|
||||
not 818M; the difference was not reconciled and is probably a day-boundary choice.
|
||||
|
||||
Rough API-weighted units (input + 2x creation + 0.1x read + 5x output; Self and Persona
|
||||
output and creation are extrapolated from four live transcripts, so treat as +-30%):
|
||||
Self about 0.6B, Starfire about 0.35 to 0.4B before any premium for Opus-class models,
|
||||
Persona about 0.25B.
|
||||
|
||||
### Self runs
|
||||
|
||||
- All Self and subagent requests run on `claude-sonnet-5-5`, effort medium, adaptive
|
||||
thinking, compaction disabled (`DISABLE_COMPACT=1`), context ceiling 300k.
|
||||
- A run starts at a median 34.8k tokens, and request 2 adds 18.3k: that is the Eureka
|
||||
`SKILL.md` (53.2k chars), invoked first in 4 of 4 live transcripts. Every later request
|
||||
carries it. Floor cost of the start prefix (P at request 2 x requests) is 1,229M, 37.7%
|
||||
of Self tokens. The skill alone is 18.3k x 23,398 = 428M (13.1%).
|
||||
- Request 1 by component (chars): tool schemas 25.6k, CLAUDE.md chain 33.3k (doctrine
|
||||
19.4k, gamecult.md 11.3k, seed 2.6k), output style 9.1k, skill listing 7.7k, system
|
||||
prompt 7.2k, agent listing 3.9k, first user message 3.2 to 6.2k of which 2.4k is the
|
||||
same every run. 11.4k tokens are cache-read at request 1; 20.6 to 24.5k are created
|
||||
fresh on every run, and about 19k on every subagent spawn.
|
||||
- Context growth, not orientation, is the mass. 59.6% of Self tokens sit in requests
|
||||
above 150k; 65.9% in the 86 runs (of 720 starts) with more than 100 requests. The first
|
||||
12 requests of a run are 4.6% of it.
|
||||
- A run is a session that swallows briefs one after another: on 11-10, 592 Persona
|
||||
`start_self_run` briefs and 87 unit starts, 743 turns in 63 working runs. Median
|
||||
context at the start of turn 1 is 34.8k; turn 2, 101.8k; turn 3, 126.1k; turn 4,
|
||||
137.4k; turn 5 and later, 208.4k. 518 of the 743 turns are turn 5 or later. Sessions
|
||||
end by Idle (52), Ceiling at 300k (39) or Rotation (15) since 11-09.
|
||||
- 345 of 720 starts made no request (325 lasted one second). They cost nothing.
|
||||
- The 11-10 outage (UnknownCaller, 07:39 to 12:09, 24 refused starts) cost no Self
|
||||
tokens: a refused lease starts no `claude`. Its token cost was a later sandbox fault
|
||||
(8 turns, 1.8M) and Persona chatter about it (54 mentions). The loss was time.
|
||||
|
||||
Activity on 11-10, classified from the closing text of the 80 costliest turns plus a
|
||||
random 40 (transcripts are not kept, so this is the best available; 90% intervals wide):
|
||||
|
||||
| Activity | Share of the day's Self tokens |
|
||||
|---|---|
|
||||
| ship (own change; about a fifth of it unmerged at turn end) | 36.9% (29.7 to 45.3) |
|
||||
| unclassifiable, mostly "Soul pass running, I'll wait" | 23.5% |
|
||||
| investigate, report only | 15.5% |
|
||||
| review a peer's work | 10.8% (4.7 to 17.1) |
|
||||
| coordinate | 10.3% (4.9 to 17.8) |
|
||||
| verify own work, no new change | 2.3% |
|
||||
| blocked | 0.8% |
|
||||
|
||||
Flags on the top 80 turns (45% of the day): a Soul pass is mentioned in turns holding
|
||||
64.8% of their tokens, mutation testing 36.0%, visual, cosmetic or docs-only change
|
||||
20.1%, re-do at a new head 14.7%, waiting for a go 13.9%. Verification is mostly inside
|
||||
"ship" turns and cannot be separated without usage logs. 146 turns (168.7M, 16.3%) close
|
||||
with only "waiting on Soul, Hands or a poll".
|
||||
|
||||
### Personas
|
||||
|
||||
- 16,041 turns in 5.5 days; about 540 per Mind on 11-10; median gap 0.8 min; 1.09
|
||||
requests per turn; mean 107k prompt tokens per request. A process starts at a median
|
||||
37.2k, grows about 1.1k per turn with no compaction, and ends at a median 83.3k after
|
||||
28 turns (median life 2.5 h).
|
||||
- The mouth (`Bifrost/tools/persona-matrix-mouth.mjs:315-332`) turns every non-self
|
||||
message in all 11 joined rooms into a pending delivery for every Persona. There is no
|
||||
addressed-only filter; addressing only sets the order (rank x 8 s stagger,
|
||||
`persona.rs:1982`). On 11-10: 1,079 Mind posts, 4,351 Persona turns, 4.0 turns per post.
|
||||
- 3,110 of those 4,351 turns (71.5%) said nothing and started nothing. 1,796 produced
|
||||
only an `intent` string; 1,126 an urgency vote; 178 a memory.
|
||||
- The prompt: 58.6k chars of appended instructions, of which `colossus.md` is 16.0k and
|
||||
`persona.md` 9.3k; a 20k-char first message of which "Your latest runs" is 9.6k.
|
||||
- 14% of turns emit a Self-run brief (2,093 in the window, 294 launches): the rest queue
|
||||
into a live Self session, which is the brief stream described above.
|
||||
|
||||
### Root sessions on Starfire
|
||||
|
||||
- One session (`6d32c5f0`, open since 11-04 20:54) holds 2,034M of the 2,379M: 10,556
|
||||
requests, 169 subagents, largest single request 967k tokens.
|
||||
- Subagents are 63.0% (1,498M): Sonnet 864M in 118 agents, Opus 629M in 89. Median
|
||||
subagent 5.0M prompt tokens, p90 16.1M. 185 of 216 Agent calls passed no model.
|
||||
- Main thread: about 2,219 requests for 881M, a mean of 397k per request.
|
||||
- 51 cache misses re-created 10.4M tokens; 18 followed a gap over an hour.
|
||||
|
||||
## 2. Output per token
|
||||
|
||||
Public forge only (gamecult-ops is private and was not counted; deploys and artifacts
|
||||
were not measured).
|
||||
|
||||
| Mind | PRs merged 11-04..11 | Merged 11-10 | Self tokens (M) | Self M per merged PR |
|
||||
|---|---|---|---|---|
|
||||
| libby | 104 | 24 | 692 | 6.7 |
|
||||
| eureka | 68 | 14 | 590 | 8.7 |
|
||||
| void | 64 | 33 | 522 | 8.2 |
|
||||
| weksa | 54 | 29 | 229 | 4.2 |
|
||||
| heimdall | 38 | 10 | 392 | 10.3 |
|
||||
| aqua | 34 | 19 | 232 | 6.8 |
|
||||
| nibu | 26 | 17 | 332 | 12.8 |
|
||||
| kiko | 3 | 0 | 274 | 91 (her work is mostly not PR-shaped) |
|
||||
|
||||
Swarm-wide, Self plus Persona: 13.1M prompt
|
||||
tokens per merged PR over the week, 10.1M on 11-10.
|
||||
|
||||
- 391 PRs merged in the week; 155 on 11-10, three times the daily mean. Median diff 171
|
||||
lines; 95 PRs are 30 lines or fewer.
|
||||
- 95% self-merged; median last push to merge 1.4 min; 81% have no comment or review on
|
||||
the forge. Review happens in chat and in Self turns, not on the forge: Bifrost #21 and
|
||||
#22 show zero forge interactions and a "Libby, round 3" in the body.
|
||||
- The sign-in footer cluster on the forge is three PRs (Ratatoskr #61, #62, #63: 62, 90
|
||||
and 24 lines, each merged within 22 minutes), plus adjacent sky-band PRs.
|
||||
|
||||
## 3. Structural causes
|
||||
|
||||
1. **The Persona is the address filter.** Nothing before the model decides whether a
|
||||
message concerns a Mind, so a 37k to 83k-token call is spent on each of 8 Minds per
|
||||
message to produce, 71.5% of the time, silence. Missing: routing by address before
|
||||
the model. The ordering logic that knows who is addressed already exists.
|
||||
2. **Chat volume becomes Self volume.** Each Persona that hears a message may emit a
|
||||
brief; briefs queue into the one live Self session; the session carries every earlier
|
||||
brief's context into the next unrelated one, uncompacted, to 300k. Missing: a session
|
||||
boundary that follows the work instead of the clock.
|
||||
3. **Self cannot speak or see across campaigns.** Libby: "I can't post in #swarm myself,
|
||||
since there is no chat tool here". Each relay costs a Persona turn and seven
|
||||
deliveries. Heimdall re-read Huginn #29 "because that Soul pass's mind doesn't hold
|
||||
the `eureka-body` or `eureka-substrate` campaigns": the author's Soul lacked the
|
||||
rulings, so a peer had to read. Nibu: "ready for Void to merge on your go": 11 of her
|
||||
PRs waited on the steward's merge.
|
||||
4. **The root never clears and its subagents inherit its model.** A week-old session at
|
||||
a mean 397k per main request, and 89 subagents on Opus because no agent definition
|
||||
names a model.
|
||||
5. **Nothing records usage.** Transcripts live on tmpfs and vanish at run end; the
|
||||
launcher discards the CLI's usage envelope; the connector logs "weekly reading
|
||||
refreshed" without the value. Agents and Self had no way to see what a habit cost.
|
||||
|
||||
## 4. Ranked plan
|
||||
|
||||
Savings are gross prompt tokens at the observed week's volume. Rank is by weighted
|
||||
effect and confidence, not by the gross number.
|
||||
|
||||
| # | Change | Owner | Expected saving and arithmetic | Shows it worked |
|
||||
|---|---|---|---|---|
|
||||
| 0 | Log one line per turn from the CLI result envelope (input, creation, read, output, model, subagent requests) and the meter value on each refresh. Adds a log line, no gate. Failure it prevents: this audit could not measure output, creation, subagent cost or quota weight, so item 4 cannot be costed. | Eureka (Huginn `self_run.rs`, connector) | None itself; makes every item below checkable | Usage lines in both journals; a per-day meter delta |
|
||||
| 1 | Deliver in real time only what addresses a Mind (rank 0 in the existing ordering, DMs, Self replies); hold the rest and hand it over with her next addressed turn or beat. Deletes the deliver-everything path. | Heimdall (Bifrost mouth) with Eureka (`persona.rs` stagger) | 71.5% of turns are silent; at 60 to 70% of 534M/day: 320 to 370M/day, 1.1 to 1.3B/week. Nearly all cache-read plus about 270 output tokens per turn. Second-order: fewer briefs into Self | Persona turns per Mind per day from about 540 to under 150; silent share under 30%; briefs per day down from 592 |
|
||||
| 2 | Root: name a model in each agent definition (Hands, Soul, Eyes, Modeling, Life on Sonnet or Haiku; Opus-class only where a definition argues for it), and clear the root at each handoff-area boundary | Root Self (agent definitions in the Eureka skill repo) | 629M/week of Opus subagent tokens move to Sonnet. Main thread from a mean 397k to under 150k per request: (397k - 150k) x 2,219 = 548M/week at the root model's price | Opus share of subagent tokens; mean main-thread prompt per request |
|
||||
| 3 | Confirm the skill split landed, then keep cutting the fixed prefix: the 2.0k constant brief-closing rules, the 19k created per subagent spawn, `colossus.md` (16.0k chars) and "Your latest runs" (9.6k chars) in the Persona prompt | Eureka | Split: 12k x 23,398 = 281M/week (8.6% of Self), if references are not re-read. Colossus: about 5.5k x 17,460 = 96M/week. Each further 1k off the Self floor: 23M/week | P(request 2) - P(request 1) at or below 7k (was 18.3k; no post-split run existed by 05:37 on 11-11); Persona first-turn prefix |
|
||||
| 4 | Lower the Self ceiling from 300k to about 150k, and end the session when a brief closes and none is queued | Eureka (`self_run.rs:99`) | At least the excess above 150k: 603M/week (18.5% of Self). Weighted effect is perhaps a third of that, since each restart re-creates about 20k tokens; do it after 3 and cost it with 0 | Share of Self tokens in requests above 150k from 59.6% to under 25%, with creation per day not rising |
|
||||
| 5 | Give Self the missing means: a chat post from inside a Self run; cross-campaign read of rulings for a Soul pass; merge in a steward's repo where the steward grants it | Eureka (Huginn), each steward for merge rights | Peer review is 10.8% of Self (about 110M/day on 11-10); halving it is about 55M/day. Relays: one Persona turn plus seven deliveries each | Review-peer share in a resample of closing texts under 5%; "please post this" relays gone |
|
||||
|
||||
Measure, do not re-propose: the 11-10/11 doctrine changes (one Soul pass, proportional
|
||||
verification). The baseline to beat is above: mutation mentioned in 36.0% and
|
||||
visual-only work in 20.1% of top-turn tokens, re-do at a new head 14.7%, peer review
|
||||
10.8%, waiting for a go 13.9%.
|
||||
|
||||
Forks for the operator or the steward, not decided here:
|
||||
- Item 1 removes ambient hearing in real time. 1,126 urgency votes a day depend on every
|
||||
Persona hearing everything. Is that vote a live mechanism worth about 350M tokens a day?
|
||||
- Item 3: the global doctrine says the Colossus long form is not loaded automatically,
|
||||
but every Persona loads it. Intent or drift?
|
||||
|
||||
Considered and not placed:
|
||||
- A 5-minute cache TTL for Self runs. 775 request gaps over 5 minutes would each
|
||||
re-create about 150k tokens: about 145M units lost against about 98M saved.
|
||||
- A fresh session per brief. Gross saving 44.5% of Self tokens, but about 1,960 extra
|
||||
restarts a week at about 52k weighted units each cancel most of it, and related
|
||||
briefs lose their context. Item 4 is the cheaper form.
|
||||
- Keeping transcripts. Item 0 gives the numbers without the disk and privacy cost.
|
||||
- A cheap-model pre-reader in front of each Persona. A new layer; the addressing rank
|
||||
already answers the question.
|
||||
- A cheaper model for Yggdrasil subagents. Not costed: no usage split exists (item 0).
|
||||
|
||||
## 5. Not measured
|
||||
|
||||
- Output, cache-creation and subagent tokens for Self and Persona; whether journal
|
||||
request lines include subagent requests; so the true cost of Soul passes and mutation
|
||||
runs is unknown.
|
||||
- Quota weight per token class and per model; the meter's values.
|
||||
- Tool-call level activity for any finished run (no transcripts); the classification
|
||||
rests on closing texts cut to 650 chars.
|
||||
- gamecult-ops #78 and all private repos; deploys and artifacts per Mind.
|
||||
- The effect of the 11-10/11 fixes: no run had started on the 17 KB skill by 05:37.
|
||||
- How many of the 2,093 Persona briefs were claimed, left waiting or dropped.
|
||||
- The 818M versus 1,032M difference for 11-10.
|
||||
|
||||
Evidence files (Starfire, session scratchpad, not retained): journal extracts per Mind,
|
||||
Persona raw pulls, the turn table for 11-10 (`turns.csv`), forge PR rows (`rows.json`),
|
||||
and the Starfire usage script.
|
||||
Loading…
Reference in a new issue