From 434db81e43e9b57c1d077e529667a600bd3fc3ce Mon Sep 17 00:00:00 2001 From: Meta Vens Date: Sun, 19 Jul 2026 17:13:51 +0100 Subject: [PATCH] Postmortem CultMesh bulk RUDP failure --- .../postmortem-2026-07-cultmesh-bulk-rudp.md | 609 ++++++++++++++++++ .../docs/reliability-migration-plan.md | 2 + 2 files changed, 611 insertions(+) create mode 100644 src/GameCult.Mesh/docs/postmortem-2026-07-cultmesh-bulk-rudp.md diff --git a/src/GameCult.Mesh/docs/postmortem-2026-07-cultmesh-bulk-rudp.md b/src/GameCult.Mesh/docs/postmortem-2026-07-cultmesh-bulk-rudp.md new file mode 100644 index 00000000..8ac88910 --- /dev/null +++ b/src/GameCult.Mesh/docs/postmortem-2026-07-cultmesh-bulk-rudp.md @@ -0,0 +1,609 @@ +# Postmortem: CultMesh bulk content over custom RUDP + +**Incident window:** 2026-07-12 through 2026-07-19 + +**Detected:** 2026-07-19 during the released EveUnity/Aetheria cold-cache witness + +**Scope:** CultMesh Phase 4 content delivery and the physical transport assumptions beneath it + +**Status:** architectural cause identified; RUDP tuning stopped; replacement data planes not yet integrated + +## Executive summary + +The CultMesh reliability migration correctly moved discovery, session reuse, +authority, content verification, resumable state, and final cache promotion out +of applications and renderers. It failed at the boundary between those control +planes and physical byte delivery. + +The canonical plan stated that the existing RUDP implementation already +covered acknowledgement, resend, ordering, duplicate suppression, +fragmentation, timeout, and bounded pending work. It therefore said the first +migration target was not a transport rewrite. That sequencing was reasonable +while CultMesh lacked identity-first sessions: replacing the wire first would +have preserved caller-owned reconnect and topology policy. + +The mistake came later. Once `CultMeshSessionManager` existed, the condition +named by the plan for adopting Iroh or QUIC had been satisfied. Instead of +benchmarking and attaching a proven content stream at that boundary, Phase 4 +introduced `CultMeshContentChunkRequestMessage` and +`CultMeshContentChunkResponseMessage` over the existing schema session. The +response embedded each raw chunk payload and therefore sent bulk bytes through +the custom RUDP fragmentation and retransmission engine. + +This preserved schema identity and moved cache authority to the correct owner, +but it did not produce the intended data plane. It changed the envelope around +the bytes while leaving the physical bottleneck intact. + +The failure was not detected promptly because the gates proved eventual +delivery and ownership, not transport fitness: + +- the initial content-session test used an in-memory schema client and server; +- the deterministic hostile-network harness tested its own schedule but was + not connected to the public content path or a physical connector; +- the later real-UDP regression allowed one MiB to take nearly ten seconds; +- the released cold witness allowed the artifact to finish under a 300-second + deadline; +- no test compared goodput, wire amplification, copies, or allocation against + TCP, QUIC, Iroh, or a same-machine mapping baseline; +- `SharedFileMapping` in witness output described mapping the client cache + after the network transfer, not direct access to a provider-owned local body. + +The result was a locally reliable but grossly inefficient bulk path. A +56,204,750-byte released Aetheria body caused 105,911,320 daemon transmit bytes +in one cold witness. A later controlled parity run emitted 129,241,066 server +bytes and delivered only 10.1 MiB/s, while the same body moved and verified at +250.7 MiB/s over TCP and 71.1 MiB/s over .NET's MsQuic-backed QUIC stream. An +existing mapped file opened in 10.9 ms without transferring the body. + +The ownership migration is not discarded. The physical content connector and +its gates must be replaced. CultNet remains the typed schema and dispatch +substrate; it must not imply that every payload belongs inside the custom RUDP +message plane. + +## Was the branch following the plan? + +Yes, substantially—and that is why this incident cannot be dismissed as one +implementation ignoring good documentation. + +| Plan commitment | Branch result | Verdict | +| --- | --- | --- | +| Stable identity and shared sessions before transport replacement | Discovery and session owners were implemented before content migration | Followed | +| Authority independent of contact and transport success | A mandatory resolver and provider/body authorization were added | Followed | +| One verified content-transfer owner | Resume state, range verification, failover, hashing, and atomic promotion moved into `CultMeshContentTransferService` | Followed | +| Remove renderer-owned chunk and cache authority | EveUnity delegates transfer and only lowers the verified result | Followed | +| Bounded adaptive concurrency | A fixed four-request window was added; adaptation was explicitly deferred | Partial | +| Caller-provided streaming destinations and backpressure | Not implemented before the managed content session became the released default | Not followed | +| Real consumer plus hostile-network proof at phase closure | A real released consumer ran, but the hostile harness did not drive that physical path | Not followed | +| Borrow QUIC/Iroh boundaries before dependencies | Connector/session/body boundaries were borrowed | Followed | +| Consider Iroh or QUIC after the boundary and harness exist | The prerequisite existed, but no mandatory comparison or adoption decision occurred | Missed transition | +| Public surface becomes smaller and more obvious | Application ownership shrank, but the physical RUDP machinery expanded | Mixed | + +The plan also lacked a throughput, amplification, flow-control, copy-budget, or +comparative transport gate. Consequently, the branch could follow much of the +written plan and still release an unfit bulk path. This is both a plan defect +and an execution defect: + +- **Plan defect:** it treated RUDP feature completeness as a sufficient reason + to defer physical transport evaluation and did not define the evaluation + trigger or fitness budget. +- **Execution defect:** once repeated cold failures supplied contrary evidence, + the work kept repairing RUDP locally instead of reopening the premise. + +## User-visible impact + +- A same-machine Aetheria daemon and Unity client copied a provider-owned asset + bundle through UDP instead of opening an authorized mapped body. +- The 0.3.91 released Unity test took 54.803 seconds. File timestamps place the + 56,204,750-byte body promotion roughly 31 seconds after partial creation. +- The 0.3.92/CultLib 1.0.33 witness reduced the Unity test to 36.028 seconds and + body transfer to roughly 12 seconds, but daemon transmit volume rose to + 105,911,320 bytes. The apparent speedup came partly from more aggressive + packet emission and retransmission. +- Cold startup competed with transient gameplay state and initially allowed a + live body publication lease to expire before subscription. +- Release and witness effort was repeatedly spent changing drains, sleeps, + ACK handling, request windows, and lifecycle ordering rather than selecting + the intended data plane. +- Remote behavior under real latency, loss, and bandwidth contention remains + unproven and is likely worse than loopback evidence. + +Content hashes, incremental chunk verification, final SHA-256 verification, +and atomic promotion prevented corrupted or partial bodies from becoming cache +truth. The incident is a performance and architecture failure, not known data +corruption. + +## Intended architecture + +The reliability plan's durable intent was: + +1. stable endpoint identity rather than application-owned physical endpoints; +2. shared long-lived sessions rather than request-scoped clients; +3. authority independent of contact and transport success; +4. content identity independent of provider and location; +5. verified resumable transfer owned outside renderers; +6. negotiated local mappings and remote streams behind the same body identity; +7. Iroh/QUIC lessons expressed in boundaries before dependencies were adopted. + +The intended phrase “CultNet as the transport substrate” meant that registered +schemas, typed envelopes, dispatch, and diagnostics remain portable across +transport connectors. It did not require every body byte to be serialized as a +schema-message field or fragmented by CultNet's RUDP implementation. + +## What was implemented + +The following pieces match the plan and remain useful: + +- `CultMeshDiscoveryService` owns route observations and freshness. +- `CultMeshSessionManager` owns identity-first connection reuse and path state. +- `CultMeshAuthorityResolver` separates contact from authorization. +- `CultMeshContentTransferService` owns partial files, verified ranges, + provider failover, final hashing, and atomic cache promotion. +- content hashes remain independent of provider identity and location. +- body publications preserve schema, layout, producer epoch, sequence, access, + lease, and transport representations. +- exact reactive subscriptions and SoA demand avoid broad per-frame snapshots. +- verified client cache files can be mapped without allocating another managed + body after transfer. + +The incorrect slice was the managed content session: + +```text +manifest chunk + -> CultMeshContentChunkRequestMessage + -> CultMeshContentChunkResponseMessage.Payload + -> MessagePack schema frame + -> custom RUDP fragmentation + -> per-packet ACK/resend/reassembly + -> managed byte[] chunk + -> verified partial file +``` + +The desired path is: + +```text +typed content request / authorized body capability + -> CultMesh session selects an advertised physical plane + -> same machine: provider-owned mapped file/shared memory + -> remote: proven independent byte stream/range fetch + -> existing verified partial/final promotion owner +``` + +Registered schemas remain the control language. They identify the body, hash, +range, lease, stream, progress, cancellation, and receipt. They do not need to +contain the body itself. + +## Timeline + +### Before the migration plan + +- **2026-06-14 to 2026-06-17:** commits `8ce89688` through `1c85eabe` + introduced the C# transport port, custom RUDP reliability core, socket + transport, fragmentation, schema ergonomics, reconnect support, and a + multi-peer listener. +- **2026-07-10 to 2026-07-12:** snapshot connection reuse and large-snapshot + fixes made RUDP the working physical path for the first Aetheria witness. + +### Reliability migration + +- **2026-07-12, `f0b5e103`:** the canonical reliability plan was committed. It + explicitly treated low-level RUDP reliability as present and deferred Iroh or + QUIC until session ownership and a failure harness existed. +- **2026-07-12, `fc65e25f`:** Phase 0 added a deterministic fault scheduler and + diagnostics. The harness could replay hostile events, but it was not wired to + the public content transfer or actual transport connectors. +- **2026-07-13, `b9f17a71`:** `CultMeshContentTransferService` correctly took + ownership of resumable verification and atomic promotion. +- **2026-07-14, `7d571402`:** the decisive wrong fork. Managed content sessions + embedded chunk bytes in typed schema responses over the existing session. + The primary test used an in-memory loopback transport and therefore proved + ownership and message shape without exercising RUDP. +- **2026-07-14, `a311cbe8`:** fixed request concurrency and RUDP polling to + improve throughput. The plan explicitly deferred adaptive scheduling and + streaming destinations instead of reopening physical-plane selection. +- **2026-07-14, `b943fa7e`:** a 13,006,384-byte cold run completing under the + unchanged 300-second deadline was recorded as content-session delivery proof. +- **2026-07-16, `25231429`:** a physical RUDP content test was finally added. + Its one-MiB payload only had to finish within ten seconds. It recorded wire + statistics on failure but asserted no goodput or lossless amplification + budget. +- **2026-07-16, `15bffc6d`:** a 46,412,384-byte released cold witness was + recorded as proof while explicitly acknowledging that the managed session + still copied and fragmented the body. +- **2026-07-17 to 2026-07-19:** ACK synchronization, concurrent session + serialization, exact body demand, reactive delivery, and live subscription + work made the path more correct under concurrency. None tested whether RUDP + deserved to carry bulk bytes. +- **2026-07-19, `2afec5c5`:** per-32-packet `Thread.Sleep(1)` calls were replaced + by `Thread.Yield`. The cold transfer became faster while wire volume nearly + doubled, revealing that the system was flooding loopback rather than + controlling flow. +- **2026-07-19, `ee0fb235`:** the first dedicated transport parity harness + compared the Aetheria-sized public content path with mapped-file, TCP, and + MsQuic-backed QUIC baselines. The result rejected further RUDP bulk tuning. + +## Detection + +The incident surfaced only after the released-package witness became fully +cold and provider-owned assets grew to 56,204,750 bytes. Earlier warm-cache +runs bypassed transfer. Earlier cold runs were framed as timeout/reliability +problems rather than fitness problems. + +The decisive signals were: + +1. transfer duration remained many seconds on loopback; +2. removing scheduler sleeps reduced duration but drove daemon transmit volume + to almost twice the body size in the released witness; +3. the dedicated parity harness showed a stable order-of-magnitude gap; +4. opening an existing provider body locally required milliseconds and no + transfer, proving that the same-machine witness was selecting the wrong + plane before RUDP performance even mattered. + +## Root cause + +### Primary cause: capability was mistaken for fitness + +The plan listed ACK, resend, ordering, duplicate suppression, fragmentation, +timeout, and bounded pending work and concluded that transport replacement was +not the first target. Those properties can establish eventual reliable message +delivery. They do not establish congestion control, receiver flow control, +packet pacing, batching, stream multiplexing, loss recovery quality, or bulk +goodput. + +The custom RUDP path had a 32-packet acknowledgement mask, 1,024-byte fragments, +managed per-fragment bookkeeping, and no congestion window tied to receiver +progress. A sender could emit multiple 256-KiB chunk responses concurrently. +Yielding after 32 sends yielded CPU time; it did not create backpressure or +wait for acknowledgement progress. + +### Secondary cause: “CultNet substrate” was underspecified + +The plan did not say plainly enough that CultNet owns typed schema semantics +independently of physical transport. Implementers treated “preserve CultNet as +the transport substrate” as a reason to carry content through the available +RUDP schema channel. This collapsed two distinct claims: + +- CultNet documents and registered schemas remain the portable protocol. +- the current CultNet RUDP implementation is suitable for every payload class. + +Only the first claim was required. + +### Secondary cause: the QUIC/Iroh adoption trigger had no gate + +The plan correctly warned against adopting Iroh or QUIC before session and +failure boundaries existed. It did not define a mandatory decision point once +those boundaries existed. Phase 2 delivered the connector/session seam, but +Phase 4 did not require candidate transports to pass a fitness comparison +before becoming the content default. + +The deferral therefore became sticky. “Not yet” silently became “continue using +RUDP.” + +### Secondary cause: evidence harnesses observed the wrong layer + +Phase 0's deterministic network proved that a fault schedule could replay +loss, duplication, delay, reordering, partition, corruption, endpoint rotation, +and restart. It did not drive the RUDP socket transport, session manager, or +content transfer service. A self-test of the harness was treated as sufficient +foundation for later phase proof. + +The first managed-content test used an in-memory schema bridge. It proved that +content did not use snapshot records, sessions were reused, hashes matched, and +warm cache avoided another request. It could not reveal fragmentation cost, +socket flooding, resend amplification, scheduling delays, or contention with +other messages. + +The later physical test's ten-second bound for one MiB was a hang detector, not +a performance contract. The 300-second released witness deadline was likewise +a liveness ceiling, not a native-experience gate. + +### Secondary cause: local failures were repaired locally + +Each symptom had a plausible narrow explanation: + +- snapshot payloads starved peers; +- poll loops slept after control packets; +- fragment sends slept per packet; +- ACKs fell outside the rolling mask; +- concurrent publishers mutated one reliability session; +- content subscription leases expired during cold loading. + +The fixes were individually defensible and often necessary for small-message +correctness. Together they kept attention inside RUDP. No rule required a +transport parity check after two transport-level fixes failed to produce an +acceptable cold path. + +This is the process failure described by the project's Jenga warning: the +machine supplied enough local context to justify each patch while withholding +the comparative evidence needed to question the owner. + +### Secondary cause: post-transfer mapping obscured pre-transfer cost + +The witness reported `SharedFileMapping` because EveUnity mapped the verified +client cache file before loading the Unity bundle. That was true at the body +lease layer, but incomplete at the user-visible layer: the provider-owned file +had already crossed RUDP into the client cache. + +The metric did not lie, but it observed the wrong interval. No durable witness +fact distinguished: + +- provider-owned same-machine mapping with zero body transfer; +- client-cache mapping after remote delivery; +- managed network-buffer fallback. + +### Secondary cause: the plan became both specification and evidence ledger + +The plan accumulated phase body maps, implementation notes, witness results, +and partial-proof language in the same document that still contained the +original `Cut first`, `Build`, and `Gate` contracts. That made useful progress +visible, but it blurred the difference between these two statements: + +- a coherent slice of the phase has evidence; +- the phase's closure contract has been satisfied. + +In Phase 4, content ownership, verification, and released-consumer delivery had +evidence while adaptive scheduling, caller-provided streaming destinations, +backpressure, hostile-network coverage of the real physical path, and transport +fitness remained open. The document could therefore tell an increasingly +convincing success story without forcing the remaining gate failures back to +the top of the decision process. + +Evidence belongs beside the plan, but phase status must be derived only from an +explicit closure checklist. A partial proof may close a named slice; it cannot +silently weaken the parent phase gate. + +## Five whys + +1. **Why was Aetheria cold asset delivery slow?** + + The provider sent a large immutable body as thousands of reliable UDP + fragments with heavy retransmission and managed per-packet work. + +2. **Why was bulk content sent through that path?** + + Phase 4 represented chunks as typed request/response schema messages on the + existing CultMesh session. + +3. **Why was the existing session assumed suitable?** + + The plan equated implemented reliability features with transport fitness and + deferred Iroh/QUIC until the session boundary existed. + +4. **Why was the assumption not revisited after the boundary existed?** + + There was no mandatory transport parity gate, and every test emphasized + ownership, eventual completion, or failure recovery rather than comparative + goodput and amplification. + +5. **Why did repeated fixes not trigger redesign?** + + The workflow classified each failure as another RUDP correctness defect and + rewarded a passing witness under generous timeouts. It lacked an escalation + rule that says repeated physical-transport repairs reopen transport + selection. + +## What went well + +- Content identity, verification, restart state, provider failover, and atomic + promotion were moved to a coherent owner and survived the incident. +- The released-package witness eventually exercised a truly cold cache and + exposed the problem at the user-visible boundary. +- Daemon transport counters made wire amplification observable. +- The later parity harness reused the public transfer path instead of comparing + only synthetic packet loops. +- The unreleased two-chunk/Sleep(0) tuning experiment was reverted rather than + released after it showed improved duration with unacceptable wire behavior. +- The plan had preserved a transport connector boundary, so replacing the + physical plane does not require returning authority to EveUnity or Aetheria. + +## What went poorly + +- Research direction was treated as future inspiration after its adoption + prerequisite had already been met. +- The deterministic failure harness was not integrated with the systems whose + reliability it purported to de-risk. +- “Large” tests used 128 KiB, 700 KiB, or one MiB while the consumer used 13 to + 56 MiB artifacts. +- Test thresholds detected hangs but certified them as performance proof. +- Lossless wire amplification was not a release gate. +- The cold witness separated gameplay from content for valid lifecycle reasons, + but that separation also reduced pressure to make cold startup feel native. +- Plan updates faithfully described partial slices yet allowed “proven” wording + before the full Phase 4 build and gate lists were complete. +- The same-machine deployment advertised and selected network delivery before + mapping the resulting client cache. + +## Corrective architecture + +### Owner + +`CultMeshSessionManager` owns physical-plane selection for an endpoint identity +and application protocol. `CultMeshContentTransferService` owns verification, +resume state, and final promotion. A selected transport connector owns byte +movement. CultNet schema dispatch owns none of the physical reliability policy. + +### Inputs + +- stable endpoint and provider identity; +- authorized content hash, byte size, ranges, and lease; +- provider-advertised physical candidates; +- consumer-supported candidates; +- same-machine evidence; +- platform capabilities and measured connector health. + +### Outputs + +- same-machine: an authorized provider-owned mapped file or shared-memory body; +- remote: an independent proven stream/range fetch feeding the transfer owner; +- typed progress, cancellation, failure, and receipt state over CultNet. + +### Derived state + +Selected plane, endpoint address, stream identifiers, byte progress, path +health, and performance telemetry are derived. They do not alter content hash, +provider authority, or cache truth. + +### Forbidden writers + +- schema-message helpers cannot embed bulk body payloads; +- renderers cannot choose physical endpoints or write final cache entries; +- a body adapter cannot confer provider authority; +- RUDP retry, chunk-window, or timeout tuning cannot make RUDP the preferred + bulk plane; +- a client-cache mapping cannot claim same-machine zero-copy provenance. + +### Shared paths + +Every remote connector terminates in the existing verified-range and atomic +promotion primitives. Local mapping and remote fallback preserve the same body +identity, schema/layout, epoch/sequence, access mode, and lease semantics. + +### Deletion line + +1. Stop adding bulk-performance behavior to custom RUDP. +2. Prevent `CultMeshContentChunkResponseMessage.Payload` from being the default + content body path. +3. Make RUDP an explicit compatibility/control candidate, not the implicit + connector for `cultmesh.content.v1`. +4. Delete RUDP content serving once every released consumer negotiates local + mapping or a dedicated remote stream. +5. Retain RUDP only for payload classes whose own fitness gates justify it; + eventual cleanup may remove the custom reliability engine entirely. + +## Corrective actions + +### Completed + +- [x] Add `tools/GameCult.TransportParity` using the public CultMesh transfer + owner and Aetheria-sized payloads. +- [x] Compare mapped-file open, TCP stream, MsQuic-backed QUIC stream, and + CultMesh RUDP on the same host. +- [x] Record duration, goodput, and RUDP server wire bytes. +- [x] Stop and revert the unreleased request-window/pacing experiment. +- [x] Amend the reliability plan with the physical body-plane ownership cut. + +### Required before another bulk transport release + +- [ ] Make same-machine evidence select a provider-owned mapped body before any + network fetch. +- [ ] Attach at least one proven remote stream connector behind + `CultMeshSessionManager` and the content protocol identity. +- [ ] Keep content request, authorization, range, progress, cancellation, and + receipt state typed while moving body bytes out of schema response payloads. +- [ ] Integrate the deterministic hostile-network schedule with the actual + connector and public content transfer path. +- [ ] Exercise Aetheria-sized payloads, not only unit-sized fixtures. +- [ ] Add cold and warm released-consumer witnesses that report plane + provenance, bytes sent, bytes copied, promotion time, and total lowering time. +- [ ] Prove remote cancellation, range resume, provider failover, and corruption + recovery over the selected stream. +- [ ] Define and enforce platform packaging for Unity, daemon, TypeScript, and + other promoted runtimes before making the connector the public default. + +### Required transport fitness gates + +The precise budgets may be tightened as deployment data accumulates, but a +candidate cannot become the default without all of these categories: + +- **Comparative goodput:** integrated verified transfer measured against the + same connector's raw stream baseline on the same host and route. +- **Lossless amplification:** total wire bytes bounded relative to body bytes; + success alone is insufficient. +- **Small-message latency:** control messages and receipts remain responsive + while a body transfers. +- **Backpressure:** sender progress follows receiver and destination capacity; + thread yields and timeout retries are not accepted as flow control. +- **Hostile network:** loss, duplication, reordering, latency, partition, + reconnect, and path change drive the actual public path. +- **Memory/copy budget:** allocation and copy counts distinguish provider-owned + mapping, streamed promotion, client-cache mapping, and managed fallback. +- **User-visible cold start:** a released generic client lowers the world within + an explicit experience budget; a multi-minute timeout is only a deadlock + guard. + +### Escalation rule + +After two fixes in the physical transport layer for the same consumer outcome, +work stops and transport selection is reopened against dedicated baselines. +Further local transport repair requires evidence that the current connector +still satisfies its fitness budget. This is not a ban on fixing bugs; it +prevents repeated correctness patches from impersonating architectural proof. + +## Plan amendments + +The reliability plan should now be read with these corrections: + +1. “CultNet as transport substrate” means typed schema identity, framing, + dispatch, diagnostics, and connector contracts. It does not bless a single + physical transport for all payloads. +2. Implemented ACK/resend/fragmentation is evidence of reliable-message + capability, not evidence of bulk, realtime, or media fitness. +3. Once the session connector boundary exists, physical candidates must pass a + workload-specific parity gate before a consumer migration closes. +4. Typed control state and body transport are separate. Registered schemas make + the control plane smaller; they do not require body bytes to become document + fields. +5. Same-machine mapping is selected before transfer. Mapping a cache file after + transfer is a useful lowering optimization but not zero-copy delivery. +6. A phase slice may be recorded as partial evidence, but “proven” and “closed” + require the complete build list, gates, real consumer, and performance + evidence relevant to that slice. + +## Evidence + +### Source and history + +- Canonical plan: `src/GameCult.Mesh/docs/reliability-migration-plan.md`, commit + `f0b5e103`. +- Managed content session fork: commit `7d571402`. +- Fixed request concurrency and drain work: commit `a311cbe8`. +- First physical one-MiB RUDP content regression: commit `25231429`. +- Scheduler-yield release: commit `2afec5c5`. +- Dedicated parity harness and initial body-plane correction: commit + `ee0fb235`. + +### Released witness + +- EveUnity `artifacts/cold-release-0391/results.xml`: 54.803-second Unity test. +- EveUnity `artifacts/cold-release-0392/results.xml`: 36.028-second Unity test. +- EveUnity `artifacts/cold-release-0392/aetheria-daemon.log`: final daemon + counters `rx=3903892 tx=105911320` for the cold run. +- Both witness documents report a 56,204,750-byte provider asset and + `SharedFileMapping`; the mapping occurred after network promotion into the + client cache. + +### Parity harness + +- Payload: 56,204,750 bytes. +- Existing mapped file: 10.9 ms to open and touch; no body transfer. +- TCP plus file write and SHA-256 verification: 213.8 ms, 250.7 MiB/s. +- MsQuic-backed QUIC plus file write and SHA-256 verification: 753.4 ms, + 71.1 MiB/s; repeated runs reached 87.0 to 107.4 MiB/s. +- CultMesh RUDP public content transfer: 5,322.9 ms, 10.1 MiB/s, + 129,241,066 server bytes; repeated runs reached 10.1 to 13.0 MiB/s. + +## Accountability + +Git author identity does not distinguish human-authored work from agent work in +this repository, so this postmortem does not infer personal authorship from the +commit metadata. The failure belongs to the engineering process and especially +to the agent execution that kept accepting narrow transport repairs as progress +toward the user's native-experience requirement. + +The user explicitly warned that cold asset delivery was not healthy enough to +claim, that larger timeouts must not conceal it, and that the daemon architecture +could not cost game performance. The eventual switch from “times out” to +“finishes before the deadline” did not satisfy those constraints. The agent +should have introduced comparative transport evidence at the first Phase 4 +physical failure, not after several releases and witness iterations. + +## Closing statement + +The reliability migration did not fail because it learned from Iroh and QUIC. +It failed because it stopped at their control-plane lessons after building the +seam where their transport machinery was meant to attach. + +The repair is not to discard the new discovery, session, authority, transfer, +or body ownership. It is to finish the architecture: keep registered schemas as +the small typed language of CultNet, attach proven physical transports beneath +the session boundary, and make every promoted plane prove that it deserves the +workload before a released consumer depends on it. diff --git a/src/GameCult.Mesh/docs/reliability-migration-plan.md b/src/GameCult.Mesh/docs/reliability-migration-plan.md index 89296dbf..bcab4537 100644 --- a/src/GameCult.Mesh/docs/reliability-migration-plan.md +++ b/src/GameCult.Mesh/docs/reliability-migration-plan.md @@ -1,5 +1,7 @@ # CultMesh Reliability Migration Plan +Incident record: [CultMesh bulk content over custom RUDP, July 2026](postmortem-2026-07-cultmesh-bulk-rudp.md). + ## Objective CultMesh should give an application one typed, watchable route to a Verse,