Improve managed content throughput

This commit is contained in:
Meta Vens 2026-07-14 03:26:43 +01:00
commit a311cbe8cb
8 changed files with 292 additions and 62 deletions

View file

@ -41,3 +41,15 @@ network descriptor without changing logical identity.
The content protocol uses one bounded response per manifest chunk. A snapshot
response containing `gamecult.mesh.cdn_artifact_chunk.v1` payload records is a
legacy transport path and must not be used for bulk delivery.
The transfer owner may keep a bounded window of requests in flight per content
hash (four by default, never more than 32). Candidate chunks can arrive out of
order, but only the transfer owner writes them, and it flushes and checkpoints
them in manifest-offset order. If one request fails, the owner observes the
rest of that window before returning; fetched-but-uncommitted bytes never become
trusted resume state. Concurrent artifacts have independent windows.
RUDP schema hosts should drain immediately available transport work with
`PollAvailableAsync`. Its result distinguishes transport items consumed from
application messages dispatched, so ACK and connection traffic counts as
progress without impersonating a message or triggering an idle delay.

View file

@ -1,10 +1,10 @@
# Distributed CDN Documents
CultMesh CDN artifacts are typed CultCache documents, not a separate asset
transport. The same manifest and chunk records can be written by an asset
pipeline daemon, replicated through CultNet raw snapshots or shard logs, and
read by Unity, TypeScript, or another runtime that implements the CultCache
document contract.
CultMesh CDN artifacts use typed CultCache storage and a separate managed
content-delivery session. Manifests may travel through snapshots or shard logs;
artifact payload chunks remain in provider storage and cross the network through
`cultmesh.content.v1`. Unity, TypeScript, and other runtimes consume the same
manifest and verification contract without making snapshots a bulk byte lane.
This is the canonical path for moving Aetheria assets into CultLib. Starbridge
Unity and Starbridge Electron should consume the same records so UI textures,
@ -67,11 +67,11 @@ updates.
Clients should:
1. Fetch or receive a manifest by record key, tag, or index.
2. Fetch every chunk record referenced by `manifest.Chunks`.
3. Verify each chunk hash before copying bytes.
4. Reassemble chunks by `Offset`.
5. Verify the full `ContentHash`.
6. Interpret bytes according to `MimeType` plus domain metadata.
2. Give it to `CultMeshContentTransferService` with one or more content providers.
3. Let that owner request missing chunks through managed content sessions.
4. Use only the atomically promoted `<content-hash>.body` returned after chunk
and full-body verification.
5. Interpret bytes according to `MimeType` plus domain metadata.
Single-channel data stays single-channel. For example, a gravity map can use
raw bytes with metadata such as `channels=r32` instead of pretending to be an
@ -147,27 +147,20 @@ await CultMesh.PublishCdnArtifactAsync(node.Cache, artifact);
For build distribution, use the same API with `Kind = CultMeshCdnArtifactKinds.Build`
and metadata such as `platform=windows-x64` or `channel=internal`.
## CultNet Replication
## CultNet Publication And Delivery
CDN documents use the regular raw document lane:
Publish manifests and chunks into the provider cache:
```csharp
var registry = CultMesh.CreateCdnDocumentRegistry(node.Cache.Registry);
var request = registry.CreateSnapshotRequest(
"cdn-request",
schemaIds:
[
node.Cache.Registry.GetRequired<CultMeshCdnArtifactManifest>().SchemaId,
node.Cache.Registry.GetRequired<CultMeshCdnArtifactChunk>().SchemaId
],
recordKeys: artifact.RecordKeys.Select(key => key.Value));
var response = registry.CreateRawSnapshotResponse(node.Cache, "cdn-response", request);
await CultMesh.PublishCdnArtifactAsync(providerCache, artifact);
using var contentServer = new CultMeshContentServer(schemaServer, providerCache);
```
The TypeScript client should mirror the C# schema names, schema versions,
record-key rules, chunk hashing, and verification order. C# remains the
reference implementation.
Snapshots may publish the manifest record, catalog, and body-control documents.
They must not publish `CultMeshCdnArtifactChunk` payload records for bulk
delivery. Consumers use `CultMeshSessionContentProvider` and
`CultMeshContentTransferService`; C# remains the reference implementation for
record keys, hashing, resumability, and atomic promotion.
## Entity Prefabs
@ -187,7 +180,8 @@ the same package graph into their native entity/component model.
Unity and TypeScript clients should implement the same rules:
- Fetch prefab packages by record key, tag, or version catalog.
- Fetch every referenced artifact manifest and chunk.
- Fetch every referenced artifact manifest, then delegate payload transfer to
the generic content-transfer owner.
- Verify signed manifests before trusting package membership.
- Verify chunk hashes and full artifact hashes before import.
- Cache artifacts locally by content hash.

View file

@ -551,9 +551,10 @@ atomic promotion primitive. Concurrent requests for one hash share the same
per-content transfer lock and verified work.
**Cut line:** the legacy distributed reader no longer owns a sequential chunk
loop or whole-artifact allocation. The first slice intentionally leaves
adaptive parallelism, quotas, pinning, signer policy, arbitrary streaming
destinations, and cleanup scheduling to later Phase 4 work.
loop or whole-artifact allocation. The transfer owner now opens a bounded
per-content request window while preserving one ordered writer/checkpointer.
Adaptive scheduling, quotas, pinning, signer policy, arbitrary streaming
destinations, and cleanup scheduling remain later Phase 4 work.
**State transition:** each verified chunk flushes a typed checkpoint. Restart
rehydrates that record, rehashes every claimed range, and fetches corrupt or
@ -563,8 +564,10 @@ partial state.
**Verification layer:** focused tests cover durable cancellation/restart,
provider corruption and failover, forged checkpoint rejection, final-hash
failure, atomic visibility, and concurrent request coalescing. Existing CDN
tests cover the compatibility reader.
failure, atomic visibility, concurrent request coalescing, bounded concurrent
fetch, ordered checkpointing under out-of-order completion, and observation of
the whole request window after an early failure. Existing CDN tests cover the
compatibility reader.
### Phase 4 managed content-session slice
@ -600,8 +603,10 @@ descriptors; the managed content protocol carries bounded payload responses.
**Verification layer:** Mesh tests transfer a multi-chunk body through one
managed session, assert zero snapshot requests, assert warm committed reuse
without another network request, and round-trip both wire messages through the
schema dispatcher. The released EveUnity/Aetheria cold witness remains the real
RUDP consumer proof required to close the slice.
schema dispatcher. RUDP schema-host tests distinguish control-packet progress
from dispatched messages and drain bounded available work without idle sleeps.
The released EveUnity/Aetheria cold witness remains the real consumer proof
required to close the slice.
### Phase 5 verified CDN mapped-body slice