Carry media payloads as bin

The three media payload fields take the derive's `bytes` attribute, so they
serialize as MessagePack `bin` instead of an array of integers. An 848-byte
payload goes from 1234 bytes on the wire to 851 — 1.45x off every media byte,
on the one link measured to be starved.

This is a correction to the previous commit's reasoning, not a new capability.
That commit claimed the derive could not emit bin and left a test ignored
against a hint that needed designing. The hint has existed all along:
cultcache-rs-derive parses `bytes` at lib.rs:92, serializes through
serde_bytes::Bytes at :135, and decodes through ByteBuf at :154 and :170. What
is inert on a DatabaseEntry struct is `#[serde(with = "serde_bytes")]`, because
the derive reads only its own attribute namespace — which is a different and
much smaller claim than the one that was made.

The ignored test is now un-ignored and passing, and the tolerant assertion it
was propping up is exact again.

Four fields elsewhere are missing `bytes` and are deliberately untouched here:
IdunnDeploymentBrakeRecord.signature, IdunnDaemonHealthTrustBindingRecord
.signer_public_key, GameCultServiceTrustAnchorRecord.signer_public_key, and
IdunnAuthenticatedProviderHealthProjectionRecord.signature. Those carry stored
state and signing history — six trust anchors live in the deployment projection
on yggdrasil, and the brake store is byte-exact checked — so they need
sequencing with a re-sign rather than a flag flip. Owned by the Swarm Migration
work.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
Meta Vens 2026-09-10 00:31:11 +02:00
commit 2ba404cc0b
2 changed files with 21 additions and 25 deletions

View file

@ -18,6 +18,12 @@
//! wire format: append new fields with `default` at the next free index, never
//! reorder or renumber. `missing_video_chunk_keys` shows the pattern.
//!
//! Payloads carry `bytes` so they serialize as MessagePack `bin` rather than an
//! array of integers, which is what the C#, TypeScript and Python runtimes
//! expect and what keeps a 848-byte payload at 851 bytes instead of 1234. Note
//! that this is the derive's own attribute — `#[serde(with = "serde_bytes")]`
//! is inert here, because `DatabaseEntry` writes its own `Serialize`.
//!
//! # Timebase
//!
//! `pts_ticks`, `duration_ticks` and `deadline_ticks` are expressed in
@ -76,7 +82,7 @@ pub struct GameCultMediaVideoAccessUnitRecord {
pub chunk_index: u16,
#[cultcache(key = 12)]
pub chunk_count: u16,
#[cultcache(key = 13)]
#[cultcache(key = 13, bytes)]
pub payload: Vec<u8>,
}
@ -123,7 +129,7 @@ pub struct GameCultMediaVideoParityShardRecord {
pub chunk_payload_bytes: u32,
#[cultcache(key = 15)]
pub last_chunk_payload_bytes: u32,
#[cultcache(key = 16)]
#[cultcache(key = 16, bytes)]
pub payload: Vec<u8>,
}
@ -153,7 +159,7 @@ pub struct GameCultMediaAudioPacketRecord {
pub timebase_den: u32,
#[cultcache(key = 8)]
pub deadline_ticks: i64,
#[cultcache(key = 9)]
#[cultcache(key = 9, bytes)]
pub payload: Vec<u8>,
}

View file

@ -67,33 +67,23 @@ fn video_access_unit_encodes_as_a_positional_array() {
assert_eq!(slots[8].as_bool(), Some(true), "slot 8 is keyframe");
assert!(slots[9].is_nil(), "an absent dependency stays nil, not 0");
assert_eq!(slots[12].as_u64(), Some(5), "slot 12 is chunk_count");
assert!(
matches!(&slots[13], Value::Binary(_) | Value::Array(_)),
"slot 13 is the payload; see payload_should_encode_as_messagepack_bin \
for which of the two it ought to be"
assert_eq!(
slots[13].as_slice(),
Some([0xDE, 0xAD, 0xBE, 0xEF].as_slice()),
"slot 13 is the payload, carried as bin"
);
}
/// Ignored until `cultcache-rs-derive` grows a `bytes` hint.
/// Payloads must reach the wire as MessagePack `bin`, not as an array of
/// integers. The C#, TypeScript and Python runtimes all read byte fields as
/// `bin`, so a payload serialized as a sequence is unreadable by the reference
/// implementation, and costs 1.45x besides: 848 bytes encode to 1234 as an
/// integer array against 851 as `bin`.
///
/// The derive writes its own `Serialize` and calls `serialize_element` per slot,
/// so a `Vec<u8>` field goes out as a MessagePack array of integers. C#,
/// TypeScript and Python all emit `bin` for byte fields, which makes Rust the
/// divergent runtime and these records unreadable by the reference
/// implementation. `#[serde(with = "serde_bytes")]` cannot fix it — the derive
/// ignores serde attributes.
///
/// It also costs 1.45x on every media byte: an 848-byte payload encodes to 1234
/// bytes as an integer array against 851 as `bin`.
///
/// The fix is not local to this file. Changing byte encoding globally would
/// invalidate every signature in the Idunn authority chain, because signatures
/// are taken over the whole positional serialization and canonical hashes over
/// the same bytes. It needs an opt-in hint so these three media records can
/// adopt `bin` without moving the signature-bearing ones. Owned by the Swarm
/// Migration work; un-ignore this when the hint lands.
/// This is what the derive's `bytes` attribute buys. `#[serde(with =
/// "serde_bytes")]` will not do it — `DatabaseEntry` emits its own `Serialize`
/// and reads only its own `cultcache` attribute namespace.
#[test]
#[ignore = "blocked on a `bytes` opt-in hint in cultcache-rs-derive"]
fn payload_should_encode_as_messagepack_bin() {
let encoded = rmp_serde::to_vec(&access_unit()).expect("encodes");
let Value::Array(slots) = decode_as_value(&encoded) else {