CultLib/contracts/cultnet/cultnet-distributed-database.md

CultNet Distributed Database Notes

CultNet's distributed database path is a typed realtime database over CultCache documents. The design should borrow proven distributed-systems ideas without copying their entire machine or laundering vague consistency through impressive words.

Prior Art To Keep

• RethinkDB changefeeds: subscriptions should emit old/new document views and support point, table/schema, and filtered watches.
• Firebase Realtime Database: client ergonomics should make realtime sync feel like the default path, but presence, disconnect handling, and server-side authority still need explicit ownership.
• Dynamo: partitioning, replication metadata, vector-ish causality, hinted recovery, and application-visible conflict resolution are useful. "Always writeable" is not the first CultNet promise.
• Raft: primary authority per shard is the first understandable consistency model. Leader election/log replication can come later if the cluster needs automatic failover.
• SWIM: peer membership and failure suspicion should be gossip-shaped when the mesh grows beyond small static clusters.
• CRDTs: offline multi-writer documents are valid only for types with declared, deterministic merge semantics.
• Hashgraph: gossip-about-gossip is interesting for compact event provenance and fair ordering research. CultNet should not adopt public-ledger machinery or crypto-token assumptions as a shortcut to database correctness.

Current Cut

Implemented:

• CultNetDatabase: database-style facade over a CultCache
• primary-shard write checks
• typed R3 watch streams
• raw CultNet put/delete application through shard policy
• CultNetDatabaseServer: schema-v0 bridge for snapshot, put, and delete
• Client and Server schema-v0 dispatch hooks
• CultNetHost.Database and CultNetHost.DatabaseServer
• subscription request/cancel/change messages
• server-side live change fanout for schema/key-filtered subscriptions
• shard catalog request/response messages
• shard descriptors with owner runtime id, epoch, schema filters, key prefix, primary endpoints, replica endpoints, read replica endpoints, and region
• epoch-aware raw put/delete messages
• routing hints on shard authority errors
• injectable non-primary write forwarding policy
• schema-v0 write forwarder that dials advertised rudp://host:port primary endpoints, while keeping cultnet://host:port as the LiteNetLib compatibility lane
• in-memory per-shard mutation logs with sequence numbers, epochs, mutation kind, schema id, key, and document references
• mutation-log catch-up by last seen sequence
• wire-level shard log request/response messages
• raw put/delete representation for accepted mutation-log entries
• replica-side shard log application with epoch checks, gap rejection, and idempotent replay
• shard log replicator that pulls non-primary shards from catalog-advertised primary endpoints
• schema-v0 shard log fetcher for rudp://host:port endpoints, while keeping cultnet://host:port as the LiteNetLib compatibility lane
• restart-safe replica cursor store, including a local MessagePack file-backed implementation
• durable authoritative shard-log storage behind ICultNetShardMutationLogStore, including a per-shard MessagePack file implementation
• shard-log compaction watermarks; requests for compacted history return ResyncRequired with reason = "compacted_history"
• shard-bounded snapshots with shard id, epoch, and represented log sequence
• replica snapshot recovery after compacted history, including local shard replacement and cursor advancement
• CultMesh node option for file-backed authoritative shard logs with a default .cultmesh/shard-logs path beside the cache file
• schema-v0 Verse catalog request/response messages
• CultMesh Verse discovery server bridge over the local Verse catalog
• schema-v0 peer exchange request/response messages
• CultMesh peer cards, peer catalog, peer exchange server bridge, and peer exchange client
• CultMesh authority leases and lease catalog checks for peer role/shard authorization
• client-authority scopes for locally predicted input documents
• predicted and reconciled database change kinds for client-side prediction
• simulation witness observations for mesh-side opinions about frame facts
• deterministic consensus candidates derived from witness observations
• committed CultMesh simulation fact documents written from quorum candidates through shard authority
• gameplay-facing CultMesh session facade for prediction, observations, catalogs, peer exchange, and quorum fact commits
• schema-v0 simulation observation and consensus candidate messages
• reactive simulation observation hub that receives witness gossip and emits consensus candidates
• server-side observation bridge that accepts observation messages and replies with current candidate messages

Not implemented yet:

• SWIM-style membership/failure detection beyond Python's hello-level health monitor
• leader election or automatic shard failover
• cross-runtime scheduled peer-to-peer snapshot fanout beyond Python's local fanout loop
• cross-runtime scheduled peer-to-peer fanout for observation gossip and candidate propagation beyond Python's local fanout loop
• cross-runtime rollback/resimulation helpers for simulation frames
• declared CRDT merge policies

Error Codes

cultnet.error.v0 carries an optional machine-readable code. The set of codes is a closed enum (cultnet.error.schema.json): Rust and TypeScript refuse a message whose code they do not know, they do not guess. Adding a code is therefore a coordinated change across every runtime, made together: the schema (both copies), the C# CultNetErrorMessage factory, the Rust CultNetErrorCode, and a captured-bytes case in packages/cultnet-rs/tests/error_message.rs for it. TypeScript reads the schema. The current codes are selection_invalid, cursor_stale, cursor_invalid, reference_outside_target, variant_unsupported, unowned_schema (a write, or a remote put or delete, for a schema and key no shard owns) and write_not_authorized (a peer's put or delete its server's remote write authority did not grant).

Live Invariants

• CultCache owns document schema, identity, local indexes, and local diffing.
• CultNet owns transport, shard authority, remote mutation delivery, and subscription fanout.
• One primary owns writes for a shard until a stronger policy exists.
• Non-owner writes are rejected or explicitly forwarded. They are not silently applied.
• A peer's put or delete is applied or forwarded only when the server's RemoteWriteAuthority grants it; none configured refuses every remote write. The connection key admits a peer to the transport and is never a grant.
• Raw snapshot/put/delete messages must pass through CultNetDatabase, not bypass shard policy.
• The accepted shard log is stored in replica catch-up wire form. Object logs are for local inspection; wire logs are the durable replication authority.
• Log retention is part of the replication contract. A replica asking before the compaction watermark must resync from a snapshot before applying newer log entries.
• A shard snapshot represents a log sequence. Applying it replaces the local shard view and advances the replica cursor to that sequence.
• Realtime subscriptions publish domain changes, not storage envelopes.
• Client-owned input documents may be predicted locally only inside explicit CultNetClientAuthorityScope declarations.
• Predicted input state is not authoritative shared state. The shard log still decides the committed ordering and emits reconciliation when it corrects or confirms local prediction.
• Witness observations are immutable reports about a shard epoch, frame, subject, claim kind, and claim hash.
• Consensus candidates are derived from observations. They may inform the authoritative commit path, but they are not themselves committed world state until a shard owner writes them into the log.
• CultMeshSimulationFact is the committed form of a quorum candidate. It is written through CultNetDatabase, so normal shard authority and replication rules still apply.
• CultMeshGameSession composes common gameplay surfaces. It does not create a second authority path; all committed state still flows through shard-owned database writes.
• Peer cards are contact candidates. They can advertise roles and authority lease ids, but they do not grant authority without Verse-specific validation.
• Authority leases bind peer id, Verse id, roles, optional shard scope, issuer, validity window, and signature. They are the first trust boundary above peer exchange.

Next Coherent Slice

Add discovery fanout and operational polish:

• standardize periodic gossip fanout over known peer cards beyond Python's local schedulers and health monitor
• standardize cross-runtime authority lease signature algorithms beyond the package-local Python HMAC verifier
• standardize cross-runtime simulation-frame rollback/resimulation helpers beyond Python's local pending-prediction rollback surface
• standardize cross-runtime scheduled peer-to-peer snapshot fanout beyond Python's local fanout loop
• standardize cross-runtime scheduled peer-to-peer fanout for observation gossip and candidate propagation beyond Python's local fanout loop

The log is wire-readable, replicas can apply it explicitly, a pull loop can drive non-primary shards from primary endpoints, and replica cursors can survive restart. Client-owned input documents can now be predicted locally and reconciled when the authoritative log arrives. Nodes can also aggregate witness observations into deterministic consensus candidates for simulation facts like "who shot first," and those reports now have schema-v0 wire contracts. The next useful layer is making Verse discovery and operator defaults first-class. Membership and leader election should wait until basic replica catch-up is boring. Authoritative mutation logs can now survive process restart through ICultNetShardMutationLogStore; compaction now has an honest resync boundary, and the shard replicator can recover from that boundary through snapshot replacement.