# CultNet Distributed Database Research Summary Purpose: ground CultNet's distributed realtime database design in prior art before adding more machinery. The goal is a coherent local-first mesh database over CultCache, not a hand-rolled distributed-systems costume party. ## Sources Stored Here - `hashgraph-swirlds-tr-2016-01.pdf` - Source: - Topic: gossip-about-gossip, virtual voting, fair ordering, asynchronous BFT. - `raft-extended.html` - Source: - Topic: understandable consensus, leader election, log replication, safety. - `raft-extended.pdf` - Source: - Topic: canonical Raft paper copy. - `dynamo-amazon-science.html` - Source: - Topic: Dynamo publication page and abstract. - `dynamo-sosp2007.pdf` - Source: - Topic: highly available key-value store, consistent hashing, versioning, quorums, hinted handoff, anti-entropy. - `swim.pdf` - Source: - Topic: scalable weakly consistent process-group membership. - `rethinkdb-changefeeds.html` - Source: - Topic: realtime query/changefeed ergonomics. - `firebase-realtime-offline.html` - Source: - Topic: offline behavior, presence, server-side disconnect operations. - `crdt-arxiv-1805.06358.html` - Source: - Topic: conflict-free replicated data types and deterministic convergence. ## Design Takeaways ### RethinkDB Keep: changefeeds as the product feel. Subscribers should receive document changes continuously, with enough old/new context to render, reconcile, or debug. Point subscriptions and filtered subscriptions are both first-class. CultNet implication: database subscriptions should be explicit schema-v0 messages. The server should stream raw document changes that clients can apply through the same CultCache reconciliation path as snapshots. ### Firebase Realtime Database Keep: realtime sync and local/offline ergonomics. Presence and disconnect behavior are database features, not application afterthoughts. Defer: full offline writes. CultNet should not pretend arbitrary offline multi-writer changes merge safely. Offline/local-first behavior needs declared per-document conflict policy. CultNet implication: add presence/disconnect records later as ordinary CultCache documents with server authority, not as hidden transport state. ### Raft Keep: the understandable authority model. One leader/primary owns ordered writes for a shard. Decompose the problem into ownership, log/mutation replication, and safety. Defer: full automatic leader election and replicated logs until shard catalogs and explicit epochs exist. CultNet implication: the current primary-shard policy is the correct first foundation. Every write should either hit the primary, be forwarded to the primary, or fail with routing information. ### Dynamo Keep: partitioning, replication metadata, vector-ish causality, hinted recovery, anti-entropy, and application-visible conflicts. Reject for now: "always writeable" semantics. That is attractive, expensive, and easy to lie about. CultNet implication: shard descriptors should grow into a shard catalog with owner runtime id, epoch, schema/key ranges, and later replica/preference-list metadata. Conflicts must surface as data, not disappear into last-writer-wins unless a document explicitly chose that policy. ### SWIM Keep: gossip-shaped membership once the mesh grows beyond a small static cluster. SWIM separates failure detection from membership dissemination and keeps per-node message load stable. Defer: membership implementation until there is a shard catalog to disseminate. CultNet implication: do not build peer discovery as a side-channel registry. When it arrives, it should update membership and shard-catalog state together. ### CRDTs Keep: CRDTs for documents whose merge law is explicit and deterministic. Reject: generic automatic merge for arbitrary domain objects. That is a language cop with a nicer hat. CultNet implication: CRDT support belongs in schema metadata or document contract metadata. A document type can opt into a known merge strategy; the default distributed write policy remains primary authority. ### Hashgraph Keep: gossip-about-gossip as an idea for compact event provenance and possible fair ordering research. Virtual voting is interesting when every member sees the same gossip history. Reject for now: adopting hashgraph consensus as CultNet's core. The public ledger/crypto smell is not the main issue; the issue is that CultNet does not currently need asynchronous Byzantine total ordering to become a good mesh database. Dragging that in now would make the machine harder to explain. CultNet implication: if we later need decentralized event ordering, use a dedicated design pass. Do not smuggle hashgraph metadata into ordinary document replication because it sounds powerful. ## Current CultNet Direction The coherent path is: 1. Primary-shard authority. 2. Explicit schema-v0 snapshot, put, delete, subscribe, unsubscribe, and change messages. 3. Shard catalog exchange with owner runtime id and epoch. 4. Optional forwarding from non-owner nodes to owners. 5. Membership/failure detection, likely SWIM-shaped. 6. Replication and failover, likely Raft-shaped per shard. 7. Optional CRDT policies for document types that deserve offline multi-writer semantics. 8. Optional gossip-history research if fair decentralized ordering becomes a real requirement. ## Live Invariants - CultCache owns document identity, schema compatibility, local indexes, and reconciliation. - CultNet owns transport, shard authority, subscriptions, and remote mutation delivery. - Raw wire records must pass through `CultNetDatabase` before mutating local cache state. - A write without authority is rejected or explicitly forwarded. It is never silently applied. - Realtime change streams publish domain changes, not storage implementation details. - Conflict policy must be declared before conflicting writes are accepted. ## Current Cut Shard catalog exchange now exists: - `cultnet.shard_catalog_request.v0` - `cultnet.shard_catalog_response.v0` - per-shard id, owner runtime id, epoch, schema ids, key prefix/range - stale epoch rejection on writes - routing error that tells clients where the primary lives when known - injectable non-primary write forwarding policy - concrete schema-v0 write forwarder for `cultnet://host:port` primary endpoints - in-memory per-shard mutation logs and catch-up by last seen sequence - wire-level shard mutation log catch-up with raw put/delete entries - replica-side application of shard log responses with epoch checks, gap rejection, and idempotent replay - background shard log replicator plus schema-v0 fetcher for primary endpoints - restart-safe replica cursor store with a local MessagePack file implementation - client authority scopes for predicted local input documents - predicted/reconciled change events for client-side prediction - simulation witness observations and deterministic consensus candidates - schema-v0 simulation observation and candidate messages - reactive observation hub for witness gossip and consensus candidates - server-side observation bridge for observation messages and candidate replies ## Next Cut Build durable authoritative mutation logs: - persist per-shard primary logs - define the snapshot fallback boundary for compacted history - return explicit resync-required responses when requested history is gone - then add simulation-frame rollback and resimulation helpers on top of predicted input streams - add peer-to-peer fanout for observation gossip and candidate propagation That is the next foundation needed before membership, Raft-style failover, or CRDT policy work can be honest.