CultLib/research/cultnet-distributed-database/summary.md

# CultNet Distributed Database Research Summary

Purpose: ground CultNet's distributed realtime database design in prior art
before adding more machinery. The goal is a coherent local-first mesh database
over CultCache, not a hand-rolled distributed-systems costume party.

## Sources Stored Here

- `hashgraph-swirlds-tr-2016-01.pdf`
  - Source: <https://www.swirlds.com/downloads/SWIRLDS-TR-2016-01.pdf>
  - Topic: gossip-about-gossip, virtual voting, fair ordering, asynchronous BFT.
- `raft-extended.html`
  - Source: <https://yygcode.com/papers/consensus-raft-extended-version.html>
  - Topic: understandable consensus, leader election, log replication, safety.
- `raft-extended.pdf`
  - Source: <https://web.stanford.edu/~ouster/cgi-bin/papers/raft-extended.pdf>
  - Topic: canonical Raft paper copy.
- `dynamo-amazon-science.html`
  - Source: <https://www.amazon.science/publications/dynamo-amazons-highly-available-key-value-store>
  - Topic: Dynamo publication page and abstract.
- `dynamo-sosp2007.pdf`
  - Source: <https://web.stanford.edu/class/cs244/papers/amazon-dynamo-sosp2007.pdf>
  - Topic: highly available key-value store, consistent hashing, versioning,
    quorums, hinted handoff, anti-entropy.
- `swim.pdf`
  - Source: <https://www.cs.cornell.edu/projects/quicksilver/public_pdfs/SWIM.pdf>
  - Topic: scalable weakly consistent process-group membership.
- `rethinkdb-changefeeds.html`
  - Source: <https://rethinkdb.com/docs/changefeeds/java/>
  - Topic: realtime query/changefeed ergonomics.
- `firebase-realtime-offline.html`
  - Source: <https://firebase.google.com/docs/database/web/offline-capabilities>
  - Topic: offline behavior, presence, server-side disconnect operations.
- `crdt-arxiv-1805.06358.html`
  - Source: <https://arxiv.org/abs/1805.06358>
  - Topic: conflict-free replicated data types and deterministic convergence.

## Design Takeaways

### RethinkDB

Keep: changefeeds as the product feel. Subscribers should receive document
changes continuously, with enough old/new context to render, reconcile, or
debug. Point subscriptions and filtered subscriptions are both first-class.

CultNet implication: database subscriptions should be explicit schema-v0
messages. The server should stream raw document changes that clients can apply
through the same CultCache reconciliation path as snapshots.

### Firebase Realtime Database

Keep: realtime sync and local/offline ergonomics. Presence and disconnect
behavior are database features, not application afterthoughts.

Defer: full offline writes. CultNet should not pretend arbitrary offline
multi-writer changes merge safely. Offline/local-first behavior needs declared
per-document conflict policy.

CultNet implication: add presence/disconnect records later as ordinary
CultCache documents with server authority, not as hidden transport state.

### Raft

Keep: the understandable authority model. One leader/primary owns ordered
writes for a shard. Decompose the problem into ownership, log/mutation
replication, and safety.

Defer: full automatic leader election and replicated logs until shard catalogs
and explicit epochs exist.

CultNet implication: the current primary-shard policy is the correct first
foundation. Every write should either hit the primary, be forwarded to the
primary, or fail with routing information.

### Dynamo

Keep: partitioning, replication metadata, vector-ish causality, hinted recovery,
anti-entropy, and application-visible conflicts.

Reject for now: "always writeable" semantics. That is attractive, expensive,
and easy to lie about.

CultNet implication: shard descriptors should grow into a shard catalog with
owner runtime id, epoch, schema/key ranges, and later replica/preference-list
metadata. Conflicts must surface as data, not disappear into last-writer-wins
unless a document explicitly chose that policy.

### SWIM

Keep: gossip-shaped membership once the mesh grows beyond a small static
cluster. SWIM separates failure detection from membership dissemination and
keeps per-node message load stable.

Defer: membership implementation until there is a shard catalog to disseminate.

CultNet implication: do not build peer discovery as a side-channel registry.
When it arrives, it should update membership and shard-catalog state together.

### CRDTs

Keep: CRDTs for documents whose merge law is explicit and deterministic.

Reject: generic automatic merge for arbitrary domain objects. That is a
language cop with a nicer hat.

CultNet implication: CRDT support belongs in schema metadata or document
contract metadata. A document type can opt into a known merge strategy; the
default distributed write policy remains primary authority.

### Hashgraph

Keep: gossip-about-gossip as an idea for compact event provenance and possible
fair ordering research. Virtual voting is interesting when every member sees
the same gossip history.

Reject for now: adopting hashgraph consensus as CultNet's core. The public
ledger/crypto smell is not the main issue; the issue is that CultNet does not
currently need asynchronous Byzantine total ordering to become a good mesh
database. Dragging that in now would make the machine harder to explain.

CultNet implication: if we later need decentralized event ordering, use a
dedicated design pass. Do not smuggle hashgraph metadata into ordinary document
replication because it sounds powerful.

## Current CultNet Direction

The coherent path is:

1. Primary-shard authority.
2. Explicit schema-v0 snapshot, put, delete, subscribe, unsubscribe, and change
   messages.
3. Shard catalog exchange with owner runtime id and epoch.
4. Optional forwarding from non-owner nodes to owners.
5. Membership/failure detection, likely SWIM-shaped.
6. Replication and failover, likely Raft-shaped per shard.
7. Optional CRDT policies for document types that deserve offline multi-writer
   semantics.
8. Optional gossip-history research if fair decentralized ordering becomes a
   real requirement.

## Live Invariants

- CultCache owns document identity, schema compatibility, local indexes, and
  reconciliation.
- CultNet owns transport, shard authority, subscriptions, and remote mutation
  delivery.
- Raw wire records must pass through `CultNetDatabase` before mutating local
  cache state.
- A write without authority is rejected or explicitly forwarded. It is never
  silently applied.
- Realtime change streams publish domain changes, not storage implementation
  details.
- Conflict policy must be declared before conflicting writes are accepted.

## Current Cut

Shard catalog exchange now exists:

- `cultnet.shard_catalog_request.v0`
- `cultnet.shard_catalog_response.v0`
- per-shard id, owner runtime id, epoch, schema ids, key prefix/range
- stale epoch rejection on writes
- routing error that tells clients where the primary lives when known
- injectable non-primary write forwarding policy
- concrete schema-v0 write forwarder for `cultnet://host:port` primary
  endpoints
- in-memory per-shard mutation logs and catch-up by last seen sequence
- wire-level shard mutation log catch-up with raw put/delete entries
- replica-side application of shard log responses with epoch checks, gap
  rejection, and idempotent replay
- background shard log replicator plus schema-v0 fetcher for primary endpoints
- restart-safe replica cursor store with a local MessagePack file
  implementation
- client authority scopes for predicted local input documents
- predicted/reconciled change events for client-side prediction
- simulation witness observations and deterministic consensus candidates
- schema-v0 simulation observation and candidate messages
- reactive observation hub for witness gossip and consensus candidates
- server-side observation bridge for observation messages and candidate replies

## Next Cut

Build durable authoritative mutation logs:

- persist per-shard primary logs
- define the snapshot fallback boundary for compacted history
- return explicit resync-required responses when requested history is gone
- then add simulation-frame rollback and resimulation helpers on top of
  predicted input streams
- add peer-to-peer fanout for observation gossip and candidate propagation

That is the next foundation needed before membership, Raft-style failover, or
CRDT policy work can be honest.