Research: prior art for a materialized read image in Huginn (serves eureka-substrate:follow_up:huginn-materialized-image)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

docs/research/huginn-materialized-image.md

@@ -0,0 +1,109 @@
+# Prior art: a materialized read image advanced by a single writer, with snapshots for cursors
+
+Serves follow-up `eureka-substrate:follow_up:huginn-materialized-image` under operator ruling `eureka-substrate:ruling:readback-gate-and-materialized-image`. Campaign `eureka-substrate`, session `self-2026-10-25-eureka`. Written by Eyes (`eyes-materialized-image`) on 2026-10-26. Facts carry evidence pointers; inference is marked "(inferred)". No recommendation is made in the body; the last section lists what each option would delete and cost, for the design fork.
+
+Method note: web sources were read through a summarizing fetch tool, not verbatim. Claims about third-party systems are as summarized from the cited page; where a page did not say something, that is stated. Nothing was measured in Huginn; the 200 ms / 11k figure is the brief's.
+
+## 1. What Huginn does today (code facts)
+
+Paths are under `F:\Projects\Huginn\crates\huginn-mind\src\` unless noted.
+
+**Storage and the in-memory image**
+- `Mind::open` takes the redb store's exclusive per-path lock for the Mind's lifetime (`mind.rs:130-147`; refusal `MindAlreadyOwned`). One process, one writer.
+- `open_checked` pulls all envelopes, runs the gates, then `attach` (`mind.rs:183-191`). `Mind.image` is a `Vec<CultCacheEnvelope>` (`mind.rs:102`), the raw encoded envelopes (msgpack payload bytes), in identity order. `Mind::envelopes()` hands it out (`mind.rs:214`).
+- After every commit, `refresh()` runs `cache.pull_all_backing_stores()` then `image = snapshot(&cache)` (`mind.rs:274-278`), called from `receipt::commit` (`receipt.rs:246-286`, the call at `:271`). The redb backing store's `pull_all` opens a redb `begin_read()` and reads the whole table (`F:\Projects\CultLib\packages\cultcache-rs\src\lib.rs:2349-2350`). So the write path is also linear in the mind: every admitted batch re-reads every envelope from redb.
+- Commit is a compare-and-swap batch: strong reads as expectation, writes plus the receipt as replacements (`receipt.rs:253-270`). Receipts are ordinary envelopes of type `huginn.mind_commit_receipt.v2` (`receipt.rs:28, 97`) with a dense ordinal `1..=N`; `head()` decodes every receipt, sorts ordinals, and refuses unless they are exactly `1..=N` (`receipt.rs:204-213`).
+
+**The read path (`query.rs`)**
+- `Mind::view` (`query.rs:456-460`): `receipt::head` (decodes all receipts), then `Reader::at(self, head)`, then a find and `view_of`.
+- `Mind::query` (`query.rs:469-487`): validates, takes `as_of` from the cursor (`Cursor::parse`, refused if beyond head, `:473-484`) or head, then `Reader::at(self, as_of)?.rows()?`, then `select`. Members/ranked paths build a Reader the same way (`:532-533`, `:589-590`).
+- `Reader::at` (`query.rs:380-391`): `Docs::from_image(mind.envelopes())` decodes every envelope of a known kind into `Held { kind, key, document }` (`docs.rs:27-54`); `AdmissionIndex::build` calls `head` again and decodes every receipt again, building a `BTreeMap<(type, key), AdmissionFacts>` over all writes, refusing if two receipts write one identity (`query.rs:315-340`); then it keeps `Held` whose receipt `ordinal <= as_of` and drops the rest. "Built once per call and thrown away with it" (`query.rs:365-367`).
+- `held()` is a linear `iter().find` (`query.rs:393-395`); `Docs::find`/`in_image` likewise (`docs.rs:65-75`); `Mind::raw_envelope` is a linear scan over the envelope Vec (`mind.rs:261-263`).
+- Status is derived per document: `view_of` calls `Docs::closing_resolution` (`query.rs:397-405`), which scans all resolutions and, per candidate, calls `in_force(Resolution, key)`, which itself is `closing_resolution_unless(...).is_none()` (`docs.rs:165-183, 204-206`). `rows()` does this for every held document plus a `base()` walk (`query.rs:420-439`). Cost shape (inferred from the code, not profiled): decode O(N), receipt decode O(R) twice, rows O(N x Rz) where Rz is the number of resolutions, recursion depth bounded by the Q19 cap at two (`query.rs:414-419`). The 200 ms may be dominated by the status scan rather than decode; that is a hypothesis to profile (inferred).
+- Admission builds its own `Docs::from_image(self.envelopes())` per batch (`admission.rs:127`; also `index.rs:128`), so admission pays the full decode as well.
+
+**Invariants the image must keep (from doc comments)**
+- A snapshot at `as_of` is the documents whose writing receipt has `ordinal <= as_of`, with every derivation (in force, closing resolution, base, citations) computed over that image alone (`query.rs:374-379`).
+- Every held document must have its receipt at any `as_of`; an orphan or a double write refuses at any snapshot (`query.rs:377-379, 351`).
+- `writes` only: a document later cited as a strong read keeps the receipt that wrote it (`query.rs:311-314`). A10: one write per identity, which means documents are written once and never overwritten in place; the only "change" is a later resolution document (inferred from `query.rs:311-314`; a persona put is the stated exception, `receipt.rs:259-260`).
+- Cursors carry `as_of` and are minted with a per-process random key (`mind.rs:190`, `CursorKey::random()`), so a cursor does not outlive the process (inferred: a restart makes old cursors unverifiable).
+
+Consequence worth stating plainly (inferred): because identities are written once and receipts are dense and monotone, the set of documents visible at `as_of` is a prefix of the receipt chain. A document's own introduction ordinal is immutable. Only derived status (closed by a resolution) varies with `as_of`, and it varies monotonically: a resolution with ordinal r closes its subject for every `as_of >= r`. This is the structure the prior art below is being asked to exploit.
+
+## 2. MVCC read views
+
+**PostgreSQL.** Under Repeatable Read the snapshot is taken at the first non-transaction-control statement and every later statement in the transaction sees the same committed data (PostgreSQL docs, "Transaction Isolation", https://www.postgresql.org/docs/current/transaction-iso.html). A snapshot can be exported with `pg_export_snapshot()` and imported by another session via `SET TRANSACTION SNAPSHOT`; the exporting transaction must stay open for the id to remain importable (https://www.postgresql.org/docs/current/functions-admin.html). Failure mode: an old row version must be kept while any transaction may still see it, so open or idle-in-transaction sessions, prepared transactions and replication slots hold back cleanup and cause bloat (https://www.postgresql.org/docs/current/routine-vacuuming.html; the xmin-horizon mechanism is named in the page's diagnostic queries and "outside the page" note, i.e. partly the fetch tool's general knowledge).
+- Relevance: PostgreSQL's cursor-like long snapshot is a held transaction. The cost of serving an old snapshot is paid in retained versions, by the writer's storage, for as long as the snapshot is held. A Huginn cursor has no connection and may be resumed minutes later; holding an engine transaction per outstanding cursor is the Postgres exported-snapshot model and carries its bloat risk (inferred).
+
+**LMDB.** Single writer, readers take no locks; read transactions pin old page versions, so "the database can grow quickly" under long-lived readers, and stale readers from aborted processes cause the same; docs advise avoiding long-lived transactions, checking stale readers with `mdb_reader_check`, and note the reader table has a fixed slot count (default 126) (lmdb.h comments, https://raw.githubusercontent.com/LMDB/lmdb/mdb.master/libraries/liblmdb/lmdb.h). `MDB_NOTLS` lets a read transaction cross threads.
+- Relevance: same cost class as redb below.
+
+**redb (already in Huginn).** MVCC on a copy-on-write B+tree. A read transaction takes a private copy of the root and registers itself, so pages that root references are not freed while it runs; freed pages are queued and reused only after readers that could reference them finish (redb `docs/design.md`, https://github.com/cberner/redb/blob/master/docs/design.md; also the 1.0 post, https://www.redb.org/post/2023/06/16/1-0-stable-release/, which says readers and writers do not block each other). `ReadTransaction` is `Send + Sync`, closes when it and all objects referencing it drop (https://docs.rs/redb/latest/redb/struct.ReadTransaction.html). Savepoints register as read transactions to preserve a snapshot and cost allocator-state copy of about 64 kB per GB of values at 4 kB pages; persistent savepoints survive restart, must be deallocated explicitly, and keep their pages from reclamation (design.md; the space-growth statement is the fetch tool's inference).
+- What a redb read transaction gives for free: a consistent, lock-free view of the whole table as of the last commit before it began, for the lifetime of the handle, with no copy. Huginn's `pull_all` already uses it, but only to copy every envelope into a Vec and drop the transaction (`cultcache-rs lib.rs:2349-2350`). Nothing in the pages read gives a way to open a read transaction at an arbitrary older commit; older views exist only as held read transactions or savepoints (the pages describe no "open at txid" API; absence in the fetched text, not proof).
+- What it does not give: decoded documents, a status index, or an ordinal filter. It would remove the in-process image copy and its refresh, not the decode and derivation work (inferred).
+- Failure mode for cursors: one pinned transaction per live cursor-snapshot prevents page reuse, so the redb file grows while cursors are held, and there is no natural expiry for a cursor that an agent abandons (inferred from the reuse rule).
+- Note a structural mismatch (inferred): redb's snapshots are per commit of the engine; Huginn's `as_of` is per receipt. Because each admitted batch is one compare-and-swap commit with its receipt in the same write transaction (`receipt.rs:268-270`), engine commits and receipt ordinals correspond 1:1, but redb does not expose the mapping and cannot be asked for "the view as of ordinal 12,345" after the fact.
+
+## 3. Immutable database values and persistent maps
+
+**Datomic.** Information accumulates and is never removed; databases are immutable values, so compute instances need no coordination for query and caches hold only immutable data (https://docs.datomic.com/whatis/architecture.html). Queries merge persistent index trees with an in-memory representation of recent change; new index trees are built occasionally by background indexing jobs (same page; the pages fetched did not use the terms novelty/memory index there). The operations page defines the memory index as transaction data logged but not yet in a disk index, triggers an indexing job at `memory-index-threshold` (default 32m), lets the transactor continue until `memory-index-max` (default 512m), and applies radical backpressure if novelty exceeds max before indexing finishes (https://docs.datomic.com/operation/capacity.html).
+- `as-of` is a filter over the same indexes, not a stored copy; filtering "is not free"; entity-plus-time queries jump via EAVT then drop newer datoms; time-only questions would need a scan of the whole database (https://docs.datomic.com/reference/filters.html). The current db from a connection does no filtering, so "now" pays no history penalty.
+- Relevance (inferred): this is the closest match to Huginn's data model: append-only, writes never overwritten, and `as_of` is a filter on a monotone position (Datomic `t`, Huginn receipt ordinal). The Datomic answer to "older snapshot" is not to keep versions; it is to keep every datom with its `t` and filter. Applied here: keep each document once, tagged with its introduction ordinal, and keep each resolution tagged with its ordinal; an `as_of` read filters by tag. A Datomic filter pays per touched datom, not per mind, because it goes through an index.
+- Failure modes: the novelty/indexing split exists because the in-memory representation must be bounded; backpressure is the cost of the writer outrunning the indexer. Huginn's single-writer, low-rate admission is far from that regime (inferred).
+
+**Persistent (structural-sharing) maps in Rust.**
+- `imbl`: clone is a refcount increment on the root; nodes are copied on write only when shared; `HashMap` is "almost neck and neck" with std, `OrdMap` about 2-3x slower than std on basic operations, `Vector` loses to `Vec` on memory and indexing; sole-owner in-place mutation is about an order of magnitude faster than immutable operations; default pointer is `Arc` (thread safe), `Rc` and `triomphe` options exist (https://docs.rs/imbl/latest/imbl/).
+- `rpds`: `Vector`, `HashTrieMap`, `RedBlackTreeMap`; thread safety is opt-in via `_sync` constructors using `Arc`, which is "significantly slower to clone and drop than Rc"; mutable methods are faster when old versions are not needed (https://github.com/orium/rpds).
+- Relevance (inferred): a persistent map gives one owner advancing a head version per admission and cheap retention of any older version by holding its root. Memory cost of N retained versions is the changed paths, not N copies. Hazards: per-version retention has no expiry unless something drops the root (a cursor table), and `OrdMap`/trie ops cost a constant factor over a plain map for reads. Because Huginn's history is a prefix chain, each version differs from its predecessor by a few documents, so path-copy cost per admission is small (inferred); but the same prefix property means a version could be reconstructed by filtering rather than retained (see Datomic).
+- Handing a version to readers: `arc-swap` provides an atomically swappable `Arc` for read-mostly state, one writer publishing, many lock-free readers each keeping the version it loaded (https://docs.rs/arc-swap/latest/arc_swap/). The page notes the crate is niche, and that 1.9 strengthened memory orderings with probable cost; it did not discuss guard hold times.
+
+## 4. Event-sourced projections with a checkpoint
+
+**Marten (PostgreSQL).** An async daemon processes events in order; a high-water mark (HWM) advances contiguously and stops at gaps; each projection shard records its own progression; rebuilds via `projections --rebuild` or `RebuildProjectionAsync` (https://martendb.io/events/projections/async-daemon.html). Failure modes listed there: non-transactional PostgreSQL sequences burn numbers on rollback, creating gaps that stall the HWM; long-running or idle-in-transaction sessions can stall it; forcing a skip (`SkipStaleGapsDespiteLiveTransactionsAfter`, `AdvanceHighWaterMarkToLatestAsync`) can permanently skip events that commit later; slow pages beyond `StopAndDrainTimeout` get cancelled and reprocessed each shutdown; poison events go to a dead-letter table; migrating an inline projection to async needs downtime.
+- Relevance (inferred): the gap hazard comes from multiple concurrent appenders with non-transactional sequence numbers. Huginn has one writer and a dense ordinal checked as `1..=N` in `head` (`receipt.rs:204-213`), so the HWM and gap machinery have no counterpart; the projection's checkpoint is simply "head ordinal applied".
+
+**EventStoreDB / Kurrent, Akka Projection.** The Kurrent projections page URL fetched returned 404 on the second path; the Akka overview page says only that an offset per event lets a projection resume after restart, with offsets kept in Cassandra, JDBC, R2DBC or Slick, and did not state delivery guarantees or rebuild procedure. Neither was verified further here. Not evidence for or against.
+
+**Fowler, "Event Sourcing".** State can be derived by replaying the log from a blank state, to any point; replay is "a slow process, particularly if there are many events"; snapshots of application state speed restarts and can be built in parallel; "Complete Rebuild" discards derived state and replays (https://martinfowler.com/eaaDev/EventSourcing.html).
+- Relevance (inferred): Huginn is already its own event log (the receipts) and its own store (the envelopes). The projection here is derived from the same file that holds its checkpoint. On a crash or version change the image rebuilds from the envelopes (what `Reader::at` does today per read), so "rebuild on version change" is the existing code run once at open instead of per read. A persisted projection (a second store) would add a checkpoint that can disagree with the file (inferred: a second owner of the head ordinal).
+
+## 5. Incremental view maintenance for derived status
+
+**DBSP (Budiu, McSherry, Ryzhyk, Tannen).** A language for computation over streams, with an algorithm that incrementalizes arbitrary programs, covering relational queries, aggregation and recursion (https://arxiv.org/abs/2203.16684, abstract only; the PDF fetch returned unreadable binary, so claims about cost bounds were not verified; the fetch tool's general-knowledge recollection is that distinct and aggregation need integrated state that grows with input).
+**Differential dataflow / Materialize arrangements.** Arrangements are shared indexed state serving many queries; logical compaction (`set_logical_compaction(frontier)`) declares that times before the frontier will no longer be queried, letting the trace merge them, after which reads of older times are not guaranteed; holding trace handles prevents compaction (https://timelydataflow.github.io/differential-dataflow/chapter_5/chapter_5_3.html).
+- Relevance (inferred): Huginn's "status" is a two-level lookup (resolution subject -> closing resolution, `docs.rs:171-183`), not a recursive query. A general IVM engine is machinery well beyond that derivation. The transferable lesson is the compaction frontier: an indexed state that serves several times needs an explicit declaration of the oldest time still readable, otherwise it never shrinks. For Huginn the analogue is a cursor TTL. The simple incremental alternative the evidence supports is recompute-per-key-on-write: when a resolution with subject S is admitted, record `closed_by[S] = (resolution key, ordinal)`; `view_of` is then one map lookup. Status at `as_of` is `closed_by[S]` if its ordinal <= as_of (inferred, from the monotone-closure property in section 1; A8 and the Q19 cap admit at most one closing resolution per document, `docs.rs:168-170`, so no multi-valued history is needed for that case; resolutions that are themselves resolved make `in_force(Resolution)` depend on the same lookup, depth capped at two).
+
+## 6. What each option would mean here
+
+Common frame: single owner of the image (the `Mind`); an admission advances it after the redb commit; `Mind::view/query` read it. Memory figures are order-of-magnitude (inferred) with decoded document size unknown; assume 1-3 KB decoded per document: 11k docs about 10-35 MB; 1M docs about 1-3 GB. The current per-read build allocates and frees that on every call; the options differ in how much stays resident.
+
+| Option | Owner / advance | Old `as_of` served by | Resident memory | Deletes from Huginn | Main risk |
+|---|---|---|---|---|---|
+| A. Hold redb read transactions as the snapshot (per cursor) | Image stays derived per read from a held transaction; no resident image | Keep the transaction open per live cursor | Engine pages for pinned versions, grows with write churn while cursors live | `refresh()` re-pull and the `image` Vec could go (inferred); not decode or status scan | Unbounded cursor lifetime pins file growth (LMDB/redb rule); does not fix the 200 ms, which is decode plus derivation (inferred) |
+| B. One resident decoded image at head, advanced per admission; old `as_of` rebuilt on demand | `Mind` owns `Image { docs by (kind,key) -> Held+ordinal, closed_by map, receipts }`; admission appends the batch's Held and closure entries after the redb commit; head reads clone nothing | Rebuild from envelopes for `as_of < head` only (today's `Reader::at` path, kept for the rare cursor) | Full decoded mind: 10-35 MB at 11k, 1-3 GB at 1M | per-read `from_image`, `AdmissionIndex::build`, double `head` receipt decode for head reads; admission's own `from_image` (`admission.rs:127`); the linear `held`/`raw_envelope` scans become map lookups | Two code paths (head fast, old slow) must agree; rebuild cost for old cursors stays O(N) |
+| C. Resident image tagged with ordinals; `as_of` is a filter (Datomic-style) | Same as B, but every Held and every closure entry keeps its ordinal; an `as_of` read filters by ordinal tag | The same image: visible iff `ordinal <= as_of`; status iff `closed_by.ordinal <= as_of` | Same as B plus an ordinal per document and closure | Everything B deletes, plus the second (rebuild) path: one image, one read path, `Reader::at(as_of)` becomes a view parameter | A filtered scan for `query` is still O(N) per page unless ordered by ordinal; correctness relies on the prefix property (documents written once) |
+| D. Persistent-map versions (imbl/rpds), one root retained per live cursor | Writer publishes a new root per admission (e.g. via `ArcSwap`); cursors hold a root | The retained root | Head image plus changed paths per retained version | Same as B; plus the rebuild path | Constant-factor read cost (OrdMap about 2-3x std); roots need an expiry; new dependency; solves a problem C does not have if history is a prefix (inferred) |
+| E. Persisted projection with checkpoint (event-sourcing style) | A second store holding the image and a checkpoint ordinal, advanced by an applier | Replay from checkpoint or rebuild | Same resident image plus disk | Nothing; adds a second owner of head | Checkpoint can disagree with the file; rebuild-on-version-change machinery (Marten/Fowler) for no gain over rebuilding at open (inferred) |
+| F. General IVM (DBSP/differential) | An IVM runtime maintains `status` | Compaction frontier | Arrangements | The status scan | Machinery far exceeding a two-level subject lookup; unverified cost bounds here |
+
+What B/C replace in the status derivation (inferred): `closing_resolution` / `in_force` scans (`docs.rs:171-206`) with a `closed_by` map written at admission time. The admission rule that already computes "which resolution closes this subject" for A8/Q19 (`docs.rs:165-183` is shared by admission and views, per its doc comment at `docs.rs:202-203`) would be the place that writes the entry, keeping one derivation.
+
+## 7. Open facts the fork will want measured (not measured here)
+
+- Where the 200 ms goes: envelope decode (`docs.rs:44-54`), receipt decode twice (`receipt.rs:204`, `query.rs:325`), or the status scan in `rows()` (`query.rs:431-439` over `docs.rs:171-183`). The brief's linear-in-mind figure does not distinguish these.
+- Decoded size per document at 11k; the memory table above is an estimate.
+- Whether `Selection` evaluation (`select`, `query.rs:486`) is itself linear per page, which an image would not fix.
+- Cursor lifetime in practice (key is per process, `mind.rs:190`) and how many outstanding `as_of` values older than head exist at once; this decides whether "rebuild on demand for old as_of only" is rare enough.
+- Whether the per-commit `refresh()` (`mind.rs:274-278`) is worth keeping once an image is advanced from the committed batch; it also serves the lost-CAS diff (`receipt.rs:272-283`).
+
+## Sources
+
+- Huginn: `F:\Projects\Huginn\crates\huginn-mind\src\{query,docs,mind,receipt,admission,index}.rs`; `F:\Projects\CultLib\packages\cultcache-rs\src\lib.rs` (lines 1975, 2349-2350).
+- redb: https://github.com/cberner/redb/blob/master/docs/design.md ; https://www.redb.org/post/2023/06/16/1-0-stable-release/ ; https://docs.rs/redb/latest/redb/struct.ReadTransaction.html
+- LMDB: https://raw.githubusercontent.com/LMDB/lmdb/mdb.master/libraries/liblmdb/lmdb.h
+- PostgreSQL: https://www.postgresql.org/docs/current/transaction-iso.html ; https://www.postgresql.org/docs/current/functions-admin.html ; https://www.postgresql.org/docs/current/routine-vacuuming.html
+- Datomic: https://docs.datomic.com/whatis/architecture.html ; https://docs.datomic.com/operation/capacity.html ; https://docs.datomic.com/reference/filters.html
+- imbl: https://docs.rs/imbl/latest/imbl/ ; rpds: https://github.com/orium/rpds ; arc-swap: https://docs.rs/arc-swap/latest/arc_swap/
+- Marten: https://martendb.io/events/projections/async-daemon.html ; Akka Projection: https://doc.akka.io/libraries/akka-projection/current/overview.html ; Fowler: https://martinfowler.com/eaaDev/EventSourcing.html
+- DBSP: https://arxiv.org/abs/2203.16684 ; differential dataflow: https://timelydataflow.github.io/differential-dataflow/chapter_5/chapter_5_3.html
+- Not reached: Datomic "Time in Datomic" and indexing pages (404), Kurrent projections page (404/redirect), LMDB docs HTML (TLS error), DBSP PDF (binary).