Huginn/docs/research/huginn-materialized-image.md

Prior art: a materialized read image advanced by a single writer, with snapshots for cursors

Serves follow-up eureka-substrate:follow_up:huginn-materialized-image under operator ruling eureka-substrate:ruling:readback-gate-and-materialized-image. Campaign eureka-substrate, session self-2026-10-25-eureka. Written by Eyes (eyes-materialized-image) on 2026-10-26. Facts carry evidence pointers; inference is marked "(inferred)". No recommendation is made in the body; the last section lists what each option would delete and cost, for the design fork.

Method note: web sources were read through a summarizing fetch tool, not verbatim. Claims about third-party systems are as summarized from the cited page; where a page did not say something, that is stated. Nothing was measured in Huginn; the 200 ms / 11k figure is the brief's.

1. What Huginn does today (code facts)

Paths are under F:\Projects\Huginn\crates\huginn-mind\src\ unless noted.

Storage and the in-memory image

• Mind::open takes the redb store's exclusive per-path lock for the Mind's lifetime (mind.rs:130-147; refusal MindAlreadyOwned). One process, one writer.
• open_checked pulls all envelopes, runs the gates, then attach (mind.rs:183-191). Mind.image is a Vec<CultCacheEnvelope> (mind.rs:102), the raw encoded envelopes (msgpack payload bytes), in identity order. Mind::envelopes() hands it out (mind.rs:214).
• After every commit, refresh() runs cache.pull_all_backing_stores() then image = snapshot(&cache) (mind.rs:274-278), called from receipt::commit (receipt.rs:246-286, the call at :271). The redb backing store's pull_all opens a redb begin_read() and reads the whole table (F:\Projects\CultLib\packages\cultcache-rs\src\lib.rs:2349-2350). So the write path is also linear in the mind: every admitted batch re-reads every envelope from redb.
• Commit is a compare-and-swap batch: strong reads as expectation, writes plus the receipt as replacements (receipt.rs:253-270). Receipts are ordinary envelopes of type huginn.mind_commit_receipt.v2 (receipt.rs:28, 97) with a dense ordinal 1..=N; head() decodes every receipt, sorts ordinals, and refuses unless they are exactly 1..=N (receipt.rs:204-213).

The read path (query.rs)

• Mind::view (query.rs:456-460): receipt::head (decodes all receipts), then Reader::at(self, head), then a find and view_of.
• Mind::query (query.rs:469-487): validates, takes as_of from the cursor (Cursor::parse, refused if beyond head, :473-484) or head, then Reader::at(self, as_of)?.rows()?, then select. Members/ranked paths build a Reader the same way (:532-533, :589-590).
• Reader::at (query.rs:380-391): Docs::from_image(mind.envelopes()) decodes every envelope of a known kind into Held { kind, key, document } (docs.rs:27-54); AdmissionIndex::build calls head again and decodes every receipt again, building a BTreeMap<(type, key), AdmissionFacts> over all writes, refusing if two receipts write one identity (query.rs:315-340); then it keeps Held whose receipt ordinal <= as_of and drops the rest. "Built once per call and thrown away with it" (query.rs:365-367).
• held() is a linear iter().find (query.rs:393-395); Docs::find/in_image likewise (docs.rs:65-75); Mind::raw_envelope is a linear scan over the envelope Vec (mind.rs:261-263).
• Status is derived per document: view_of calls Docs::closing_resolution (query.rs:397-405), which scans all resolutions and, per candidate, calls in_force(Resolution, key), which itself is closing_resolution_unless(...).is_none() (docs.rs:165-183, 204-206). rows() does this for every held document plus a base() walk (query.rs:420-439). Cost shape (inferred from the code, not profiled): decode O(N), receipt decode O(R) twice, rows O(N x Rz) where Rz is the number of resolutions, recursion depth bounded by the Q19 cap at two (query.rs:414-419). The 200 ms may be dominated by the status scan rather than decode; that is a hypothesis to profile (inferred).
• Admission builds its own Docs::from_image(self.envelopes()) per batch (admission.rs:127; also index.rs:128), so admission pays the full decode as well.

Invariants the image must keep (from doc comments)

• A snapshot at as_of is the documents whose writing receipt has ordinal <= as_of, with every derivation (in force, closing resolution, base, citations) computed over that image alone (query.rs:374-379).
• Every held document must have its receipt at any as_of; an orphan or a double write refuses at any snapshot (query.rs:377-379, 351).
• writes only: a document later cited as a strong read keeps the receipt that wrote it (query.rs:311-314). A10: one write per identity, which means documents are written once and never overwritten in place; the only "change" is a later resolution document (inferred from query.rs:311-314; a persona put is the stated exception, receipt.rs:259-260).
• Cursors carry as_of and are minted with a per-process random key (mind.rs:190, CursorKey::random()), so a cursor does not outlive the process (inferred: a restart makes old cursors unverifiable).

Consequence worth stating plainly (inferred): because identities are written once and receipts are dense and monotone, the set of documents visible at as_of is a prefix of the receipt chain. A document's own introduction ordinal is immutable. Only derived status (closed by a resolution) varies with as_of, and it varies monotonically: a resolution with ordinal r closes its subject for every as_of >= r. This is the structure the prior art below is being asked to exploit.

2. MVCC read views

PostgreSQL. Under Repeatable Read the snapshot is taken at the first non-transaction-control statement and every later statement in the transaction sees the same committed data (PostgreSQL docs, "Transaction Isolation", https://www.postgresql.org/docs/current/transaction-iso.html). A snapshot can be exported with pg_export_snapshot() and imported by another session via SET TRANSACTION SNAPSHOT; the exporting transaction must stay open for the id to remain importable (https://www.postgresql.org/docs/current/functions-admin.html). Failure mode: an old row version must be kept while any transaction may still see it, so open or idle-in-transaction sessions, prepared transactions and replication slots hold back cleanup and cause bloat (https://www.postgresql.org/docs/current/routine-vacuuming.html; the xmin-horizon mechanism is named in the page's diagnostic queries and "outside the page" note, i.e. partly the fetch tool's general knowledge).

• Relevance: PostgreSQL's cursor-like long snapshot is a held transaction. The cost of serving an old snapshot is paid in retained versions, by the writer's storage, for as long as the snapshot is held. A Huginn cursor has no connection and may be resumed minutes later; holding an engine transaction per outstanding cursor is the Postgres exported-snapshot model and carries its bloat risk (inferred).

LMDB. Single writer, readers take no locks; read transactions pin old page versions, so "the database can grow quickly" under long-lived readers, and stale readers from aborted processes cause the same; docs advise avoiding long-lived transactions, checking stale readers with mdb_reader_check, and note the reader table has a fixed slot count (default 126) (lmdb.h comments, https://raw.githubusercontent.com/LMDB/lmdb/mdb.master/libraries/liblmdb/lmdb.h). MDB_NOTLS lets a read transaction cross threads.

• Relevance: same cost class as redb below.

redb (already in Huginn). MVCC on a copy-on-write B+tree. A read transaction takes a private copy of the root and registers itself, so pages that root references are not freed while it runs; freed pages are queued and reused only after readers that could reference them finish (redb docs/design.md, https://github.com/cberner/redb/blob/master/docs/design.md; also the 1.0 post, https://www.redb.org/post/2023/06/16/1-0-stable-release/, which says readers and writers do not block each other). ReadTransaction is Send + Sync, closes when it and all objects referencing it drop (https://docs.rs/redb/latest/redb/struct.ReadTransaction.html). Savepoints register as read transactions to preserve a snapshot and cost allocator-state copy of about 64 kB per GB of values at 4 kB pages; persistent savepoints survive restart, must be deallocated explicitly, and keep their pages from reclamation (design.md; the space-growth statement is the fetch tool's inference).

• What a redb read transaction gives for free: a consistent, lock-free view of the whole table as of the last commit before it began, for the lifetime of the handle, with no copy. Huginn's pull_all already uses it, but only to copy every envelope into a Vec and drop the transaction (cultcache-rs lib.rs:2349-2350). Nothing in the pages read gives a way to open a read transaction at an arbitrary older commit; older views exist only as held read transactions or savepoints (the pages describe no "open at txid" API; absence in the fetched text, not proof).
• What it does not give: decoded documents, a status index, or an ordinal filter. It would remove the in-process image copy and its refresh, not the decode and derivation work (inferred).
• Failure mode for cursors: one pinned transaction per live cursor-snapshot prevents page reuse, so the redb file grows while cursors are held, and there is no natural expiry for a cursor that an agent abandons (inferred from the reuse rule).
• Note a structural mismatch (inferred): redb's snapshots are per commit of the engine; Huginn's as_of is per receipt. Because each admitted batch is one compare-and-swap commit with its receipt in the same write transaction (receipt.rs:268-270), engine commits and receipt ordinals correspond 1:1, but redb does not expose the mapping and cannot be asked for "the view as of ordinal 12,345" after the fact.

3. Immutable database values and persistent maps

Datomic. Information accumulates and is never removed; databases are immutable values, so compute instances need no coordination for query and caches hold only immutable data (https://docs.datomic.com/whatis/architecture.html). Queries merge persistent index trees with an in-memory representation of recent change; new index trees are built occasionally by background indexing jobs (same page; the pages fetched did not use the terms novelty/memory index there). The operations page defines the memory index as transaction data logged but not yet in a disk index, triggers an indexing job at memory-index-threshold (default 32m), lets the transactor continue until memory-index-max (default 512m), and applies radical backpressure if novelty exceeds max before indexing finishes (https://docs.datomic.com/operation/capacity.html).

• as-of is a filter over the same indexes, not a stored copy; filtering "is not free"; entity-plus-time queries jump via EAVT then drop newer datoms; time-only questions would need a scan of the whole database (https://docs.datomic.com/reference/filters.html). The current db from a connection does no filtering, so "now" pays no history penalty.
• Relevance (inferred): this is the closest match to Huginn's data model: append-only, writes never overwritten, and as_of is a filter on a monotone position (Datomic t, Huginn receipt ordinal). The Datomic answer to "older snapshot" is not to keep versions; it is to keep every datom with its t and filter. Applied here: keep each document once, tagged with its introduction ordinal, and keep each resolution tagged with its ordinal; an as_of read filters by tag. A Datomic filter pays per touched datom, not per mind, because it goes through an index.
• Failure modes: the novelty/indexing split exists because the in-memory representation must be bounded; backpressure is the cost of the writer outrunning the indexer. Huginn's single-writer, low-rate admission is far from that regime (inferred).

Persistent (structural-sharing) maps in Rust.

• imbl: clone is a refcount increment on the root; nodes are copied on write only when shared; HashMap is "almost neck and neck" with std, OrdMap about 2-3x slower than std on basic operations, Vector loses to Vec on memory and indexing; sole-owner in-place mutation is about an order of magnitude faster than immutable operations; default pointer is Arc (thread safe), Rc and triomphe options exist (https://docs.rs/imbl/latest/imbl/).
• rpds: Vector, HashTrieMap, RedBlackTreeMap; thread safety is opt-in via _sync constructors using Arc, which is "significantly slower to clone and drop than Rc"; mutable methods are faster when old versions are not needed (https://github.com/orium/rpds).
• Relevance (inferred): a persistent map gives one owner advancing a head version per admission and cheap retention of any older version by holding its root. Memory cost of N retained versions is the changed paths, not N copies. Hazards: per-version retention has no expiry unless something drops the root (a cursor table), and OrdMap/trie ops cost a constant factor over a plain map for reads. Because Huginn's history is a prefix chain, each version differs from its predecessor by a few documents, so path-copy cost per admission is small (inferred); but the same prefix property means a version could be reconstructed by filtering rather than retained (see Datomic).
• Handing a version to readers: arc-swap provides an atomically swappable Arc for read-mostly state, one writer publishing, many lock-free readers each keeping the version it loaded (https://docs.rs/arc-swap/latest/arc_swap/). The page notes the crate is niche, and that 1.9 strengthened memory orderings with probable cost; it did not discuss guard hold times.

4. Event-sourced projections with a checkpoint

Marten (PostgreSQL). An async daemon processes events in order; a high-water mark (HWM) advances contiguously and stops at gaps; each projection shard records its own progression; rebuilds via projections --rebuild or RebuildProjectionAsync (https://martendb.io/events/projections/async-daemon.html). Failure modes listed there: non-transactional PostgreSQL sequences burn numbers on rollback, creating gaps that stall the HWM; long-running or idle-in-transaction sessions can stall it; forcing a skip (SkipStaleGapsDespiteLiveTransactionsAfter, AdvanceHighWaterMarkToLatestAsync) can permanently skip events that commit later; slow pages beyond StopAndDrainTimeout get cancelled and reprocessed each shutdown; poison events go to a dead-letter table; migrating an inline projection to async needs downtime.

• Relevance (inferred): the gap hazard comes from multiple concurrent appenders with non-transactional sequence numbers. Huginn has one writer and a dense ordinal checked as 1..=N in head (receipt.rs:204-213), so the HWM and gap machinery have no counterpart; the projection's checkpoint is simply "head ordinal applied".

EventStoreDB / Kurrent, Akka Projection. The Kurrent projections page URL fetched returned 404 on the second path; the Akka overview page says only that an offset per event lets a projection resume after restart, with offsets kept in Cassandra, JDBC, R2DBC or Slick, and did not state delivery guarantees or rebuild procedure. Neither was verified further here. Not evidence for or against.

Fowler, "Event Sourcing". State can be derived by replaying the log from a blank state, to any point; replay is "a slow process, particularly if there are many events"; snapshots of application state speed restarts and can be built in parallel; "Complete Rebuild" discards derived state and replays (https://martinfowler.com/eaaDev/EventSourcing.html).

• Relevance (inferred): Huginn is already its own event log (the receipts) and its own store (the envelopes). The projection here is derived from the same file that holds its checkpoint. On a crash or version change the image rebuilds from the envelopes (what Reader::at does today per read), so "rebuild on version change" is the existing code run once at open instead of per read. A persisted projection (a second store) would add a checkpoint that can disagree with the file (inferred: a second owner of the head ordinal).

5. Incremental view maintenance for derived status

DBSP (Budiu, McSherry, Ryzhyk, Tannen). A language for computation over streams, with an algorithm that incrementalizes arbitrary programs, covering relational queries, aggregation and recursion (https://arxiv.org/abs/2203.16684, abstract only; the PDF fetch returned unreadable binary, so claims about cost bounds were not verified; the fetch tool's general-knowledge recollection is that distinct and aggregation need integrated state that grows with input). Differential dataflow / Materialize arrangements. Arrangements are shared indexed state serving many queries; logical compaction (set_logical_compaction(frontier)) declares that times before the frontier will no longer be queried, letting the trace merge them, after which reads of older times are not guaranteed; holding trace handles prevents compaction (https://timelydataflow.github.io/differential-dataflow/chapter_5/chapter_5_3.html).

• Relevance (inferred): Huginn's "status" is a two-level lookup (resolution subject -> closing resolution, docs.rs:171-183), not a recursive query. A general IVM engine is machinery well beyond that derivation. The transferable lesson is the compaction frontier: an indexed state that serves several times needs an explicit declaration of the oldest time still readable, otherwise it never shrinks. For Huginn the analogue is a cursor TTL. The simple incremental alternative the evidence supports is recompute-per-key-on-write: when a resolution with subject S is admitted, record closed_by[S] = (resolution key, ordinal); view_of is then one map lookup. Status at as_of is closed_by[S] if its ordinal <= as_of (inferred, from the monotone-closure property in section 1; A8 and the Q19 cap admit at most one closing resolution per document, docs.rs:168-170, so no multi-valued history is needed for that case; resolutions that are themselves resolved make in_force(Resolution) depend on the same lookup, depth capped at two).

6. What each option would mean here

Common frame: single owner of the image (the Mind); an admission advances it after the redb commit; Mind::view/query read it. Memory figures are order-of-magnitude (inferred) with decoded document size unknown; assume 1-3 KB decoded per document: 11k docs about 10-35 MB; 1M docs about 1-3 GB. The current per-read build allocates and frees that on every call; the options differ in how much stays resident.

Option	Owner / advance	Old as_of served by	Resident memory	Deletes from Huginn	Main risk
A. Hold redb read transactions as the snapshot (per cursor)	Image stays derived per read from a held transaction; no resident image	Keep the transaction open per live cursor	Engine pages for pinned versions, grows with write churn while cursors live	refresh() re-pull and the image Vec could go (inferred); not decode or status scan	Unbounded cursor lifetime pins file growth (LMDB/redb rule); does not fix the 200 ms, which is decode plus derivation (inferred)
B. One resident decoded image at head, advanced per admission; old as_of rebuilt on demand	Mind owns Image { docs by (kind,key) -> Held+ordinal, closed_by map, receipts }; admission appends the batch's Held and closure entries after the redb commit; head reads clone nothing	Rebuild from envelopes for as_of < head only (today's Reader::at path, kept for the rare cursor)	Full decoded mind: 10-35 MB at 11k, 1-3 GB at 1M	per-read from_image, AdmissionIndex::build, double head receipt decode for head reads; admission's own from_image (admission.rs:127); the linear held/raw_envelope scans become map lookups	Two code paths (head fast, old slow) must agree; rebuild cost for old cursors stays O(N)
C. Resident image tagged with ordinals; as_of is a filter (Datomic-style)	Same as B, but every Held and every closure entry keeps its ordinal; an as_of read filters by ordinal tag	The same image: visible iff ordinal <= as_of; status iff closed_by.ordinal <= as_of	Same as B plus an ordinal per document and closure	Everything B deletes, plus the second (rebuild) path: one image, one read path, Reader::at(as_of) becomes a view parameter	A filtered scan for query is still O(N) per page unless ordered by ordinal; correctness relies on the prefix property (documents written once)
D. Persistent-map versions (imbl/rpds), one root retained per live cursor	Writer publishes a new root per admission (e.g. via ArcSwap); cursors hold a root	The retained root	Head image plus changed paths per retained version	Same as B; plus the rebuild path	Constant-factor read cost (OrdMap about 2-3x std); roots need an expiry; new dependency; solves a problem C does not have if history is a prefix (inferred)
E. Persisted projection with checkpoint (event-sourcing style)	A second store holding the image and a checkpoint ordinal, advanced by an applier	Replay from checkpoint or rebuild	Same resident image plus disk	Nothing; adds a second owner of head	Checkpoint can disagree with the file; rebuild-on-version-change machinery (Marten/Fowler) for no gain over rebuilding at open (inferred)
F. General IVM (DBSP/differential)	An IVM runtime maintains status	Compaction frontier	Arrangements	The status scan	Machinery far exceeding a two-level subject lookup; unverified cost bounds here

What B/C replace in the status derivation (inferred): closing_resolution / in_force scans (docs.rs:171-206) with a closed_by map written at admission time. The admission rule that already computes "which resolution closes this subject" for A8/Q19 (docs.rs:165-183 is shared by admission and views, per its doc comment at docs.rs:202-203) would be the place that writes the entry, keeping one derivation.

7. Open facts the fork will want measured (not measured here)

• Where the 200 ms goes: envelope decode (docs.rs:44-54), receipt decode twice (receipt.rs:204, query.rs:325), or the status scan in rows() (query.rs:431-439 over docs.rs:171-183). The brief's linear-in-mind figure does not distinguish these.
• Decoded size per document at 11k; the memory table above is an estimate.
• Whether Selection evaluation (select, query.rs:486) is itself linear per page, which an image would not fix.
• Cursor lifetime in practice (key is per process, mind.rs:190) and how many outstanding as_of values older than head exist at once; this decides whether "rebuild on demand for old as_of only" is rare enough.
• Whether the per-commit refresh() (mind.rs:274-278) is worth keeping once an image is advanced from the committed batch; it also serves the lost-CAS diff (receipt.rs:272-283).

Sources

• Huginn: F:\Projects\Huginn\crates\huginn-mind\src\{query,docs,mind,receipt,admission,index}.rs; F:\Projects\CultLib\packages\cultcache-rs\src\lib.rs (lines 1975, 2349-2350).
• redb: https://github.com/cberner/redb/blob/master/docs/design.md ; https://www.redb.org/post/2023/06/16/1-0-stable-release/ ; https://docs.rs/redb/latest/redb/struct.ReadTransaction.html
• LMDB: https://raw.githubusercontent.com/LMDB/lmdb/mdb.master/libraries/liblmdb/lmdb.h
• PostgreSQL: https://www.postgresql.org/docs/current/transaction-iso.html ; https://www.postgresql.org/docs/current/functions-admin.html ; https://www.postgresql.org/docs/current/routine-vacuuming.html
• Datomic: https://docs.datomic.com/whatis/architecture.html ; https://docs.datomic.com/operation/capacity.html ; https://docs.datomic.com/reference/filters.html
• imbl: https://docs.rs/imbl/latest/imbl/ ; rpds: https://github.com/orium/rpds ; arc-swap: https://docs.rs/arc-swap/latest/arc_swap/
• Marten: https://martendb.io/events/projections/async-daemon.html ; Akka Projection: https://doc.akka.io/libraries/akka-projection/current/overview.html ; Fowler: https://martinfowler.com/eaaDev/EventSourcing.html
• DBSP: https://arxiv.org/abs/2203.16684 ; differential dataflow: https://timelydataflow.github.io/differential-dataflow/chapter_5/chapter_5_3.html
• Not reached: Datomic "Time in Datomic" and indexing pages (404), Kurrent projections page (404/redirect), LMDB docs HTML (TLS error), DBSP PDF (binary).