From 2364979f0d2bb5a29f68f06ed2fac1cd998199c5 Mon Sep 17 00:00:00 2001 From: Heimdall Date: Fri, 6 Nov 2026 17:24:39 +0000 Subject: [PATCH] Rule the open points from the CultLib consent-records review Scopeless replies are stored and excluded; tombstone is HMAC-SHA-256 with a member-side salt; representation, training-rule scope match, scope partition and MemberTerms ownership fixed. Docs only. Co-Authored-By: Claude Sonnet 5.5 --- docs/answer-collector-consent.md | 62 ++++++++++++++++++++++++++++---- 1 file changed, 55 insertions(+), 7 deletions(-) diff --git a/docs/answer-collector-consent.md b/docs/answer-collector-consent.md index 129f279..fe0a321 100644 --- a/docs/answer-collector-consent.md +++ b/docs/answer-collector-consent.md @@ -50,7 +50,10 @@ Key: `(experiment_id, member_id, notice_version)`. Written once, never updated. Joining Heimdall is gated by terms. Acceptance is the terms version recorded by the Matrix `m.login.terms` stage at registration: member field `terms_version` -(string) and `terms_accepted_at` (UTC timestamp). This applies to newcomers +(string) and `terms_accepted_at` (UTC timestamp). These fields belong to the +Heimdall member record. A typed `MemberTerms` record keyed by `member_id` is the +shape of that pair in a typed library; it is not a second store, and the +collector does not hold it. This applies to newcomers only. Members who joined before the terms stage existed, and Minds, are not re-asked and have no `terms_version`; the collector does not read it and does not gate on it. The notice above is the only consent the collector itself checks. @@ -68,6 +71,7 @@ One record per answer. Fields: | `scope` | `show` \| `study` \| `train` | copied from that notice version at answer time. Never edited afterwards. | | `text` | string, or absent | the answer as given. Absent once exited. | | `exited` | boolean, default false | one-way: set by redaction, never cleared. A new answer is a new record. | +| `text_hash` | 32 bytes, or absent | the tombstone hash of `text` (see "Tombstone"). Present only once exited, and only when the collector held `text` before the redaction. | | `received_at` | UTC timestamp | `origin_server_ts` of the reply. | | `event_id` | Matrix event id | the reply; the receipt. | @@ -75,9 +79,25 @@ Key: `(experiment_id, sample_id, member_id)`. A member's later answer to the same key replaces the earlier one (latest `received_at` wins); the earlier text is not retained. -A record whose `notice_version` or `scope` is missing, unparseable or has no -matching acceptance record is not collected. It is excluded from every use, and -from training in particular. +A reply whose `notice_version` or `scope` is missing, unparseable or has no +matching acceptance record is stored, with those fields absent. It is excluded +from every use: no tally, no display, and from training in particular. It +writes no acceptance record. The reply is kept so the member can still exit it +and the collector can show what it refused, but no scope was declared, so none +is inferred. The notice version of a reply is the one pinned at the reply's +`origin_server_ts`; if the collector cannot establish what was pinned then, the +reply has no notice version. + +### Representation + +- Timestamps are Unix milliseconds in UTC, stored as a signed 64-bit integer, + the unit of Matrix `origin_server_ts`. Every "UTC timestamp" above means this. +- `member_id`, `experiment_id`, `event_id` and `terms_version` are non-empty + strings; `member_id` is opaque and `event_id` is the Matrix event id verbatim. + `notice_version` and the `` of `sample_id` are positive integers + (unsigned 32-bit). +- A composite key is its parts joined with U+001F. A part that is empty or + contains U+001F is refused, not escaped. ## Versions @@ -92,13 +112,39 @@ A record is training or evaluation data only if all hold: 1. `scope = train`, copied from a notice version that declared `train`; 2. an acceptance record exists for `(experiment_id, member_id, notice_version)` - with `accepted_at <= received_at`, i.e. the author had accepted that notice + of the answer, its own `scope` is `train` and equals the answer's `scope`, + and `accepted_at <= received_at`, i.e. the author had accepted that notice version when they answered; 3. `exited = false` at export time. No recorded scope, or no recorded acceptance, means excluded. Records wider than -`show` are kept apart from `show` records: separate store or partition, so a read -for one level cannot return another. +`show` are kept apart from `show` records: one store per scope (`show`, `study`, +`train`) plus a store for scopeless replies that no read path serves. A record +lives in the store its `scope` names, so a read for one level cannot return +another. A typed-record library need not model the partition; the collector's +store layout owns it. + +## Tombstone + +- Function: HMAC-SHA-256 (RFC 2104 with SHA-256, FIPS 180-4), output the raw 32 + bytes stored in `text_hash`. No truncation. +- Message: the ASCII string `heimdall.tombstone.v1`, U+001F, `event_id` + (UTF-8), U+001F, `text` (UTF-8, as received, no normalisation). The `event_id` + makes the hash specific to the reply, so equal texts hash differently. +- Key (the salt): 32 bytes from a cryptographically secure random source, + generated once per member by the member-side service, which is the service that + already maps Matrix sender to `member_id`. It is held there, beside the member + record, and is never given to the collector, a researcher, an export or a log. + The collector cannot compute or check a tombstone hash on its own. +- Redaction: when the collector sees a redaction of a reply whose text it holds, + it sends `event_id` and `text` to the member-side service, which returns the + hash; the collector writes `text_hash`, sets `exited`, and erases `text`. If the + service does not answer, `text` is kept, the record is not yet exited, and the + run retries. A reply the collector never collected has no text and gets no + tombstone. +- Verification: only the member-side service can check a claimed text against a + tombstone. If a member's salt is destroyed, their tombstones can no longer be + tied to any text, and that is the member's erasure. ## Identity: count once, reward without exposure @@ -137,6 +183,8 @@ not stored under a placeholder. and a salted hash of `text`. Hash only, never text, and no `member_id` beyond what the record's key already holds. The count and the exit are provable without the content. The record is marked `exited`; `text` is erased. + The tombstone is the exited `Answer` record itself, with `text_hash` set; it is + not a separate record. - **Edits.** An edit is a new event, not the receipt. Only the original event counts; edits are ignored, so what was accepted is what was received. To change an answer, redact and reply again.