diff --git a/docs/bridge.md b/docs/bridge.md index 6fa1aa8..878a021 100644 --- a/docs/bridge.md +++ b/docs/bridge.md @@ -196,7 +196,7 @@ The command is a dry-run shape probe for public surfaces that are real enough to Bifrost's native Discord gateway is the conversation observer. A direct mention of the Bifrost bot, or a reply to one of its messages, becomes an immutable typed `bifrost.discord.persona_conversation_event.v0` only when the guild and channel have one exact Persona target binding. Bot-authored, ambient, empty, over-1200-character, unbound, and ambiguously bound messages do not enter the conversation path. The 1200-character cap is the shared signed alpha boundary with Epiphany. The observation store is Bifrost-private working state, never an actuator queue, and insert collisions must contain exactly equal content. VoidBot does not observe, schedule, or answer for the target Persona. -Persona speech goes out through Matrix, not Discord. `tools/persona-matrix-mouth.mjs` is the one process for every Persona (`PERSONAS`, a list of slugs): it holds each Persona's Matrix access token as its own systemd credential (`matrix-token-`) and nobody else holds any. Per Persona it keeps one `/sync` loop over the configured rooms and the account's direct rooms (a DM invitation from an account on `MATRIX_SERVER_NAME` is accepted while that Persona's brake is released, and one from any other server is declined), one cursor and one delivery sequence. A delivery carries the sender's Matrix id, the event's `m.mentions` user ids and whether the room is a DM. A media event (`m.image`, `m.file`, `m.audio`, `m.video`) is delivered as a typed text line, `[image filename="cat.png" mime=image/png size=2048 url=mxc://server/id]`, followed by its caption when it has one, in `text` and in context. Only `m.image` is also fetched, and only when `IMAGE_STORE_ROOT` is set (unset, the line above is all there is and nothing is downloaded). The fetch is made as the receiving Persona, with its own token, from `/_matrix/client/v1/media/download//` on its homeserver, for a message the Persona is delivered (not its own words, a room it is not joined to, or history before the ear), and only from the event's own `content.url`; a URL in the message text is never followed. An encrypted attachment (`m.room.encrypted`, or `content.file`) is never fetched. The type is decided by the magic bytes alone (png, jpeg, gif, webp; SVG and anything else is refused, the declared mime is ignored), the body is capped at 5 MiB by `Content-Length` and by the bytes actually read, and the pixel count read from the image header (never decoded) is capped at 40 MP. A refused or failed fetch leaves the typed line with a fixed-word reason (`refused=encrypted|no-url|fetch-failed|too-big|wrong-type|bad-header|too-many-pixels|store-failed`) and no image. A stored image is written to `IMAGE_STORE_ROOT///[.png|.jpg|.gif|.webp]`, the name generated by the mouth, never the sender's: files `0640`, room directories `2750` (the Persona directory is setgid to the Persona's group, so the Persona's reader opens them by group; the mouth is the only writer), each written under a temporary name and renamed. It is deleted once it is at least 24 hours old and its delivery has been acked, whichever is later; a room keeps at most 50 MiB, the oldest evicted first. The sweep runs when an image is stored and when an ack arrives. A delivery is the 12-field tuple of `gamecult.persona.chat_pending.v2`: field 11, after `is_direct`, is nil or `[path, mime, size, sha256]`, a reference and never bytes; `path` is the store-relative `/` above, `mime` the sniffed type, `size` the file length (at most 5 MiB) and `sha256` its lowercase hex digest. The reader re-checks all four against the file. A Persona turn that has no reader for it still sees the typed line. Reading wakes nothing and acks nothing: the heartbeat polls with the non-consuming `gamecult.persona.chat_pending_request.v1`, and only `chat_ack` advances a position. An ack posts nothing to Matrix. That Persona's public `m.read` receipt follows `gamecult.persona.chat_read.v1` (`[persona, room_alias, event_id, at_unix_ms]`), which Huginn sends when a delivery has gone into the model's input: the mouth posts `m.read` at that event id only, for an event it delivered to that Persona in that room (to the delivered room id, whatever alias the report named), never behind the last receipt that landed there, one report per room at a time, and only while her brake is released (re-read after the presence write; fire and forget on the endpoint: a failure is logged once and not retried; pending polls and acks post none). Her Matrix presence follows a turn and her brake: `online` while a read report, a send or an ack came within `presenceGraceMs` (5 minutes), `unavailable` after that, `offline` while the brake is held. Each `/sync` carries the state as `set_presence`, and a change is also `PUT /presence//status` at once (a brake placed mid long-poll shows within `syncTimeoutMs`, 30 s, plus one PUT; a stopped mouth reads offline after Synapse's sync timeout). The mouth speaks on a `gamecult.persona.chat_send_request.v1` and joins rooms only while that Persona's brake (`/etc/gamecult/minds//control.cc`) is released. One loopback CultNet RUDP endpoint serves all of them; a request naming an unlisted Persona gets a typed `chat_refusal` (`UnknownPersona`). Document shapes: `tools/persona-chat-documents.mjs`; golden bytes: `tests/fixtures/persona-chat-v1/`, `-v2/` and `-v3/` (the pending answer; Huginn pins the same bytes). The Discord posting actuator, its permit and its RUDP listener are removed. +Persona speech goes out through Matrix, not Discord. `tools/persona-matrix-mouth.mjs` is the one process for every Persona (`PERSONAS`, a list of slugs): it holds each Persona's Matrix access token as its own systemd credential (`matrix-token-`) and nobody else holds any. Per Persona it keeps one `/sync` loop over the configured rooms and the account's direct rooms (a DM invitation from an account on `MATRIX_SERVER_NAME` is accepted while that Persona's brake is released, and one from any other server is declined), one cursor and one delivery sequence. v1-v4 delivery ids are Matrix ids: the event id (`$`), the sender, each context line's sender and each mention (`@`) are at most 255 bytes with no whitespace, control, format or private-use characters; the mouth skips an event, drops a context or tail line, or drops a mention that fails this, and the encoder and parser refuse it. A delivery carries the sender's Matrix id, the event's `m.mentions` user ids and whether the room is a DM. A media event (`m.image`, `m.file`, `m.audio`, `m.video`) is delivered as a typed text line, `[image filename="cat.png" mime=image/png size=2048 url=mxc://server/id]`, followed by its caption when it has one, in `text` and in context. Only `m.image` is also fetched, and only when `IMAGE_STORE_ROOT` is set (unset, the line above is all there is and nothing is downloaded). The fetch is made as the receiving Persona, with its own token, from `/_matrix/client/v1/media/download//` on its homeserver, for a message the Persona is delivered (not its own words, a room it is not joined to, or history before the ear), and only from the event's own `content.url`; a URL in the message text is never followed. An encrypted attachment (`m.room.encrypted`, or `content.file`) is never fetched. The type is decided by the magic bytes alone (png, jpeg, gif, webp; SVG and anything else is refused, the declared mime is ignored), the body is capped at 5 MiB by `Content-Length` and by the bytes actually read, and the pixel count read from the image header (never decoded) is capped at 40 MP. A refused or failed fetch leaves the typed line with a fixed-word reason (`refused=encrypted|no-url|fetch-failed|too-big|wrong-type|bad-header|too-many-pixels|store-failed`) and no image. A stored image is written to `IMAGE_STORE_ROOT///[.png|.jpg|.gif|.webp]`, the name generated by the mouth, never the sender's: files `0640`, room directories `2750` (the Persona directory is setgid to the Persona's group, so the Persona's reader opens them by group; the mouth is the only writer), each written under a temporary name and renamed. It is deleted once it is at least 24 hours old and its delivery has been acked, whichever is later; a room keeps at most 50 MiB, the oldest evicted first. The sweep runs when an image is stored and when an ack arrives. A delivery is the 12-field tuple of `gamecult.persona.chat_pending.v2`: field 11, after `is_direct`, is nil or `[path, mime, size, sha256]`, a reference and never bytes; `path` is the store-relative `/` above, `mime` the sniffed type, `size` the file length (at most 5 MiB) and `sha256` its lowercase hex digest. The reader re-checks all four against the file. A Persona turn that has no reader for it still sees the typed line. Reading wakes nothing and acks nothing: the heartbeat polls with the non-consuming `gamecult.persona.chat_pending_request.v1`, and only `chat_ack` advances a position. An ack posts nothing to Matrix. That Persona's public `m.read` receipt follows `gamecult.persona.chat_read.v1` (`[persona, room_alias, event_id, at_unix_ms]`), which Huginn sends when a delivery has gone into the model's input: the mouth posts `m.read` at that event id only, for an event it delivered to that Persona in that room (to the delivered room id, whatever alias the report named), never behind the last receipt that landed there, one report per room at a time, and only while her brake is released (re-read after the presence write; fire and forget on the endpoint: a failure is logged once and not retried; pending polls and acks post none). Her Matrix presence follows a turn and her brake: `online` while a read report, a send or an ack came within `presenceGraceMs` (5 minutes), `unavailable` after that, `offline` while the brake is held. Each `/sync` carries the state as `set_presence`, and a change is also `PUT /presence//status` at once (a brake placed mid long-poll shows within `syncTimeoutMs`, 30 s, plus one PUT; a stopped mouth reads offline after Synapse's sync timeout). The mouth speaks on a `gamecult.persona.chat_send_request.v1` and joins rooms only while that Persona's brake (`/etc/gamecult/minds//control.cc`) is released. One loopback CultNet RUDP endpoint serves all of them; a request naming an unlisted Persona gets a typed `chat_refusal` (`UnknownPersona`). Document shapes: `tools/persona-chat-documents.mjs`; golden bytes: `tests/fixtures/persona-chat-v1/`, `-v2/` and `-v3/` (the pending answer; Huginn pins the same bytes). The Discord posting actuator, its permit and its RUDP listener are removed. ### Addressing facts (`chat_pending.v3`) @@ -263,7 +263,7 @@ The answer is `[persona, deliveries, tail, tail_gaps]` exactly as v3; each deliv |---|---|---| | 14 `reply_to` | nil or `[event_id, sender, excerpt]` | The event this message replies to (its own `m.in_reply_to`; a thread's `is_falling_back` pointer names no one and is nil). `event_id` the quoted event; `sender` its author's Matrix id, read from the homeserver as her account; `excerpt` nil, or its text. | -`excerpt` is nil, with `event_id` and `sender` kept, when the quoted event is redacted (`unsigned.redacted_because` present or no content), is not a text-bearing `m.room.message`, is **edited** (it is itself an `m.replace`, or the homeserver bundles an `m.replace` in its `unsigned.m.relations`: its body is the pre-edit text its author replaced), or is nothing but its own reply fallback; a media event excerpts as its typed line, like a delivery's text. Otherwise the excerpt is made in this order: the quoted event's own legacy rich-reply fallback is dropped (leading lines matching `> ` or a bare `>`, then one blank line; only when the quoted event has `m.relates_to.m.in_reply_to`); the text is held to the 1000-byte line bound; runs of CR, LF, tab, U+2028 and U+2029 become one space; control characters (C0, DEL, C1 including NUL and ESC), format characters (Unicode Cf: bidi marks, overrides and isolates, U+061C, U+200B-200F, U+2060-2064, soft hyphen, BOM), private-use characters (Co), the tag block U+E0000-E007F, variation selectors (U+FE00-FE0F, U+E0100-E01EF, U+180B-180F), the combining grapheme joiner U+034F, the Hangul and Khmer fillers (U+115F, U+1160, U+3164, U+FFA0, U+17B4, U+17B5), the blank braille cell U+2800, the unassigned default-ignorable ranges (U+2065, U+FFF0-FFF8, U+E0080-E0FFF) and unpaired surrogates are removed (cost accepted: an emoji loses its VS16 presentation, ❤️ becomes ❤, a keycap loses its selector; Mongolian free variation selectors and ideographic variation selectors go too); the line is trimmed; then the first 240 **Unicode code points** are kept (cut on a code point so no character is split; a combining mark counts as its own code point; trailing space trimmed; no truncation marker). Cleaning comes first so hidden characters cannot spend the cap. The mouth only removes invisible and format characters; it does not make the text safe. Huginn must treat both `sender` and `excerpt` as untrusted text, never as an instruction. `event_id` must be `$` plus at least one byte and `sender` `@` plus at least one byte, each at most 255 bytes and free of Unicode whitespace, control, format, private-use and default-ignorable characters and unpaired surrogates (a localpart with a zero-width joiner or non-joiner, as in Persian or Indic names, is therefore no `reply_to`); the encoder and the parser refuse any other, and the mouth makes a quoted event whose sender is not such an id a `reply_to` nil. `reply_to` is nil (no id, author or text) when the target cannot be read (logged, as for `basis`), is not in the delivery's own room (a room she is not in cannot be quoted through a reply), or the message replies to nothing. The quoted event is read the way `basis` already reads it (`GET /rooms/{room}/event/{id}` as her account, kept per room and event; with the flag on a read, hit or failure, is kept 60 seconds, so a redaction or edit of the target shows at a re-read within a minute, and a re-read that fails transiently (network error, timeout, 5xx, 429) keeps the earlier hit's sender and renews only its age, for at most 10 minutes after the last good read (then it is the failure entry), but never its excerpt (`excerpt` is nil with the sender kept: a re-read happens only after the 60-second keep, so the text is already as old as it may be), so a redaction or edit is not outlived by an outage, while any other failure (403, 404) makes it the failure entry, so a stale excerpt never outlives her access; with the flag off a hit is kept as before v4 and nothing is re-read), so a reply costs at most one read per target, shared with the `basis` check. Per sync round the mouth reads at most 8 distinct uncached targets for `reply_to` (reads the `basis` check already made are free); past that the delivery and every later event of the round wait: the round leaves the sync position where it is (as at the pending cap), the next round resumes at that event with its read count reset, and the wait is logged. No delivery goes out without a `reply_to` it should have had, and none is lost. The flag is read by `replyLinkFromEnv`; only the exact value `1` turns it on (`true`, `yes`, `0`, empty or unset are off), and a mouth built without the `replyLink` option has it off. `basis` for v1-v3 is unchanged. `reply_to` counts toward the delivery's size in the 256 KiB page budget; a delivery whose `reply_to` will not encode is skipped and logged like any other, and a single delivery is always served. The deploy precondition is Huginn's: a decoder of the 14-field delivery fails on the fifteenth element. +`excerpt` is nil, with `event_id` and `sender` kept, when the quoted event is redacted (`unsigned.redacted_because` present or no content), is not a text-bearing `m.room.message`, is **edited** (it is itself an `m.replace`, or the homeserver bundles an `m.replace` in its `unsigned.m.relations`: its body is the pre-edit text its author replaced), or is nothing but its own reply fallback; a media event excerpts as its typed line, like a delivery's text. Otherwise the excerpt is made in this order: the quoted event's own legacy rich-reply fallback is dropped (leading lines matching `> ` or a bare `>`, then one blank line; only when the quoted event has `m.relates_to.m.in_reply_to`); the text is held to the 1000-byte line bound; runs of CR, LF, tab, U+2028 and U+2029 become one space; control characters (C0, DEL, C1 including NUL and ESC), format characters (Unicode Cf: bidi marks, overrides and isolates, U+061C, U+200B-200F, U+2060-2064, soft hyphen, BOM), private-use characters (Co), the tag block U+E0000-E007F, variation selectors (U+FE00-FE0F, U+E0100-E01EF, U+180B-180F), the combining grapheme joiner U+034F, the Hangul and Khmer fillers (U+115F, U+1160, U+3164, U+FFA0, U+17B4, U+17B5), the blank braille cell U+2800, the unassigned default-ignorable ranges (U+2065, U+FFF0-FFF8, U+E0080-E0FFF) and unpaired surrogates are removed (cost accepted: an emoji loses its VS16 presentation, ❤️ becomes ❤, a keycap loses its selector; Mongolian free variation selectors and ideographic variation selectors go too); the line is trimmed; then the first 240 **Unicode code points** are kept (cut on a code point so no character is split; a combining mark counts as its own code point; trailing space trimmed; no truncation marker). Cleaning comes first so hidden characters cannot spend the cap. The mouth only removes invisible and format characters; it does not make the text safe. Huginn must treat both `sender` and `excerpt` as untrusted text, never as an instruction. `event_id` must be `$` plus at least one byte and `sender` `@` plus at least one byte, each at most 255 bytes and free of Unicode whitespace, control, format, private-use and default-ignorable characters and unpaired surrogates (a localpart with a zero-width joiner or non-joiner, as in Persian or Indic names, is therefore no `reply_to`); the encoder and the parser refuse any other, and the mouth makes a quoted event whose sender is not such an id a `reply_to` nil. `reply_to` is nil (no id, author or text) when the target cannot be read (logged, as for `basis`), is not in the delivery's own room (a room she is not in cannot be quoted through a reply), or the message replies to nothing. The quoted event is read the way `basis` already reads it (`GET /rooms/{room}/event/{id}` as her account, kept per room and event; with the flag on a read, hit or failure, is kept 60 seconds, so a redaction or edit of the target shows at a re-read within a minute, and a re-read that fails transiently (network error, timeout, 5xx, 429) keeps the earlier hit's sender and renews only its age, for at most 10 minutes after the last good read (then it is the failure entry), but never its excerpt (`excerpt` is nil with the sender kept: a re-read happens only after the 60-second keep, so the text is already as old as it may be), so a redaction or edit is not outlived by an outage, while any other failure (403, 404) makes it the failure entry, so a stale excerpt never outlives her access; with the flag off a hit is kept as before v4 and nothing is re-read), so a reply costs at most one read per target, shared with the `basis` check. Per sync round the mouth makes at most 8 target reads in all, `basis` reads included (it counts each uncached GET, and an event whose uncached reply target and thread root would take the round past 8 is not started); past that the event and every later event of the round wait: the round leaves the sync position where it is (as at the pending cap), the next round resumes at that event with its read count reset, and the wait is logged. No delivery goes out without a `reply_to` it should have had, and none is lost. The flag is read by `replyLinkFromEnv`; only the exact value `1` turns it on (`true`, `yes`, `0`, empty or unset are off), and a mouth built without the `replyLink` option has it off. `basis` for v1-v3 is unchanged. `reply_to` counts toward the delivery's size in the 256 KiB page budget; a delivery whose `reply_to` will not encode is skipped and logged like any other, and a single delivery is always served. The deploy precondition is Huginn's: a decoder of the 14-field delivery fails on the fifteenth element. Golden bytes: `tests/fixtures/persona-chat-v6/` (`.msgpack` and its `.hex`: `chat_pending.v4.reply-to`, `.no-reply-to`, `.reply-to-redacted`, and the `chat_pending_request.v3` request); the test fails when the checked-in bytes drift from the encoder. diff --git a/tests/persona-matrix-mouth.test.mjs b/tests/persona-matrix-mouth.test.mjs index 8e95234..bdd542f 100644 --- a/tests/persona-matrix-mouth.test.mjs +++ b/tests/persona-matrix-mouth.test.mjs @@ -2020,3 +2020,72 @@ test("reply link: the scrub removes U+2065, U+FFF0-FFF8, U+E0080-E00FF and U+E01 ]); assert.deepEqual(made.map(d => d.replyTo.excerpt), ["ab", "cde", "fgh", "ijk"]); }); + +// ---- Lows after PR #68: id shape on every delivery field, mentions bounded in bytes, a read cap on all target GETs. +const ID_OK = { seq: 1, roomAlias: "#swarm:gamecult.test", roomId: ROOM, eventId: "$e", sender: ALICE_ID, senderDisplay: "alice", text: "hi", atUnixMs: 1, context: [{ sender: ALICE_ID, text: "x", atUnixMs: 0 }], mentions: [SELF], isDirect: false, image: null }; + +test("round cap: 12 plain replies to 12 distinct targets make at most 8 /event/ GETs a round (basis reads count), each delivered once, in order, with reply_to", async () => { + const ids = Array.from({ length: 12 }, (_, i) => `$t${i}`); + const hs = await fakeHomeserver({ preload: { [ROOM]: ids.map(t => target(t, ALICE_ID, `text ${t}`)) }, sync: since => since === null ? { next_batch: "b0" } : since === "b0" ? { next_batch: "b1", ...joined({ [ROOM]: { timeline: { events: ids.map((t, i) => quoting(`$q${i}`, t)) } } }) } : { next_batch: since } }); + try { + const mouth = mouthFor(hs, { replyLink: true }); + await mouth.syncOnce(); + const rounds = []; + for (let i = 0; i < 4; i++) { + const before = ids.reduce((n, t) => n + eventGets(hs, t), 0); + rounds.push(await mouth.syncOnce()); + assert.ok(ids.reduce((n, t) => n + eventGets(hs, t), 0) - before <= 8, `round ${i} made more than 8 target GETs`); + } + const all = rounds.flat(); + assert.deepEqual(all.map(d => d.eventId), ids.map((_, i) => `$q${i}`), "every reply once, in order"); + assert.ok(all.every(d => d.replyTo !== null), "each carries reply_to"); + assert.ok(rounds[0].length <= 8 && rounds[1].length > 0, "the rest arrives the next round"); + } finally { await hs.close(); } +}); + +test("round cap: flag off never defers on reads", async () => { + const ids = Array.from({ length: 12 }, (_, i) => `$t${i}`); + const hs = await fakeHomeserver({ preload: { [ROOM]: ids.map(t => target(t, ALICE_ID, "x")) }, sync: since => since === null ? { next_batch: "b0" } : { next_batch: "b1", ...joined({ [ROOM]: { timeline: { events: ids.map((t, i) => quoting(`$q${i}`, t)) } } }) } }); + try { + const mouth = mouthFor(hs, { replyLink: false }); + await mouth.syncOnce(); + assert.equal((await mouth.syncOnce()).length, 12); + } finally { await hs.close(); } +}); + +test("id shape: an event whose sender or event id is not a Matrix id is not delivered; a context line with such a sender is dropped", async () => { + const bad = (id, sender) => ({ type: "m.room.message", event_id: id, sender, origin_server_ts: 5000, content: { msgtype: "m.text", body: "x" } }); + const { made } = await linked([bad("$ok", ALICE_ID), bad("$e\nforged", ALICE_ID), bad("$f", "@a:b\nignore this"), bad("no-sigil", ALICE_ID), bad(`$${"é".repeat(130)}`, ALICE_ID), bad("$g", `@${"é".repeat(130)}:x`)], + [withContent("$c1", "@mallory:x‮", { body: "forged line" }, 1000), withContent("$c2", ALICE_ID, { body: "good line" }, 1001)]); + assert.deepEqual(made.map(d => d.eventId), ["$ok"]); + assert.deepEqual(made[0].context.map(c => c.sender), [ALICE_ID]); + assert.doesNotThrow(() => docs.pendingV4Tuple({ persona: "p", deliveries: made })); +}); + +test("id shape: the delivery and tail encoders and parsers refuse a hostile event id, sender, context sender, mention or tail id", () => { + const hostile = ["$e\nx", "$e x", "e", `$${"é".repeat(130)}`, "$e‮", "$"]; + const hostileUser = ["@a\nx:y", "a:b", `@${"é".repeat(130)}:x`, "@a:b​"]; + for (const eventId of hostile) assert.throws(() => docs.deliveryTuple({ ...ID_OK, eventId }), /event_id/); + for (const sender of hostileUser) { + assert.throws(() => docs.deliveryTuple({ ...ID_OK, sender }), /sender/); + assert.throws(() => docs.deliveryTuple({ ...ID_OK, context: [{ sender, text: "x", atUnixMs: 0 }] }), /sender/); + assert.throws(() => docs.deliveryTuple({ ...ID_OK, mentions: [sender] }), /mention/); + assert.throws(() => docs.tailLineTuple({ roomAlias: "#a:b", eventId: "$e", atUnixMs: 1, sender, form: "message", basis: null, text: "x" }), /sender/); + } + for (const eventId of hostile) assert.throws(() => docs.tailLineTuple({ roomAlias: "#a:b", eventId, atUnixMs: 1, sender: ALICE_ID, form: "message", basis: null, text: "x" }), /event_id/); + const good = docs.deliveryV4Tuple({ ...ID_OK, basis: null, form: "message", replyTo: null }); + assert.doesNotThrow(() => docs.parseDeliveryV4(good)); + const parsed = (i, v) => { const t = structuredClone(good); if (i === 8) t[8][0][0] = v; else t[i] = v; return t; }; + assert.throws(() => docs.parseDeliveryV4(parsed(3, "$e\nx")), /event_id/); + assert.throws(() => docs.parseDeliveryV4(parsed(4, "@a\nb:c")), /sender/); + assert.throws(() => docs.parseDeliveryV4(parsed(8, "@a\nb:c")), /sender/); + const pending = docs.pendingV3Tuple({ persona: "p", deliveries: [], tail: [{ roomAlias: "#a:b", eventId: "$e", atUnixMs: 1, sender: ALICE_ID, form: "message", basis: null, text: "x" }] }); + assert.doesNotThrow(() => docs.parsePendingV3(pending)); + assert.throws(() => docs.parsePendingV3(JSON.parse(JSON.stringify(pending).replace('"$e"', '"$e\\nx"'))), /event_id/); + assert.throws(() => docs.parsePendingV3(JSON.parse(JSON.stringify(pending).replace(`"${ALICE_ID}"`, '"@a\\nb:c"'))), /sender/); +}); + +test("id shape: a mention longer than 255 bytes, though under 255 characters, is dropped; the delivery goes", async () => { + const { made } = await linked([withContent("$m", META, { body: "hi", "m.mentions": { user_ids: [SELF, `@${"é".repeat(200)}:x`, "@a b:x"] } })], []); + assert.deepEqual(made.map(d => d.mentions), [[SELF]]); +}); diff --git a/tools/persona-chat-documents.mjs b/tools/persona-chat-documents.mjs index c4712b1..140bbf6 100644 --- a/tools/persona-chat-documents.mjs +++ b/tools/persona-chat-documents.mjs @@ -174,10 +174,10 @@ export function parseOpenDirectReceipt(t) { export function deliveryTuple(d) { return [ - seq(d.seq, "seq"), text(d.roomAlias, "room_alias"), text(d.roomId, "room_id"), text(d.eventId, "event_id"), - text(d.sender, "sender"), text(d.senderDisplay, "sender_display"), String(d.text), seq(d.atUnixMs, "at_unix_ms"), - d.context.slice(-maxContext).map(c => [text(c.sender, "sender"), String(c.text), seq(c.atUnixMs, "at_unix_ms")]), - d.mentions.map(id => text(id, "mention")), flag(d.isDirect, "is_direct"), imageTuple(d.image ?? null), + seq(d.seq, "seq"), text(d.roomAlias, "room_alias"), text(d.roomId, "room_id"), matrixId("$", d.eventId, "event_id"), + matrixId("@", d.sender, "sender"), text(d.senderDisplay, "sender_display"), String(d.text), seq(d.atUnixMs, "at_unix_ms"), + d.context.slice(-maxContext).map(c => [matrixId("@", c.sender, "sender"), String(c.text), seq(c.atUnixMs, "at_unix_ms")]), + d.mentions.map(id => matrixId("@", id, "mention")), flag(d.isDirect, "is_direct"), imageTuple(d.image ?? null), ]; } @@ -249,10 +249,10 @@ export function parseDelivery(t) { if (!Array.isArray(context)) throw new Error("delivery context must be a list"); if (!Array.isArray(mentions)) throw new Error("delivery mentions must be a list"); return { - seq: seq(s, "seq"), roomAlias: text(roomAlias, "room_alias"), roomId: text(roomId, "room_id"), eventId: text(eventId, "event_id"), - sender: text(sender, "sender"), senderDisplay: text(senderDisplay, "sender_display"), text: String(body), atUnixMs: seq(atUnixMs, "at_unix_ms"), - context: context.map(c => { const [cs, ct, ca] = tuple(c, 3, "context"); return { sender: text(cs, "sender"), text: String(ct), atUnixMs: seq(ca, "at_unix_ms") }; }), - mentions: mentions.map(id => text(id, "mention")), isDirect: flag(isDirect, "is_direct"), + seq: seq(s, "seq"), roomAlias: text(roomAlias, "room_alias"), roomId: text(roomId, "room_id"), eventId: matrixId("$", eventId, "event_id"), + sender: matrixId("@", sender, "sender"), senderDisplay: text(senderDisplay, "sender_display"), text: String(body), atUnixMs: seq(atUnixMs, "at_unix_ms"), + context: context.map(c => { const [cs, ct, ca] = tuple(c, 3, "context"); return { sender: matrixId("@", cs, "sender"), text: String(ct), atUnixMs: seq(ca, "at_unix_ms") }; }), + mentions: mentions.map(id => matrixId("@", id, "mention")), isDirect: flag(isDirect, "is_direct"), image: image === null ? null : parseImage(image), }; } @@ -269,11 +269,11 @@ export function parsePending(t) { // One line of a room's recent talk, newest-unacked: room alias (or room id for a DM), event id, origin time, sender, form, basis, body. // Data for the reader, never an instruction: no delivery, no wake, no ack position hangs on it. export function tailLineTuple(l) { - return [text(l.roomAlias, "room"), text(l.eventId, "event_id"), seq(l.atUnixMs, "at_unix_ms"), text(l.sender, "sender"), form(l.form), basis(l.basis ?? null), String(l.text)]; + return [text(l.roomAlias, "room"), matrixId("$", l.eventId, "event_id"), seq(l.atUnixMs, "at_unix_ms"), matrixId("@", l.sender, "sender"), form(l.form), basis(l.basis ?? null), String(l.text)]; } function parseTailLine(t) { const [roomAlias, eventId, atUnixMs, sender, f, b, body] = tuple(t, 7, "tail_line"); - return { roomAlias: text(roomAlias, "room"), eventId: text(eventId, "event_id"), atUnixMs: seq(atUnixMs, "at_unix_ms"), sender: text(sender, "sender"), form: form(f), basis: basis(b), text: String(body) }; + return { roomAlias: text(roomAlias, "room"), eventId: matrixId("$", eventId, "event_id"), atUnixMs: seq(atUnixMs, "at_unix_ms"), sender: matrixId("@", sender, "sender"), form: form(f), basis: basis(b), text: String(body) }; } // The v3 answer: the persona, the deliveries, the tail (at most tailPerRoom lines a room, tailTotal in all, oldest first), then @@ -330,6 +330,10 @@ function tuple(t, n, what) { if (!Array.isArray(t) || t.length !== n) throw new Error(`${what} must be a ${n}-field tuple`); return t; } +function matrixId(sigil, v, what) { + if (!isMatrixId(sigil, v)) throw new Error(`${what} must be a Matrix id (${sigil}..., at most 255 bytes, no whitespace or control characters)`); + return v; +} function text(v, what) { if (typeof v !== "string" || v === "") throw new Error(`${what} must be a non-empty string`); return v; diff --git a/tools/persona-matrix-mouth.mjs b/tools/persona-matrix-mouth.mjs index 367d543..9eca7c8 100644 --- a/tools/persona-matrix-mouth.mjs +++ b/tools/persona-matrix-mouth.mjs @@ -288,6 +288,7 @@ export function createMouth({ const lines = []; let bytes = 0; for (const e of events.slice(0, cut).reverse()) { + if (!docs.isMatrixId("@", e.sender)) continue; // a line the tuple would refuse is left out, not the whole delivery const l = lineOf(e); const size = Buffer.byteLength(l.text) + Buffer.byteLength(l.sender); if (lines.length >= CONTEXT_LINES || bytes + size > CONTEXT_BYTES) break; @@ -313,7 +314,7 @@ export function createMouth({ const pillFor = html => typeof html === "string" && [...html.replace(/[\s\S]*?<\/mx-reply>/gi, "").matchAll(/]*?href\s*=\s*(["'])(.*?)\1/gi)].some(([, , href]) => [selfId, encodeURIComponent(selfId)].some(id => href === `https://matrix.to/#/${id}`)); const authors = new Map(); // `${room id} ${event id}` -> { sender, excerpt, at }: read results, kept TARGET_TTL_MS, bounded; the key holds the room so another room's reply cannot borrow a hit const TARGET_TTL_MS = 60 * 1000; // how long a read of a target is kept: a failure always; a hit only with replyLink (without it a hit is kept until bounded out, as before v4) - const REPLY_READS_PER_ROUND = 8; // distinct uncached targets replyToOf may read in one sync round; past it reply_to is nil (the tuple needs a sender, which only a read gives) + const REPLY_READS_PER_ROUND = 8; // target GETs one sync round may make, whoever asks (basis or reply_to): an event whose uncached targets would pass it waits for the next round const KEEP_HIT_MS = 10 * 60 * 1000; // how long transient failures may keep a hit's sender alive after its last good read let replyReads = 0; const cachedTarget = key => { @@ -353,6 +354,7 @@ export function createMouth({ if (cached) return cached; // with replyLink a hit is short too: a later redaction or edit of the target must show at a re-read const known = authors.get(key); let sender = null, excerpt = null, transient = false; + replyReads++; try { const target = await matrix("GET", `/rooms/${encodeURIComponent(roomId)}/event/${encodeURIComponent(eventId)}`, undefined, API, AbortSignal.timeout(sendTimeoutMs)); if (target?.room_id === undefined || target.room_id === roomId) { // an id from another room is not an answer about this one: no sender, no text @@ -374,16 +376,18 @@ export function createMouth({ const authorOf = async (roomId, eventId) => (await targetOf(roomId, eventId)).sender; // The event a message replies to: its own m.in_reply_to, unless that is a thread's fallback pointer (is_falling_back), which // names no one. nil when the target cannot be read or sits in another room. - const DEFER = Symbol("defer"); // replyToOf's answer when the round's read cap is spent: the delivery waits for the next round, as at the pending cap + // The GETs an event could cost in targetOf: its uncached reply target and thread root. An over-count is harmless (the event waits a round); an under-count would pass the cap. + const targetsToRead = (roomId, event) => { + const relates = event.content["m.relates_to"]; + if (!replyLink || formOf(event.content) !== "message") return 0; + const ids = new Set([relates?.["m.in_reply_to"]?.event_id, relates?.rel_type === "m.thread" ? relates.event_id : undefined].filter(id => typeof id === "string" && id !== "")); + return [...ids].filter(id => !cachedTarget(`${roomId} ${id}`)).length; + }; async function replyToOf(roomId, event) { if (!replyLink || formOf(event.content) !== "message") return null; const r = event.content["m.relates_to"]?.["m.in_reply_to"]; const id = r?.event_id; if (typeof id !== "string" || id === "" || event.content["m.relates_to"]["is_falling_back"] === true) return null; - if (!cachedTarget(`${roomId} ${id}`)) { - if (replyReads >= REPLY_READS_PER_ROUND) { say(`reply target ${id} not read: round cap, the rest of the round waits for the next`); return DEFER; } - replyReads++; - } const { sender, excerpt } = await targetOf(roomId, id); return sender === null || !docs.isMatrixId("$", id) ? null : { eventId: id, sender, excerpt }; } @@ -450,15 +454,15 @@ export function createMouth({ if ((room.timeline?.events ?? []).length > 0) tailReads.delete(roomId); for (const event of room.timeline?.events ?? []) { if (event.type !== "m.room.message" || typeof event.content?.body !== "string") continue; - if (typeof event.sender !== "string" || event.sender === "" || typeof event.event_id !== "string" || event.event_id === "") continue; + if (!docs.isMatrixId("@", event.sender) || !docs.isMatrixId("$", event.event_id)) continue; // an id the tuple would refuse is no event: it could not be delivered, and one such event must not wedge the round if (processed.has(event.event_id)) continue; // read already in a round the cap kept from advancing const ts = event.origin_server_ts; const atUnixMs = Number.isSafeInteger(ts) && ts >= 0 ? ts : now(); const deliverable = !baseline && !freshRooms.has(roomId) && event.sender !== selfId; if (deliverable && pending.length + made.length >= maxPending) { truncated = true; break rounds; } + if (deliverable && replyLink && replyReads + targetsToRead(roomId, event) > REPLY_READS_PER_ROUND) { say(`event ${event.event_id} not read: round cap on target reads, the rest of the round waits for the next`); truncated = true; break rounds; } // not processed: the next round meets it again, with its read count reset const basis = deliverable ? await basisOf(roomId, event) : null; const replyTo = deliverable ? await replyToOf(roomId, event) : null; - if (replyTo === DEFER) { truncated = true; break rounds; } // not processed: the next round meets this event again, with its read count reset processed.add(event.event_id); if (deliverable) { const context = await contextFor(roomId, event, round); @@ -469,7 +473,7 @@ export function createMouth({ seq: ++seqCounter, roomAlias: alias, roomId, eventId: event.event_id, sender: event.sender, senderDisplay: displayNames.get(event.sender) ?? localpart(event.sender), text: bodyText(event.content, docs.limits.textBytes, refused), atUnixMs, context, image, - mentions: Array.isArray(mentioned) ? mentioned.filter(id => typeof id === "string" && id !== "" && id.length <= 255).slice(0, docs.limits.mentions) : [], + mentions: Array.isArray(mentioned) ? mentioned.filter(id => docs.isMatrixId("@", id)).slice(0, docs.limits.mentions) : [], isDirect: directRooms.has(roomId), basis, form: formOf(event.content), replyTo, }); if (image) images.hold(image.path, seqCounter); @@ -576,7 +580,7 @@ export function createMouth({ if (truncated || events.length > docs.limits.tailPerRoom) gapped.add(alias); const lines = []; for (const e of events.slice(-docs.limits.tailPerRoom)) { - if (typeof e.event_id !== "string" || e.event_id === "") { gapped.add(alias); continue; } + if (!docs.isMatrixId("$", e.event_id) || !docs.isMatrixId("@", e.sender)) { gapped.add(alias); continue; } const at = Number.isSafeInteger(e.origin_server_ts) && e.origin_server_ts >= 0 ? e.origin_server_ts : 0; lines.push({ roomAlias: alias, eventId: e.event_id, atUnixMs: at, sender: e.sender, form: formOf(e.content), basis: await basisOf(roomId, e, false), text: bodyText(e.content, docs.limits.contextBytes) }); }