From 98ead531b04c8c741e3b43b5ded5d3f86810dd36 Mon Sep 17 00:00:00 2001 From: Heimdall Date: Wed, 11 Nov 2026 16:12:41 +0000 Subject: [PATCH 1/3] Mouth: v1-v3 target reads as before v4, failed re-read keeps the hit, wider excerpt scrub, reply_to read cap Co-Authored-By: Claude Sonnet 5.5 --- docs/bridge.md | 2 +- tests/persona-matrix-mouth.test.mjs | 58 +++++++++++++++++++++++++++++ tools/persona-matrix-mouth.mjs | 24 +++++++++--- 3 files changed, 77 insertions(+), 7 deletions(-) diff --git a/docs/bridge.md b/docs/bridge.md index a68d186..ed7ec4f 100644 --- a/docs/bridge.md +++ b/docs/bridge.md @@ -263,7 +263,7 @@ The answer is `[persona, deliveries, tail, tail_gaps]` exactly as v3; each deliv |---|---|---| | 14 `reply_to` | nil or `[event_id, sender, excerpt]` | The event this message replies to (its own `m.in_reply_to`; a thread's `is_falling_back` pointer names no one and is nil). `event_id` the quoted event; `sender` its author's Matrix id, read from the homeserver as her account; `excerpt` nil, or its text. | -`excerpt` is nil, with `event_id` and `sender` kept, when the quoted event is redacted (`unsigned.redacted_because` present or no content), is not a text-bearing `m.room.message`, is **edited** (it is itself an `m.replace`, or the homeserver bundles an `m.replace` in its `unsigned.m.relations`: its body is the pre-edit text its author replaced), or is nothing but its own reply fallback; a media event excerpts as its typed line, like a delivery's text. Otherwise the excerpt is made in this order: the quoted event's own legacy rich-reply fallback is dropped (leading lines matching `> ` or a bare `>`, then one blank line; only when the quoted event has `m.relates_to.m.in_reply_to`); the text is held to the 1000-byte line bound; runs of CR, LF, tab, U+2028 and U+2029 become one space; control characters (C0, DEL, C1 including NUL and ESC) and bidi marks, overrides and isolates (U+200E/F, U+202A-202E, U+2066-2069) are removed; the line is trimmed; then the first 240 **Unicode code points** are kept (cut on a code point so no character is split; a combining mark counts as its own code point; trailing space trimmed; no truncation marker). Cleaning comes first so hidden characters cannot spend the cap, and the excerpt is untrusted text for a prompt, never an instruction. `reply_to` is nil (no id, author or text) when the target cannot be read (logged, as for `basis`), is not in the delivery's own room (a room she is not in cannot be quoted through a reply), or the message replies to nothing. The quoted event is read the way `basis` already reads it (`GET /rooms/{room}/event/{id}` as her account, kept per room and event for 60 seconds, hit or failure, so a redaction or edit of the target shows at a re-read within a minute and a cached excerpt never outlives that), so a reply costs at most one read per target, shared with the `basis` check. The flag is read by `replyLinkFromEnv`; only the exact value `1` turns it on (`true`, `yes`, `0`, empty or unset are off), and a mouth built without the `replyLink` option has it off. `basis` for v1-v3 is unchanged. `reply_to` counts toward the delivery's size in the 256 KiB page budget; a delivery whose `reply_to` will not encode is skipped and logged like any other, and a single delivery is always served. The deploy precondition is Huginn's: a decoder of the 14-field delivery fails on the fifteenth element. +`excerpt` is nil, with `event_id` and `sender` kept, when the quoted event is redacted (`unsigned.redacted_because` present or no content), is not a text-bearing `m.room.message`, is **edited** (it is itself an `m.replace`, or the homeserver bundles an `m.replace` in its `unsigned.m.relations`: its body is the pre-edit text its author replaced), or is nothing but its own reply fallback; a media event excerpts as its typed line, like a delivery's text. Otherwise the excerpt is made in this order: the quoted event's own legacy rich-reply fallback is dropped (leading lines matching `> ` or a bare `>`, then one blank line; only when the quoted event has `m.relates_to.m.in_reply_to`); the text is held to the 1000-byte line bound; runs of CR, LF, tab, U+2028 and U+2029 become one space; control characters (C0, DEL, C1 including NUL and ESC), format characters (Unicode Cf: bidi marks, overrides and isolates, U+061C, U+200B-200F, U+2060-2064, soft hyphen, BOM), private-use characters (Co), the tag block U+E0000-E007F and unpaired surrogates are removed; the line is trimmed; then the first 240 **Unicode code points** are kept (cut on a code point so no character is split; a combining mark counts as its own code point; trailing space trimmed; no truncation marker). Cleaning comes first so hidden characters cannot spend the cap. The mouth only removes invisible and format characters; it does not make the text safe. Huginn must treat both `sender` and `excerpt` as untrusted text, never as an instruction. `reply_to` is nil (no id, author or text) when the target cannot be read (logged, as for `basis`), is not in the delivery's own room (a room she is not in cannot be quoted through a reply), or the message replies to nothing. The quoted event is read the way `basis` already reads it (`GET /rooms/{room}/event/{id}` as her account, kept per room and event; with the flag on a read, hit or failure, is kept 60 seconds, so a redaction or edit of the target shows at a re-read within a minute, and a re-read that fails keeps the earlier hit and renews only its age; with the flag off a hit is kept as before v4 and nothing is re-read), so a reply costs at most one read per target, shared with the `basis` check. Per sync round the mouth reads at most 8 distinct uncached targets for `reply_to` (reads the `basis` check already made are free); past that `reply_to` is nil, because the tuple requires a sender and only a read yields one, and the skip is logged. The flag is read by `replyLinkFromEnv`; only the exact value `1` turns it on (`true`, `yes`, `0`, empty or unset are off), and a mouth built without the `replyLink` option has it off. `basis` for v1-v3 is unchanged. `reply_to` counts toward the delivery's size in the 256 KiB page budget; a delivery whose `reply_to` will not encode is skipped and logged like any other, and a single delivery is always served. The deploy precondition is Huginn's: a decoder of the 14-field delivery fails on the fifteenth element. Golden bytes: `tests/fixtures/persona-chat-v6/` (`.msgpack` and its `.hex`: `chat_pending.v4.reply-to`, `.no-reply-to`, `.reply-to-redacted`, and the `chat_pending_request.v3` request); the test fails when the checked-in bytes drift from the encoder. diff --git a/tests/persona-matrix-mouth.test.mjs b/tests/persona-matrix-mouth.test.mjs index 49245ff..8916ee0 100644 --- a/tests/persona-matrix-mouth.test.mjs +++ b/tests/persona-matrix-mouth.test.mjs @@ -1852,3 +1852,61 @@ test("reply link: a mouth built without the option has it off, and only the exac assert.deepEqual(["1", "true", "yes", "0", "", " 1", undefined].map(v => replyLinkFromEnv({ CHAT_PENDING_V4: v })), [true, false, false, false, false, false, false]); assert.equal(replyLinkFromEnv({}), false); }); + +// ---- Fixes after PR #63: the target cache, the excerpt scrub, the per-round read cap. +const eventGets = (hs, id) => hs.seen.filter(r => r.method === "GET" && r.path.endsWith(`/event/${id}`)).length; +async function replyAcrossTtl(replyLink) { + const clock = { t: 1_000_000 }; + let down = false, attempts = 0; + const hs = await fakeHomeserver({ preload: { [ROOM]: [target("$hers", SELF, "my words")] }, sync: since => since === null ? { next_batch: "b0" } : { next_batch: `b${since}x`, ...joined({ [ROOM]: { timeline: { events: [quoting(`$q${since}`, "$hers")] } } }) } }); + try { + const fetch = (u, o) => (String(u).includes("/event/%24hers") && attempts++, down && String(u).includes("/event/") ? Promise.resolve(new Response("{}", { status: 500 })) : hs.fetch(u, o)); + const mouth = mouthFor(hs, { replyLink, now: () => clock.t, fetch }); + await mouth.syncOnce(); + const first = await mouth.syncOnce(); + clock.t += 61_000; down = true; + const later = await mouth.syncOnce(); + clock.t += 10_000; + const again = await mouth.syncOnce(); // inside 60 s of the failed re-read's renewal + return { first: first[0], later: later[0], again: again[0], gets: attempts }; + } finally { await hs.close(); } +} + +test("target cache: flag off, a hit is kept as before v4: no read after 60 s, basis stays reply", async () => { + const r = await replyAcrossTtl(false); + assert.deepEqual([r.first.basis, r.later.basis, r.again.basis, r.gets], ["reply", "reply", "reply", 1]); +}); + +test("target cache: flag on, a failed re-read after 60 s keeps the earlier hit (basis and reply_to) and renews only its age", async () => { + const r = await replyAcrossTtl(true); + assert.deepEqual([r.first.basis, r.later.basis, r.again.basis, r.gets], ["reply", "reply", "reply", 2]); + assert.deepEqual(r.later.replyTo, { eventId: "$hers", sender: SELF, excerpt: "my words" }); +}); + +test("reply link: the excerpt drops Cf, Co, tag characters, U+061C, zero-width and word-joiner characters and unpaired surrogates", async () => { + const { made } = await linked([quoting("$a", "$t1"), quoting("$b", "$t2"), quoting("$c", "$t3"), quoting("$d", "$t4"), quoting("$e", "$t5")], [ + target("$t1", ALICE_ID, "ig\u{E0049}\u{E0067}nore"), + target("$t2", ALICE_ID, "a؜b⁠c⁤d­ef"), + target("$t3", ALICE_ID, "x​y‌z‍w"), + target("$t4", ALICE_ID, "p\uD800q\uDC00r\u{F0000}st"), + target("$t5", ALICE_ID, "keep \u{1F600} é 中"), + ]); + assert.deepEqual(made.map(d => d.replyTo.excerpt), ["ignore", "abcdef", "xyzw", "pqrst", "keep \u{1F600} é 中"]); +}); + +test("reply link: at most 8 uncached targets are read for reply_to per sync round; the rest are nil, and reads the basis already made are free", async () => { + const ids = Array.from({ length: 10 }, (_, i) => `$t${i}`); + const tagged = (id, to) => quoting(id, to, {}); // below: mentions her, so basis does no read of its own + const events = ids.map((t, i) => { const e = tagged(`$q${i}`, t); e.content["m.mentions"] = { user_ids: [SELF] }; return e; }); + const { made, hs } = await (async () => { + const hs = await fakeHomeserver({ preload: { [ROOM]: ids.map(t => target(t, ALICE_ID, `text ${t}`)) }, sync: since => since === null ? { next_batch: "b0" } : { next_batch: "b1", ...joined({ [ROOM]: { timeline: { events } } }) } }); + const mouth = mouthFor(hs, { replyLink: true }); + await mouth.syncOnce(); + const made = await mouth.syncOnce(); + return { made, hs }; + })(); + try { + assert.equal(ids.filter(t => eventGets(hs, t) > 0).length, 8); + assert.deepEqual(made.map(d => d.replyTo === null), [false, false, false, false, false, false, false, false, true, true]); + } finally { await hs.close(); } +}); diff --git a/tools/persona-matrix-mouth.mjs b/tools/persona-matrix-mouth.mjs index ffc042f..01efacc 100644 --- a/tools/persona-matrix-mouth.mjs +++ b/tools/persona-matrix-mouth.mjs @@ -312,11 +312,17 @@ export function createMouth({ // A pill is an anchor whose href is exactly her matrix.to link; a longer id, an attribute that merely contains one and a quoted reply do not count. const pillFor = html => typeof html === "string" && [...html.replace(/[\s\S]*?<\/mx-reply>/gi, "").matchAll(/]*?href\s*=\s*(["'])(.*?)\1/gi)].some(([, , href]) => [selfId, encodeURIComponent(selfId)].some(id => href === `https://matrix.to/#/${id}`)); const authors = new Map(); // `${room id} ${event id}` -> { sender, excerpt, at }: read results, kept TARGET_TTL_MS, bounded; the key holds the room so another room's reply cannot borrow a hit - const TARGET_TTL_MS = 60 * 1000; // how long a read of a target (hit or failure) is kept - // The quoted text, as a prompt-safe line: nil when the event is redacted (the homeserver strips its content), is not text, or is edited + const TARGET_TTL_MS = 60 * 1000; // how long a read of a target is kept: a failure always; a hit only with replyLink (without it a hit is kept until bounded out, as before v4) + const REPLY_READS_PER_ROUND = 8; // distinct uncached targets replyToOf may read in one sync round; past it reply_to is nil (the tuple needs a sender, which only a read gives) + let replyReads = 0; + const cachedTarget = key => { + const known = authors.get(key); + return known && (now() - known.at < TARGET_TTL_MS || (!replyLink && known.sender !== null)) ? known : null; + }; + // The quoted text, as a single line (invisible and format characters removed; still untrusted text): nil when the event is redacted (the homeserver strips its content), is not text, or is edited // (an edit target, or one with an m.replace bundled: the original body is the pre-edit text, which its author replaced), or is nothing but // its own reply fallback. Else: the legacy rich-reply fallback (leading "> " lines and the blank line after) is dropped, CR/LF/tab runs - // become one space, control characters (C0, DEL, C1) and bidi marks/overrides/isolates are removed, the result is trimmed, then cut to the + // become one space, control characters (C0, DEL, C1), format characters (Cf, incl. bidi marks, U+061C, U+200B-200F, U+2060-2064), private use (Co), the tag block U+E0000-E007F and unpaired surrogates are removed, the result is trimmed, then cut to the // first 240 code points (not bytes, so no character is split; a combining mark is its own code point) and trimmed again. Computed // only when replyLink is on. const FALLBACK_LINE = /^>( |$)/; @@ -333,15 +339,16 @@ export function createMouth({ } const line = bodyText({ ...c, body }, docs.limits.contextBytes) .replace(/[\r\n\t\u2028\u2029]+/g, " ") - .replace(/[\u0000-\u001f\u007f-\u009f\u200e\u200f\u202a-\u202e\u2066-\u2069]/g, "") + .replace(/[\u0000-\u001f\u007f-\u009f\u061c\u200b-\u200f\u202a-\u202e\u2060-\u2064\u2066-\u2069]|[\p{Cf}\p{Co}\u{E0000}-\u{E007F}]|[\uD800-\uDFFF]/gu, "") .trim(); const cut = [...line].slice(0, docs.limits.replyExcerpt).join("").trimEnd(); return cut === "" ? null : cut; }; async function targetOf(roomId, eventId) { const key = `${roomId} ${eventId}`; + const cached = cachedTarget(key); + if (cached) return cached; // with replyLink a hit is short too: a later redaction or edit of the target must show at a re-read const known = authors.get(key); - if (known && now() - known.at < TARGET_TTL_MS) return known; // short for hits too: a later redaction or edit of the target must show at a re-read let sender = null, excerpt = null; try { const target = await matrix("GET", `/rooms/${encodeURIComponent(roomId)}/event/${encodeURIComponent(eventId)}`, undefined, API, AbortSignal.timeout(sendTimeoutMs)); @@ -352,7 +359,7 @@ export function createMouth({ } catch (error) { say(`reply target ${eventId} not read: ${error.message}`); } - const entry = { sender, excerpt, at: now() }; + const entry = sender === null && known?.sender != null ? { ...known, at: now() } : { sender, excerpt, at: now() }; // a failed re-read keeps the earlier hit: only its age is renewed authors.set(key, entry); if (authors.size > DELIVERED_INDEX) authors.delete(authors.keys().next().value); return entry; @@ -365,6 +372,10 @@ export function createMouth({ const r = event.content["m.relates_to"]?.["m.in_reply_to"]; const id = r?.event_id; if (typeof id !== "string" || id === "" || event.content["m.relates_to"]["is_falling_back"] === true) return null; + if (!cachedTarget(`${roomId} ${id}`)) { + if (replyReads >= REPLY_READS_PER_ROUND) { say(`reply target ${id} not read: round cap`); return null; } + replyReads++; + } const { sender, excerpt } = await targetOf(roomId, id); return sender === null ? null : { eventId: id, sender, excerpt }; } @@ -405,6 +416,7 @@ export function createMouth({ // One /sync round. Returns the deliveries it made. async function syncOnce() { if (!started || unjoined()) await start(); + replyReads = 0; const free = await released(); // the round's brake read for presence, taken before any early return await setPresence(free); if (pending.length >= maxPending) return []; // unanswered mail is not outrun: the position stays where it is -- 2.47.3 From 298e7790c46ee10e91ca34f60b9937bbb159a4ea Mon Sep 17 00:00:00 2001 From: Heimdall Date: Wed, 11 Nov 2026 16:13:37 +0000 Subject: [PATCH 2/3] Mouth: a failed re-read keeps the earlier hit only on transient failures Co-Authored-By: Claude Sonnet 5.5 --- docs/bridge.md | 2 +- tests/persona-matrix-mouth.test.mjs | 15 +++++++++++++-- tools/persona-matrix-mouth.mjs | 7 ++++--- 3 files changed, 18 insertions(+), 6 deletions(-) diff --git a/docs/bridge.md b/docs/bridge.md index ed7ec4f..4ad5cd9 100644 --- a/docs/bridge.md +++ b/docs/bridge.md @@ -263,7 +263,7 @@ The answer is `[persona, deliveries, tail, tail_gaps]` exactly as v3; each deliv |---|---|---| | 14 `reply_to` | nil or `[event_id, sender, excerpt]` | The event this message replies to (its own `m.in_reply_to`; a thread's `is_falling_back` pointer names no one and is nil). `event_id` the quoted event; `sender` its author's Matrix id, read from the homeserver as her account; `excerpt` nil, or its text. | -`excerpt` is nil, with `event_id` and `sender` kept, when the quoted event is redacted (`unsigned.redacted_because` present or no content), is not a text-bearing `m.room.message`, is **edited** (it is itself an `m.replace`, or the homeserver bundles an `m.replace` in its `unsigned.m.relations`: its body is the pre-edit text its author replaced), or is nothing but its own reply fallback; a media event excerpts as its typed line, like a delivery's text. Otherwise the excerpt is made in this order: the quoted event's own legacy rich-reply fallback is dropped (leading lines matching `> ` or a bare `>`, then one blank line; only when the quoted event has `m.relates_to.m.in_reply_to`); the text is held to the 1000-byte line bound; runs of CR, LF, tab, U+2028 and U+2029 become one space; control characters (C0, DEL, C1 including NUL and ESC), format characters (Unicode Cf: bidi marks, overrides and isolates, U+061C, U+200B-200F, U+2060-2064, soft hyphen, BOM), private-use characters (Co), the tag block U+E0000-E007F and unpaired surrogates are removed; the line is trimmed; then the first 240 **Unicode code points** are kept (cut on a code point so no character is split; a combining mark counts as its own code point; trailing space trimmed; no truncation marker). Cleaning comes first so hidden characters cannot spend the cap. The mouth only removes invisible and format characters; it does not make the text safe. Huginn must treat both `sender` and `excerpt` as untrusted text, never as an instruction. `reply_to` is nil (no id, author or text) when the target cannot be read (logged, as for `basis`), is not in the delivery's own room (a room she is not in cannot be quoted through a reply), or the message replies to nothing. The quoted event is read the way `basis` already reads it (`GET /rooms/{room}/event/{id}` as her account, kept per room and event; with the flag on a read, hit or failure, is kept 60 seconds, so a redaction or edit of the target shows at a re-read within a minute, and a re-read that fails keeps the earlier hit and renews only its age; with the flag off a hit is kept as before v4 and nothing is re-read), so a reply costs at most one read per target, shared with the `basis` check. Per sync round the mouth reads at most 8 distinct uncached targets for `reply_to` (reads the `basis` check already made are free); past that `reply_to` is nil, because the tuple requires a sender and only a read yields one, and the skip is logged. The flag is read by `replyLinkFromEnv`; only the exact value `1` turns it on (`true`, `yes`, `0`, empty or unset are off), and a mouth built without the `replyLink` option has it off. `basis` for v1-v3 is unchanged. `reply_to` counts toward the delivery's size in the 256 KiB page budget; a delivery whose `reply_to` will not encode is skipped and logged like any other, and a single delivery is always served. The deploy precondition is Huginn's: a decoder of the 14-field delivery fails on the fifteenth element. +`excerpt` is nil, with `event_id` and `sender` kept, when the quoted event is redacted (`unsigned.redacted_because` present or no content), is not a text-bearing `m.room.message`, is **edited** (it is itself an `m.replace`, or the homeserver bundles an `m.replace` in its `unsigned.m.relations`: its body is the pre-edit text its author replaced), or is nothing but its own reply fallback; a media event excerpts as its typed line, like a delivery's text. Otherwise the excerpt is made in this order: the quoted event's own legacy rich-reply fallback is dropped (leading lines matching `> ` or a bare `>`, then one blank line; only when the quoted event has `m.relates_to.m.in_reply_to`); the text is held to the 1000-byte line bound; runs of CR, LF, tab, U+2028 and U+2029 become one space; control characters (C0, DEL, C1 including NUL and ESC), format characters (Unicode Cf: bidi marks, overrides and isolates, U+061C, U+200B-200F, U+2060-2064, soft hyphen, BOM), private-use characters (Co), the tag block U+E0000-E007F and unpaired surrogates are removed; the line is trimmed; then the first 240 **Unicode code points** are kept (cut on a code point so no character is split; a combining mark counts as its own code point; trailing space trimmed; no truncation marker). Cleaning comes first so hidden characters cannot spend the cap. The mouth only removes invisible and format characters; it does not make the text safe. Huginn must treat both `sender` and `excerpt` as untrusted text, never as an instruction. `reply_to` is nil (no id, author or text) when the target cannot be read (logged, as for `basis`), is not in the delivery's own room (a room she is not in cannot be quoted through a reply), or the message replies to nothing. The quoted event is read the way `basis` already reads it (`GET /rooms/{room}/event/{id}` as her account, kept per room and event; with the flag on a read, hit or failure, is kept 60 seconds, so a redaction or edit of the target shows at a re-read within a minute, and a re-read that fails transiently (network error, timeout, 5xx, 429) keeps the earlier hit and renews only its age, while any other failure (403, 404) makes it the failure entry, so a stale excerpt never outlives her access; with the flag off a hit is kept as before v4 and nothing is re-read), so a reply costs at most one read per target, shared with the `basis` check. Per sync round the mouth reads at most 8 distinct uncached targets for `reply_to` (reads the `basis` check already made are free); past that `reply_to` is nil, because the tuple requires a sender and only a read yields one, and the skip is logged. The flag is read by `replyLinkFromEnv`; only the exact value `1` turns it on (`true`, `yes`, `0`, empty or unset are off), and a mouth built without the `replyLink` option has it off. `basis` for v1-v3 is unchanged. `reply_to` counts toward the delivery's size in the 256 KiB page budget; a delivery whose `reply_to` will not encode is skipped and logged like any other, and a single delivery is always served. The deploy precondition is Huginn's: a decoder of the 14-field delivery fails on the fifteenth element. Golden bytes: `tests/fixtures/persona-chat-v6/` (`.msgpack` and its `.hex`: `chat_pending.v4.reply-to`, `.no-reply-to`, `.reply-to-redacted`, and the `chat_pending_request.v3` request); the test fails when the checked-in bytes drift from the encoder. diff --git a/tests/persona-matrix-mouth.test.mjs b/tests/persona-matrix-mouth.test.mjs index 8916ee0..012f87a 100644 --- a/tests/persona-matrix-mouth.test.mjs +++ b/tests/persona-matrix-mouth.test.mjs @@ -1855,12 +1855,12 @@ test("reply link: a mouth built without the option has it off, and only the exac // ---- Fixes after PR #63: the target cache, the excerpt scrub, the per-round read cap. const eventGets = (hs, id) => hs.seen.filter(r => r.method === "GET" && r.path.endsWith(`/event/${id}`)).length; -async function replyAcrossTtl(replyLink) { +async function replyAcrossTtl(replyLink, failure = 500) { const clock = { t: 1_000_000 }; let down = false, attempts = 0; const hs = await fakeHomeserver({ preload: { [ROOM]: [target("$hers", SELF, "my words")] }, sync: since => since === null ? { next_batch: "b0" } : { next_batch: `b${since}x`, ...joined({ [ROOM]: { timeline: { events: [quoting(`$q${since}`, "$hers")] } } }) } }); try { - const fetch = (u, o) => (String(u).includes("/event/%24hers") && attempts++, down && String(u).includes("/event/") ? Promise.resolve(new Response("{}", { status: 500 })) : hs.fetch(u, o)); + const fetch = (u, o) => (String(u).includes("/event/%24hers") && attempts++, down && String(u).includes("/event/") ? (failure === "timeout" ? Promise.reject(new Error("timed out")) : Promise.resolve(new Response("{}", { status: failure }))) : hs.fetch(u, o)); const mouth = mouthFor(hs, { replyLink, now: () => clock.t, fetch }); await mouth.syncOnce(); const first = await mouth.syncOnce(); @@ -1910,3 +1910,14 @@ test("reply link: at most 8 uncached targets are read for reply_to per sync roun assert.deepEqual(made.map(d => d.replyTo === null), [false, false, false, false, false, false, false, false, true, true]); } finally { await hs.close(); } }); + +test("target cache: flag on, 500, 429 and a timeout keep the earlier hit at the re-read; 403 and 404 drop it (reply_to nil, basis as the failure case)", async () => { + for (const failure of [500, 429, "timeout"]) { + const r = await replyAcrossTtl(true, failure); + assert.deepEqual([r.later.basis, r.later.replyTo?.excerpt], ["reply", "my words"], String(failure)); + } + for (const failure of [403, 404]) { + const r = await replyAcrossTtl(true, failure); + assert.deepEqual([r.first.basis, r.later.basis, r.later.replyTo, r.again.replyTo], ["reply", null, null, null], String(failure)); + } +}); diff --git a/tools/persona-matrix-mouth.mjs b/tools/persona-matrix-mouth.mjs index 01efacc..0218817 100644 --- a/tools/persona-matrix-mouth.mjs +++ b/tools/persona-matrix-mouth.mjs @@ -166,7 +166,7 @@ export function createMouth({ } catch (error) { throw new Error(`homeserver unreachable: ${redact(error?.message ?? String(error), token)}`); } - if (!response.ok) throw new Error(`homeserver answered ${response.status} for ${method} ${path.split("?")[0]}`); + if (!response.ok) throw Object.assign(new Error(`homeserver answered ${response.status} for ${method} ${path.split("?")[0]}`), { status: response.status }); return response.json(); } @@ -349,7 +349,7 @@ export function createMouth({ const cached = cachedTarget(key); if (cached) return cached; // with replyLink a hit is short too: a later redaction or edit of the target must show at a re-read const known = authors.get(key); - let sender = null, excerpt = null; + let sender = null, excerpt = null, transient = false; try { const target = await matrix("GET", `/rooms/${encodeURIComponent(roomId)}/event/${encodeURIComponent(eventId)}`, undefined, API, AbortSignal.timeout(sendTimeoutMs)); if (target?.room_id === undefined || target.room_id === roomId) { // an id from another room is not an answer about this one: no sender, no text @@ -357,9 +357,10 @@ export function createMouth({ if (sender !== null) excerpt = excerptOf(target); } } catch (error) { + transient = error.status === undefined || error.status >= 500 || error.status === 429; // network, timeout, 5xx, 429: the answer is unknown; 403, 404 and any other refusal are an answer say(`reply target ${eventId} not read: ${error.message}`); } - const entry = sender === null && known?.sender != null ? { ...known, at: now() } : { sender, excerpt, at: now() }; // a failed re-read keeps the earlier hit: only its age is renewed + const entry = sender === null && transient && known?.sender != null ? { ...known, at: now() } : { sender, excerpt, at: now() }; // a re-read that fails transiently (network, timeout, 5xx, 429) keeps the earlier hit and renews only its age; any other failure (403, 404, wrong room, no sender) replaces it with the failure entry authors.set(key, entry); if (authors.size > DELIVERED_INDEX) authors.delete(authors.keys().next().value); return entry; -- 2.47.3 From 5a7873c4ede9d878240ba4f849152c267f373ffd Mon Sep 17 00:00:00 2001 From: Heimdall Date: Wed, 11 Nov 2026 16:16:38 +0000 Subject: [PATCH 3/3] Mouth: bound transient-failure retention to 10 min from the last good read; test the per-round read reset Co-Authored-By: Claude Sonnet 5.5 --- docs/bridge.md | 2 +- tests/persona-matrix-mouth.test.mjs | 37 +++++++++++++++++++++++++++++ tools/persona-matrix-mouth.mjs | 6 ++++- 3 files changed, 43 insertions(+), 2 deletions(-) diff --git a/docs/bridge.md b/docs/bridge.md index 4ad5cd9..802d667 100644 --- a/docs/bridge.md +++ b/docs/bridge.md @@ -263,7 +263,7 @@ The answer is `[persona, deliveries, tail, tail_gaps]` exactly as v3; each deliv |---|---|---| | 14 `reply_to` | nil or `[event_id, sender, excerpt]` | The event this message replies to (its own `m.in_reply_to`; a thread's `is_falling_back` pointer names no one and is nil). `event_id` the quoted event; `sender` its author's Matrix id, read from the homeserver as her account; `excerpt` nil, or its text. | -`excerpt` is nil, with `event_id` and `sender` kept, when the quoted event is redacted (`unsigned.redacted_because` present or no content), is not a text-bearing `m.room.message`, is **edited** (it is itself an `m.replace`, or the homeserver bundles an `m.replace` in its `unsigned.m.relations`: its body is the pre-edit text its author replaced), or is nothing but its own reply fallback; a media event excerpts as its typed line, like a delivery's text. Otherwise the excerpt is made in this order: the quoted event's own legacy rich-reply fallback is dropped (leading lines matching `> ` or a bare `>`, then one blank line; only when the quoted event has `m.relates_to.m.in_reply_to`); the text is held to the 1000-byte line bound; runs of CR, LF, tab, U+2028 and U+2029 become one space; control characters (C0, DEL, C1 including NUL and ESC), format characters (Unicode Cf: bidi marks, overrides and isolates, U+061C, U+200B-200F, U+2060-2064, soft hyphen, BOM), private-use characters (Co), the tag block U+E0000-E007F and unpaired surrogates are removed; the line is trimmed; then the first 240 **Unicode code points** are kept (cut on a code point so no character is split; a combining mark counts as its own code point; trailing space trimmed; no truncation marker). Cleaning comes first so hidden characters cannot spend the cap. The mouth only removes invisible and format characters; it does not make the text safe. Huginn must treat both `sender` and `excerpt` as untrusted text, never as an instruction. `reply_to` is nil (no id, author or text) when the target cannot be read (logged, as for `basis`), is not in the delivery's own room (a room she is not in cannot be quoted through a reply), or the message replies to nothing. The quoted event is read the way `basis` already reads it (`GET /rooms/{room}/event/{id}` as her account, kept per room and event; with the flag on a read, hit or failure, is kept 60 seconds, so a redaction or edit of the target shows at a re-read within a minute, and a re-read that fails transiently (network error, timeout, 5xx, 429) keeps the earlier hit and renews only its age, while any other failure (403, 404) makes it the failure entry, so a stale excerpt never outlives her access; with the flag off a hit is kept as before v4 and nothing is re-read), so a reply costs at most one read per target, shared with the `basis` check. Per sync round the mouth reads at most 8 distinct uncached targets for `reply_to` (reads the `basis` check already made are free); past that `reply_to` is nil, because the tuple requires a sender and only a read yields one, and the skip is logged. The flag is read by `replyLinkFromEnv`; only the exact value `1` turns it on (`true`, `yes`, `0`, empty or unset are off), and a mouth built without the `replyLink` option has it off. `basis` for v1-v3 is unchanged. `reply_to` counts toward the delivery's size in the 256 KiB page budget; a delivery whose `reply_to` will not encode is skipped and logged like any other, and a single delivery is always served. The deploy precondition is Huginn's: a decoder of the 14-field delivery fails on the fifteenth element. +`excerpt` is nil, with `event_id` and `sender` kept, when the quoted event is redacted (`unsigned.redacted_because` present or no content), is not a text-bearing `m.room.message`, is **edited** (it is itself an `m.replace`, or the homeserver bundles an `m.replace` in its `unsigned.m.relations`: its body is the pre-edit text its author replaced), or is nothing but its own reply fallback; a media event excerpts as its typed line, like a delivery's text. Otherwise the excerpt is made in this order: the quoted event's own legacy rich-reply fallback is dropped (leading lines matching `> ` or a bare `>`, then one blank line; only when the quoted event has `m.relates_to.m.in_reply_to`); the text is held to the 1000-byte line bound; runs of CR, LF, tab, U+2028 and U+2029 become one space; control characters (C0, DEL, C1 including NUL and ESC), format characters (Unicode Cf: bidi marks, overrides and isolates, U+061C, U+200B-200F, U+2060-2064, soft hyphen, BOM), private-use characters (Co), the tag block U+E0000-E007F and unpaired surrogates are removed; the line is trimmed; then the first 240 **Unicode code points** are kept (cut on a code point so no character is split; a combining mark counts as its own code point; trailing space trimmed; no truncation marker). Cleaning comes first so hidden characters cannot spend the cap. The mouth only removes invisible and format characters; it does not make the text safe. Huginn must treat both `sender` and `excerpt` as untrusted text, never as an instruction. `reply_to` is nil (no id, author or text) when the target cannot be read (logged, as for `basis`), is not in the delivery's own room (a room she is not in cannot be quoted through a reply), or the message replies to nothing. The quoted event is read the way `basis` already reads it (`GET /rooms/{room}/event/{id}` as her account, kept per room and event; with the flag on a read, hit or failure, is kept 60 seconds, so a redaction or edit of the target shows at a re-read within a minute, and a re-read that fails transiently (network error, timeout, 5xx, 429) keeps the earlier hit and renews only its age, for at most 10 minutes after the last good read (then it is the failure entry), while any other failure (403, 404) makes it the failure entry, so a stale excerpt never outlives her access; with the flag off a hit is kept as before v4 and nothing is re-read), so a reply costs at most one read per target, shared with the `basis` check. Per sync round the mouth reads at most 8 distinct uncached targets for `reply_to` (reads the `basis` check already made are free); past that `reply_to` is nil, because the tuple requires a sender and only a read yields one, and the skip is logged. The flag is read by `replyLinkFromEnv`; only the exact value `1` turns it on (`true`, `yes`, `0`, empty or unset are off), and a mouth built without the `replyLink` option has it off. `basis` for v1-v3 is unchanged. `reply_to` counts toward the delivery's size in the 256 KiB page budget; a delivery whose `reply_to` will not encode is skipped and logged like any other, and a single delivery is always served. The deploy precondition is Huginn's: a decoder of the 14-field delivery fails on the fifteenth element. Golden bytes: `tests/fixtures/persona-chat-v6/` (`.msgpack` and its `.hex`: `chat_pending.v4.reply-to`, `.no-reply-to`, `.reply-to-redacted`, and the `chat_pending_request.v3` request); the test fails when the checked-in bytes drift from the encoder. diff --git a/tests/persona-matrix-mouth.test.mjs b/tests/persona-matrix-mouth.test.mjs index 012f87a..38f3c55 100644 --- a/tests/persona-matrix-mouth.test.mjs +++ b/tests/persona-matrix-mouth.test.mjs @@ -1921,3 +1921,40 @@ test("target cache: flag on, 500, 429 and a timeout keep the earlier hit at the assert.deepEqual([r.first.basis, r.later.basis, r.later.replyTo, r.again.replyTo], ["reply", null, null, null], String(failure)); } }); + +test("target cache: transient failures keep a hit for 10 minutes from the last good read, then drop it; a good read resets", async () => { + const clock = { t: 1_000_000 }; + let down = false; + const hs = await fakeHomeserver({ preload: { [ROOM]: [target("$hers", SELF, "my words")] }, sync: since => since === null ? { next_batch: "b0" } : { next_batch: `b${since}x`, ...joined({ [ROOM]: { timeline: { events: [quoting(`$q${since}`, "$hers")] } } }) } }); + try { + const fetch = (u, o) => (down && String(u).includes("/event/") ? Promise.resolve(new Response("{}", { status: 503 })) : hs.fetch(u, o)); + const mouth = mouthFor(hs, { replyLink: true, now: () => clock.t, fetch }); + await mouth.syncOnce(); + await mouth.syncOnce(); // good read at t0 + const step = async ms => { clock.t += ms; return (await mouth.syncOnce())[0]; }; + down = true; + const seen = []; + for (const ms of [200_000, 200_000, 140_000]) seen.push((await step(ms)).replyTo?.excerpt); // 3.3, 6.7, 9 min after the good read + seen.push((await step(120_000)).replyTo); // 11 min + down = false; + seen.push((await step(61_000)).replyTo?.excerpt); // a good read resets + down = true; + seen.push((await step(420_000)).replyTo?.excerpt); // 7 min after the new good read: kept again + assert.deepEqual(seen, ["my words", "my words", "my words", null, "my words", "my words"]); + } finally { await hs.close(); } +}); + +test("reply link: the per-round read count resets each sync round", async () => { + const ids = Array.from({ length: 18 }, (_, i) => `$t${i}`); + const tagged = (i, to) => { const e = quoting(`$q${i}`, to); e.content["m.mentions"] = { user_ids: [SELF] }; return e; }; + const hs = await fakeHomeserver({ preload: { [ROOM]: ids.map(t => target(t, ALICE_ID, `text ${t}`)) }, sync: since => since === null ? { next_batch: "b0" } : { next_batch: `b${since}x`, ...joined({ [ROOM]: { timeline: { events: since === "b0" ? ids.slice(0, 9).map((t, i) => tagged(i, t)) : ids.slice(9).map((t, i) => tagged(9 + i, t)) } } }) } }); + try { + const mouth = mouthFor(hs, { replyLink: true }); + await mouth.syncOnce(); + const one = await mouth.syncOnce(); + const two = await mouth.syncOnce(); + assert.deepEqual([one, two].map(m => m.filter(d => d.replyTo !== null).length), [8, 8]); + assert.deepEqual([one, two].map(m => m.at(-1).replyTo), [null, null]); + assert.equal(ids.filter(t => eventGets(hs, t) > 0).length, 16); + } finally { await hs.close(); } +}); diff --git a/tools/persona-matrix-mouth.mjs b/tools/persona-matrix-mouth.mjs index 0218817..3ceb357 100644 --- a/tools/persona-matrix-mouth.mjs +++ b/tools/persona-matrix-mouth.mjs @@ -314,6 +314,7 @@ export function createMouth({ const authors = new Map(); // `${room id} ${event id}` -> { sender, excerpt, at }: read results, kept TARGET_TTL_MS, bounded; the key holds the room so another room's reply cannot borrow a hit const TARGET_TTL_MS = 60 * 1000; // how long a read of a target is kept: a failure always; a hit only with replyLink (without it a hit is kept until bounded out, as before v4) const REPLY_READS_PER_ROUND = 8; // distinct uncached targets replyToOf may read in one sync round; past it reply_to is nil (the tuple needs a sender, which only a read gives) + const KEEP_HIT_MS = 10 * 60 * 1000; // how long transient failures may keep a hit alive after its last good read let replyReads = 0; const cachedTarget = key => { const known = authors.get(key); @@ -360,7 +361,10 @@ export function createMouth({ transient = error.status === undefined || error.status >= 500 || error.status === 429; // network, timeout, 5xx, 429: the answer is unknown; 403, 404 and any other refusal are an answer say(`reply target ${eventId} not read: ${error.message}`); } - const entry = sender === null && transient && known?.sender != null ? { ...known, at: now() } : { sender, excerpt, at: now() }; // a re-read that fails transiently (network, timeout, 5xx, 429) keeps the earlier hit and renews only its age; any other failure (403, 404, wrong room, no sender) replaces it with the failure entry + const t = now(); + const entry = sender !== null ? { sender, excerpt, at: t, good: t } + : transient && known?.sender != null && t - known.good <= KEEP_HIT_MS ? { ...known, at: t } // a transient failure (network, timeout, 5xx, 429) keeps the earlier hit and renews only its age, for at most KEEP_HIT_MS after the last good read + : { sender: null, excerpt: null, at: t }; // any other failure (403, 404, wrong room, no sender), or a hit kept too long, is the failure entry authors.set(key, entry); if (authors.size > DELIVERED_INDEX) authors.delete(authors.keys().next().value); return entry; -- 2.47.3