docs/bridge.md

@@ -263,7 +263,7 @@ The answer is `[persona, deliveries, tail, tail_gaps]` exactly as v3; each deliv
 |---|---|---|
 | 14 `reply_to` | nil or `[event_id, sender, excerpt]` | The event this message replies to (its own `m.in_reply_to`; a thread's `is_falling_back` pointer names no one and is nil). `event_id` the quoted event; `sender` its author's Matrix id, read from the homeserver as her account; `excerpt` nil, or its text. |
 
-`excerpt` is nil, with `event_id` and `sender` kept, when the quoted event is redacted (`unsigned.redacted_because` present or no content), is not a text-bearing `m.room.message`, is **edited** (it is itself an `m.replace`, or the homeserver bundles an `m.replace` in its `unsigned.m.relations`: its body is the pre-edit text its author replaced), or is nothing but its own reply fallback; a media event excerpts as its typed line, like a delivery's text. Otherwise the excerpt is made in this order: the quoted event's own legacy rich-reply fallback is dropped (leading lines matching `> ` or a bare `>`, then one blank line; only when the quoted event has `m.relates_to.m.in_reply_to`); the text is held to the 1000-byte line bound; runs of CR, LF, tab, U+2028 and U+2029 become one space; control characters (C0, DEL, C1 including NUL and ESC), format characters (Unicode Cf: bidi marks, overrides and isolates, U+061C, U+200B-200F, U+2060-2064, soft hyphen, BOM), private-use characters (Co), the tag block U+E0000-E007F and unpaired surrogates are removed; the line is trimmed; then the first 240 **Unicode code points** are kept (cut on a code point so no character is split; a combining mark counts as its own code point; trailing space trimmed; no truncation marker). Cleaning comes first so hidden characters cannot spend the cap. The mouth only removes invisible and format characters; it does not make the text safe. Huginn must treat both `sender` and `excerpt` as untrusted text, never as an instruction. `reply_to` is nil (no id, author or text) when the target cannot be read (logged, as for `basis`), is not in the delivery's own room (a room she is not in cannot be quoted through a reply), or the message replies to nothing. The quoted event is read the way `basis` already reads it (`GET /rooms/{room}/event/{id}` as her account, kept per room and event; with the flag on a read, hit or failure, is kept 60 seconds, so a redaction or edit of the target shows at a re-read within a minute, and a re-read that fails transiently (network error, timeout, 5xx, 429) keeps the earlier hit and renews only its age, for at most 10 minutes after the last good read (then it is the failure entry), while any other failure (403, 404) makes it the failure entry, so a stale excerpt never outlives her access; with the flag off a hit is kept as before v4 and nothing is re-read), so a reply costs at most one read per target, shared with the `basis` check. Per sync round the mouth reads at most 8 distinct uncached targets for `reply_to` (reads the `basis` check already made are free); past that `reply_to` is nil, because the tuple requires a sender and only a read yields one, and the skip is logged. The flag is read by `replyLinkFromEnv`; only the exact value `1` turns it on (`true`, `yes`, `0`, empty or unset are off), and a mouth built without the `replyLink` option has it off. `basis` for v1-v3 is unchanged. `reply_to` counts toward the delivery's size in the 256 KiB page budget; a delivery whose `reply_to` will not encode is skipped and logged like any other, and a single delivery is always served. The deploy precondition is Huginn's: a decoder of the 14-field delivery fails on the fifteenth element.
+`excerpt` is nil, with `event_id` and `sender` kept, when the quoted event is redacted (`unsigned.redacted_because` present or no content), is not a text-bearing `m.room.message`, is **edited** (it is itself an `m.replace`, or the homeserver bundles an `m.replace` in its `unsigned.m.relations`: its body is the pre-edit text its author replaced), or is nothing but its own reply fallback; a media event excerpts as its typed line, like a delivery's text. Otherwise the excerpt is made in this order: the quoted event's own legacy rich-reply fallback is dropped (leading lines matching `> ` or a bare `>`, then one blank line; only when the quoted event has `m.relates_to.m.in_reply_to`); the text is held to the 1000-byte line bound; runs of CR, LF, tab, U+2028 and U+2029 become one space; control characters (C0, DEL, C1 including NUL and ESC), format characters (Unicode Cf: bidi marks, overrides and isolates, U+061C, U+200B-200F, U+2060-2064, soft hyphen, BOM), private-use characters (Co), the tag block U+E0000-E007F, variation selectors (U+FE00-FE0F, U+E0100-E01EF, U+180B-180F), the combining grapheme joiner U+034F, the Hangul and Khmer fillers (U+115F, U+1160, U+3164, U+FFA0, U+17B4, U+17B5), the blank braille cell U+2800, the unassigned default-ignorable ranges (U+2065, U+FFF0-FFF8, U+E0080-E0FFF) and unpaired surrogates are removed (cost accepted: an emoji loses its VS16 presentation, ❤️ becomes ❤, a keycap loses its selector; Mongolian free variation selectors and ideographic variation selectors go too); the line is trimmed; then the first 240 **Unicode code points** are kept (cut on a code point so no character is split; a combining mark counts as its own code point; trailing space trimmed; no truncation marker). Cleaning comes first so hidden characters cannot spend the cap. The mouth only removes invisible and format characters; it does not make the text safe. Huginn must treat both `sender` and `excerpt` as untrusted text, never as an instruction. `event_id` must be `$` plus at least one byte and `sender` `@` plus at least one byte, each at most 255 bytes and free of Unicode whitespace, control, format, private-use and default-ignorable characters and unpaired surrogates (a localpart with a zero-width joiner or non-joiner, as in Persian or Indic names, is therefore no `reply_to`); the encoder and the parser refuse any other, and the mouth makes a quoted event whose sender is not such an id a `reply_to` nil. `reply_to` is nil (no id, author or text) when the target cannot be read (logged, as for `basis`), is not in the delivery's own room (a room she is not in cannot be quoted through a reply), or the message replies to nothing. The quoted event is read the way `basis` already reads it (`GET /rooms/{room}/event/{id}` as her account, kept per room and event; with the flag on a read, hit or failure, is kept 60 seconds, so a redaction or edit of the target shows at a re-read within a minute, and a re-read that fails transiently (network error, timeout, 5xx, 429) keeps the earlier hit's sender and renews only its age, for at most 10 minutes after the last good read (then it is the failure entry), but never its excerpt (`excerpt` is nil with the sender kept: a re-read happens only after the 60-second keep, so the text is already as old as it may be), so a redaction or edit is not outlived by an outage, while any other failure (403, 404) makes it the failure entry, so a stale excerpt never outlives her access; with the flag off a hit is kept as before v4 and nothing is re-read), so a reply costs at most one read per target, shared with the `basis` check. Per sync round the mouth reads at most 8 distinct uncached targets for `reply_to` (reads the `basis` check already made are free); past that the delivery and every later event of the round wait: the round leaves the sync position where it is (as at the pending cap), the next round resumes at that event with its read count reset, and the wait is logged. No delivery goes out without a `reply_to` it should have had, and none is lost. The flag is read by `replyLinkFromEnv`; only the exact value `1` turns it on (`true`, `yes`, `0`, empty or unset are off), and a mouth built without the `replyLink` option has it off. `basis` for v1-v3 is unchanged. `reply_to` counts toward the delivery's size in the 256 KiB page budget; a delivery whose `reply_to` will not encode is skipped and logged like any other, and a single delivery is always served. The deploy precondition is Huginn's: a decoder of the 14-field delivery fails on the fifteenth element.
 
 Golden bytes: `tests/fixtures/persona-chat-v6/` (`.msgpack` and its `.hex`: `chat_pending.v4.reply-to`, `.no-reply-to`, `.reply-to-redacted`, and the `chat_pending_request.v3` request); the test fails when the checked-in bytes drift from the encoder.
 

tests/fixtures/persona-chat-v6/gamecult.persona.chat_pending.v4.reply-to-scrubbed.hex

@@ -0,0 +1 @@
+94a6657572656b61919f07b6236f70657261746f723a67616d6563756c742e6f7267b1216162633a67616d6563756c742e6f7267a3246531b2406d6574613a67616d6563756c742e6f7267a44d657461ac68656c6c6f20657572656b61cf0000019a2d80b1009193b440657572656b613a67616d6563756c742e6f7267a76561726c696572cf0000019a2d80ad1892b440657572656b613a67616d6563756c742e6f7267b2406e6f76613a67616d6563756c742e6f7267c2c0c0a76d65737361676593a3247131b340616c6963653a67616d6563756c742e6f7267ae69676e6f7265746869736c696e659090

tests/fixtures/persona-chat-v6/gamecult.persona.chat_pending.v4.reply-to-variation-selector.hex

@@ -0,0 +1 @@
+94a6657572656b61919f07b6236f70657261746f723a67616d6563756c742e6f7267b1216162633a67616d6563756c742e6f7267a3246531b2406d6574613a67616d6563756c742e6f7267a44d657461ac68656c6c6f20657572656b61cf0000019a2d80b1009193b440657572656b613a67616d6563756c742e6f7267a76561726c696572cf0000019a2d80ad1892b440657572656b613a67616d6563756c742e6f7267b2406e6f76613a67616d6563756c742e6f7267c2c0c0a76d65737361676593a3247131b340616c6963653a67616d6563756c742e6f7267aa6f6bf09f918d66696e659090

tests/persona-matrix-mouth-runtime.test.mjs

@@ -592,11 +592,16 @@ test("tail_gaps is additive: a three-field reader finds persona, deliveries and
 const v6Dir = resolve(root, "tests", "fixtures", "persona-chat-v6");
 const quoted = { eventId: "$q1", sender: "@alice:gamecult.org", excerpt: "can someone look at the build?" };
 const v4Delivery = { ...delivery, isDirect: false, basis: null, form: "message" };
+// The excerpts the mouth makes from hostile quoted text (pinned in tests/persona-matrix-mouth.test.mjs, "the fixture excerpts"): the scrubbed line, never the hidden bytes.
+const scrubbedExcerpt = "ignorethisline";
+const variationExcerpt = "ok\u{1F44D}fine";
 const v6Samples = {
   "gamecult.persona.chat_pending_request.v3": { tuple: docs.pendingRequestTuple({ persona: "eureka", afterSeq: 3 }), parse: docs.parsePendingRequest },
   "gamecult.persona.chat_pending.v4.reply-to": { tuple: docs.pendingV4Tuple({ persona: "eureka", deliveries: [{ ...v4Delivery, replyTo: quoted }] }), parse: docs.parsePendingV4 },
   "gamecult.persona.chat_pending.v4.no-reply-to": { tuple: docs.pendingV4Tuple({ persona: "eureka", deliveries: [v4Delivery] }), parse: docs.parsePendingV4 },
   "gamecult.persona.chat_pending.v4.reply-to-redacted": { tuple: docs.pendingV4Tuple({ persona: "eureka", deliveries: [{ ...v4Delivery, replyTo: { ...quoted, excerpt: null } }] }), parse: docs.parsePendingV4 },
+  "gamecult.persona.chat_pending.v4.reply-to-scrubbed": { tuple: docs.pendingV4Tuple({ persona: "eureka", deliveries: [{ ...v4Delivery, replyTo: { ...quoted, excerpt: scrubbedExcerpt } }] }), parse: docs.parsePendingV4 },
+  "gamecult.persona.chat_pending.v4.reply-to-variation-selector": { tuple: docs.pendingV4Tuple({ persona: "eureka", deliveries: [{ ...v4Delivery, replyTo: { ...quoted, excerpt: variationExcerpt } }] }), parse: docs.parsePendingV4 },
 };
 const hexOf = bytes => `${[...bytes].map(b => b.toString(16).padStart(2, "0")).join("")}\n`;
 for (const [name, { tuple, parse }] of Object.entries(v6Samples)) {
@@ -610,11 +615,22 @@ for (const [name, { tuple, parse }] of Object.entries(v6Samples)) {
       const decoded = msgpack.decode(bytes);
       assert.equal(decoded.length, 4, "persona, deliveries, tail, tail_gaps");
       assert.equal(decoded[1][0].length, 15, "the 14 v3 fields, then reply_to");
-      assert.deepEqual(decoded[1][0][14], name.endsWith("no-reply-to") ? null : [quoted.eventId, quoted.sender, name.endsWith("redacted") ? null : quoted.excerpt]);
+      assert.deepEqual(decoded[1][0][14], name.endsWith("no-reply-to") ? null : [quoted.eventId, quoted.sender, name.endsWith("redacted") ? null : name.endsWith("scrubbed") ? scrubbedExcerpt : name.endsWith("variation-selector") ? variationExcerpt : quoted.excerpt]);
     }
   });
 }
 
+test("documents: reply_to refuses an event id or sender that is not an id of at most 255 bytes, on encode and on parse", () => {
+  const ok = { eventId: "$q1", sender: "@alice:gamecult.org", excerpt: null };
+  const d = { ...delivery, isDirect: false, basis: null, form: "message" };
+  for (const bad of [{ eventId: "q1" }, { eventId: "$" }, { eventId: `$${"x".repeat(255)}` }, { eventId: "$a b" }, { sender: "alice" }, { sender: "@" }, { sender: `@${"é".repeat(128)}` }, { sender: "@a:b\u0085" }, { sender: "@a\u00a0b" }, { sender: "@a\u3164b" }, { sender: "@a\u2065b" }, { sender: "@a\u200db" }, { sender: "@a\ue000b" }, { sender: "@a\u007fb" }, { sender: "@a\u200bb" }, { sender: "@a\u202eb" }, { sender: "@a\ud800b" }, { eventId: "$a\u3000b" }])
+    assert.throws(() => docs.deliveryV4Tuple({ ...d, replyTo: { ...ok, ...bad } }), /reply_to/, JSON.stringify(bad).slice(0, 40));
+  assert.doesNotThrow(() => docs.deliveryV4Tuple({ ...d, replyTo: { ...ok, eventId: `$${"x".repeat(254)}` } }));
+  const tuple = docs.deliveryV4Tuple({ ...d, replyTo: ok });
+  tuple[14] = ["q1", ok.sender, null];
+  assert.throws(() => docs.parseDeliveryV4(tuple), /reply_to/);
+});
+
 test("v1, v2 and v3 bytes are unchanged by v4: a delivery without reply_to encodes to the v3 bytes plus one nil, and the older tuples never see the field", async () => {
   const v3 = msgpack.encode(docs.pendingV3Tuple({ persona: "eureka", deliveries: [v4Delivery] }));
   assert.deepEqual(docs.deliveryV3Tuple({ ...v4Delivery, replyTo: quoted }), docs.deliveryV3Tuple(v4Delivery), "reply_to never reaches a v3 tuple");

tests/persona-matrix-mouth.test.mjs

@@ -1877,10 +1877,11 @@ test("target cache: flag off, a hit is kept as before v4: no read after 60 s, ba
   assert.deepEqual([r.first.basis, r.later.basis, r.again.basis, r.gets], ["reply", "reply", "reply", 1]);
 });
 
-test("target cache: flag on, a failed re-read after 60 s keeps the earlier hit (basis and reply_to) and renews only its age", async () => {
+test("target cache: flag on, a failed re-read after 60 s keeps the earlier hit's basis and reply_to sender, not its excerpt, and renews only its age", async () => {
   const r = await replyAcrossTtl(true);
   assert.deepEqual([r.first.basis, r.later.basis, r.again.basis, r.gets], ["reply", "reply", "reply", 2]);
-  assert.deepEqual(r.later.replyTo, { eventId: "$hers", sender: SELF, excerpt: "my words" });
+  assert.deepEqual(r.later.replyTo, { eventId: "$hers", sender: SELF, excerpt: null });
+  assert.equal(r.again.replyTo.excerpt, null, "the renewed entry does not serve the old excerpt");
 });
 
 test("reply link: the excerpt drops Cf, Co, tag characters, U+061C, zero-width and word-joiner characters and unpaired surrogates", async () => {
@@ -1894,7 +1895,7 @@ test("reply link: the excerpt drops Cf, Co, tag characters, U+061C, zero-width a
   assert.deepEqual(made.map(d => d.replyTo.excerpt), ["ignore", "abcdef", "xyzw", "pqrst", "keep \u{1F600} é 中"]);
 });
 
-test("reply link: at most 8 uncached targets are read for reply_to per sync round; the rest are nil, and reads the basis already made are free", async () => {
+test("reply link: at most 8 uncached targets are read for reply_to per sync round; the rest wait for the next round, none is delivered without its reply_to or lost", async () => {
   const ids = Array.from({ length: 10 }, (_, i) => `$t${i}`);
   const tagged = (id, to) => quoting(id, to, {}); // below: mentions her, so basis does no read of its own
   const events = ids.map((t, i) => { const e = tagged(`$q${i}`, t); e.content["m.mentions"] = { user_ids: [SELF] }; return e; });
@@ -1903,18 +1904,21 @@ test("reply link: at most 8 uncached targets are read for reply_to per sync roun
     const mouth = mouthFor(hs, { replyLink: true });
     await mouth.syncOnce();
     const made = await mouth.syncOnce();
-    return { made, hs };
+    const next = await mouth.syncOnce();
+    return { made: [made, next], hs };
   })();
   try {
-    assert.equal(ids.filter(t => eventGets(hs, t) > 0).length, 8);
-    assert.deepEqual(made.map(d => d.replyTo === null), [false, false, false, false, false, false, false, false, true, true]);
+    assert.deepEqual(made.map(m => m.length), [8, 2]);
+    assert.deepEqual(made.flat().map(d => d.eventId), ids.map((_, i) => `$q${i}`));
+    assert.ok(made.flat().every(d => d.replyTo !== null));
+    assert.equal(ids.filter(t => eventGets(hs, t) > 0).length, 10);
   } finally { await hs.close(); }
 });
 
 test("target cache: flag on, 500, 429 and a timeout keep the earlier hit at the re-read; 403 and 404 drop it (reply_to nil, basis as the failure case)", async () => {
   for (const failure of [500, 429, "timeout"]) {
     const r = await replyAcrossTtl(true, failure);
-    assert.deepEqual([r.later.basis, r.later.replyTo?.excerpt], ["reply", "my words"], String(failure));
+    assert.deepEqual([r.later.basis, r.later.replyTo?.sender, r.later.replyTo?.excerpt], ["reply", SELF, null], String(failure));
   }
   for (const failure of [403, 404]) {
     const r = await replyAcrossTtl(true, failure);
@@ -1922,7 +1926,7 @@ test("target cache: flag on, 500, 429 and a timeout keep the earlier hit at the
   }
 });
 
-test("target cache: transient failures keep a hit for 10 minutes from the last good read, then drop it; a good read resets", async () => {
+test("target cache: transient failures keep a hit's sender for 10 minutes from the last good read and its excerpt for 60 s, then drop the hit; a good read resets", async () => {
   const clock = { t: 1_000_000 };
   let down = false;
   const hs = await fakeHomeserver({ preload: { [ROOM]: [target("$hers", SELF, "my words")] }, sync: since => since === null ? { next_batch: "b0" } : { next_batch: `b${since}x`, ...joined({ [ROOM]: { timeline: { events: [quoting(`$q${since}`, "$hers")] } } }) } });
@@ -1934,13 +1938,14 @@ test("target cache: transient failures keep a hit for 10 minutes from the last g
     const step = async ms => { clock.t += ms; return (await mouth.syncOnce())[0]; };
     down = true;
     const seen = [];
-    for (const ms of [200_000, 200_000, 140_000]) seen.push((await step(ms)).replyTo?.excerpt); // 3.3, 6.7, 9 min after the good read
+    for (const ms of [200_000, 200_000, 140_000]) seen.push((await step(ms)).replyTo?.excerpt); // 3.3, 6.7, 9 min after the good read: sender kept, excerpt gone
+    assert.equal((await step(0)).replyTo?.sender, SELF);
     seen.push((await step(120_000)).replyTo); // 11 min
     down = false;
     seen.push((await step(61_000)).replyTo?.excerpt); // a good read resets
     down = true;
-    seen.push((await step(420_000)).replyTo?.excerpt); // 7 min after the new good read: kept again
-    assert.deepEqual(seen, ["my words", "my words", "my words", null, "my words", "my words"]);
+    seen.push((await step(420_000)).replyTo?.sender); // 7 min after the new good read: the sender is kept again
+    assert.deepEqual(seen, [null, null, null, null, "my words", SELF]);
   } finally { await hs.close(); }
 });
 
@@ -1951,10 +1956,67 @@ test("reply link: the per-round read count resets each sync round", async () =>
   try {
     const mouth = mouthFor(hs, { replyLink: true });
     await mouth.syncOnce();
-    const one = await mouth.syncOnce();
-    const two = await mouth.syncOnce();
-    assert.deepEqual([one, two].map(m => m.filter(d => d.replyTo !== null).length), [8, 8]);
-    assert.deepEqual([one, two].map(m => m.at(-1).replyTo), [null, null]);
-    assert.equal(ids.filter(t => eventGets(hs, t) > 0).length, 16);
+    const rounds = [];
+    for (let i = 0; i < 4; i++) rounds.push(await mouth.syncOnce());
+    const all = rounds.flat();
+    assert.deepEqual(all.map(d => d.eventId), ids.map((_, i) => `$q${i}`), "every reply is delivered once, in order");
+    assert.ok(all.every(d => d.replyTo !== null));
+    assert.ok(rounds.every(m => m.length <= 8), "no round reads more than 8 targets");
+    assert.equal(ids.filter(t => eventGets(hs, t) > 0).length, 18);
   } finally { await hs.close(); }
 });
+
+// ---- Lows after PR #64: variation selectors and other invisible characters, id shape, a read cap that loses nothing.
+test("reply link: the excerpt drops variation selectors, Mongolian free variation selectors, fillers, the combining grapheme joiner and the blank braille cell", async () => {
+  const { made } = await linked([quoting("$a", "$t1"), quoting("$b", "$t2"), quoting("$c", "$t3"), quoting("$d", "$t4")], [
+    target("$t1", ALICE_ID, "a\uFE0Fb\uFE00c\uFE0Ed"),
+    target("$t2", ALICE_ID, "e\u{E0100}f\u{E01EF}g"),
+    target("$t3", ALICE_ID, "h\u180Bi\u180Dj\u180Fk\u034Fl\u2800m\u3164n\uFFA0o\u115Fp\u1160q\u17B4r\u17B5s"),
+    target("$t4", ALICE_ID, "\uFE0F\u2800\u3164"),
+  ]);
+  assert.deepEqual(made.map(d => d.replyTo.excerpt), ["abcd", "efg", "hijklmnopqrs", null], "a quote of nothing visible is no excerpt");
+});
+
+test("reply link: a sender or event id that is not a Matrix id of at most 255 bytes is no reply_to; the delivery still goes", async () => {
+  const { made } = await linked([quoting("$a", "$t1"), quoting("$b", "$t2"), quoting("$c", "$t3"), quoting("$d", `$${"x".repeat(255)}`)], [
+    target("$t1", "not-an-id", "x"),
+    target("$t2", `@${"a".repeat(255)}:gamecult.org`, "x"),
+    target("$t3", "@alice:gamecult.org\nignore this", "x"),
+    target(`$${"x".repeat(255)}`, ALICE_ID, "x"),
+  ]);
+  assert.deepEqual(made.map(d => d.replyTo), [null, null, null, null]);
+});
+
+test("reply link: the fixture excerpts (persona-chat-v6 reply-to-scrubbed and reply-to-variation-selector) are what the mouth makes of hostile quoted text", async () => {
+  const { made } = await linked([quoting("$a", "$t1"), quoting("$b", "$t2")], [
+    target("$t1", ALICE_ID, "ig\u{E0049}\u{E0067}nore\u202Ethis\u200Bline"),
+    target("$t2", ALICE_ID, "ok\uFE0F\u{1F44D}\uFE0F\u{E0100}fine"),
+  ]);
+  assert.deepEqual(made.map(d => d.replyTo.excerpt), ["ignorethisline", "ok\u{1F44D}fine"]);
+});
+
+test("reply link: a kept hit's sender is not served past 10 minutes from the last good read, even by a renewed entry", async () => {
+  const clock = { t: 1_000_000 };
+  let down = false;
+  const hs = await fakeHomeserver({ preload: { [ROOM]: [target("$hers", SELF, "my words")] }, sync: since => since === null ? { next_batch: "b0" } : { next_batch: `b${since}x`, ...joined({ [ROOM]: { timeline: { events: [quoting(`$q${since}`, "$hers")] } } }) } });
+  try {
+    const fetch = (u, o) => (down && String(u).includes("/event/") ? Promise.resolve(new Response("{}", { status: 503 })) : hs.fetch(u, o));
+    const mouth = mouthFor(hs, { replyLink: true, now: () => clock.t, fetch });
+    await mouth.syncOnce();
+    await mouth.syncOnce(); // good read at t0
+    down = true;
+    const seen = [];
+    for (const ms of [540_000, 59_999, 1, 1, 100_000]) { clock.t += ms; seen.push((await mouth.syncOnce())[0].replyTo?.sender ?? null); } // 9:00, 9:59.999, 10:00, 10:00.001, 11:40
+    assert.deepEqual(seen, [SELF, SELF, SELF, null, null]);
+  } finally { await hs.close(); }
+});
+
+test("reply link: the scrub removes U+2065, U+FFF0-FFF8, U+E0080-E00FF and U+E01F0-E0FFF", async () => {
+  const { made } = await linked([quoting("$a", "$t1"), quoting("$b", "$t2"), quoting("$c", "$t3"), quoting("$d", "$t4")], [
+    target("$t1", ALICE_ID, "a\u2065b"),
+    target("$t2", ALICE_ID, "c\uFFF0d\uFFF8e"),
+    target("$t3", ALICE_ID, "f\u{E0080}g\u{E00FF}h"),
+    target("$t4", ALICE_ID, "i\u{E01F0}j\u{E0FFF}k"),
+  ]);
+  assert.deepEqual(made.map(d => d.replyTo.excerpt), ["ab", "cde", "fgh", "ijk"]);
+});

tools/persona-chat-documents.mjs

@@ -227,9 +227,14 @@ export function parseDeliveryV4(t) {
   const fields = tuple(t, 15, "delivery_v4");
   return { ...parseDeliveryV3(fields.slice(0, 14)), replyTo: parseReplyTo(fields[14]) };
 }
+// A Matrix id as the tuple carries it: its sigil, a non-empty rest, at most 255 bytes, no Unicode whitespace, control, format or private-use characters and no unpaired surrogates.
+// The mouth applies the same test to what a homeserver returned, so a hostile id is no reply_to rather than a skipped delivery.
+export const isMatrixId = (sigil, v) => typeof v === "string" && Buffer.byteLength(v) <= 255 && v.length > 1 && v[0] === sigil && !/[\p{White_Space}\p{Cc}\p{Cf}\p{Co}\p{Default_Ignorable_Code_Point}\uD800-\uDFFF]/u.test(v);
 function replyTo(r) {
   if (r === null) return null;
   const { eventId, sender, excerpt = null } = r;
+  if (!isMatrixId("$", eventId)) throw new Error("reply_to event_id must be an event id ($..., at most 255 bytes)");
+  if (!isMatrixId("@", sender)) throw new Error("reply_to sender must be a user id (@..., at most 255 bytes)");
   if (excerpt !== null && (typeof excerpt !== "string" || [...excerpt].length > limits.replyExcerpt)) throw new Error(`reply_to excerpt must be a string of at most ${limits.replyExcerpt} code points, or nil`);
   return [text(eventId, "reply_to event_id"), text(sender, "reply_to sender"), excerpt];
 }

tools/persona-matrix-mouth.mjs

@@ -314,11 +314,13 @@ export function createMouth({
   const authors = new Map(); // `${room id} ${event id}` -> { sender, excerpt, at }: read results, kept TARGET_TTL_MS, bounded; the key holds the room so another room's reply cannot borrow a hit
   const TARGET_TTL_MS = 60 * 1000; // how long a read of a target is kept: a failure always; a hit only with replyLink (without it a hit is kept until bounded out, as before v4)
   const REPLY_READS_PER_ROUND = 8; // distinct uncached targets replyToOf may read in one sync round; past it reply_to is nil (the tuple needs a sender, which only a read gives)
-  const KEEP_HIT_MS = 10 * 60 * 1000; // how long transient failures may keep a hit alive after its last good read
+  const KEEP_HIT_MS = 10 * 60 * 1000; // how long transient failures may keep a hit's sender alive after its last good read
   let replyReads = 0;
   const cachedTarget = key => {
     const known = authors.get(key);
-    return known && (now() - known.at < TARGET_TTL_MS || (!replyLink && known.sender !== null)) ? known : null;
+    if (!known) return null;
+    if (known.good !== undefined && now() - known.good > KEEP_HIT_MS) return null; // a renewed hit is never served past KEEP_HIT_MS after its last good read, whatever its age
+    return now() - known.at < TARGET_TTL_MS || (!replyLink && known.sender !== null) ? known : null;
   };
   // The quoted text, as a single line (invisible and format characters removed; still untrusted text): nil when the event is redacted (the homeserver strips its content), is not text, or is edited
   // (an edit target, or one with an m.replace bundled: the original body is the pre-edit text, which its author replaced), or is nothing but
@@ -340,7 +342,7 @@ export function createMouth({
     }
     const line = bodyText({ ...c, body }, docs.limits.contextBytes)
       .replace(/[\r\n\t\u2028\u2029]+/g, " ")
-      .replace(/[\u0000-\u001f\u007f-\u009f\u061c\u200b-\u200f\u202a-\u202e\u2060-\u2064\u2066-\u2069]|[\p{Cf}\p{Co}\u{E0000}-\u{E007F}]|[\uD800-\uDFFF]/gu, "")
+      .replace(/[\u0000-\u001f\u007f-\u009f\u061c\u200b-\u200f\u202a-\u202e\u2060-\u2064\u2066-\u2069\u034f\u115f\u1160\u17b4\u17b5\u180b-\u180f\u2065\u2800\u3164\ufe00-\ufe0f\uffa0\ufff0-\ufff8]|[\p{Cf}\p{Co}\u{E0000}-\u{E007F}\u{E0080}-\u{E00FF}\u{E0100}-\u{E0FFF}]|[\uD800-\uDFFF]/gu, "")
       .trim();
     const cut = [...line].slice(0, docs.limits.replyExcerpt).join("").trimEnd();
     return cut === "" ? null : cut;
@@ -354,7 +356,7 @@ export function createMouth({
     try {
       const target = await matrix("GET", `/rooms/${encodeURIComponent(roomId)}/event/${encodeURIComponent(eventId)}`, undefined, API, AbortSignal.timeout(sendTimeoutMs));
       if (target?.room_id === undefined || target.room_id === roomId) { // an id from another room is not an answer about this one: no sender, no text
-        sender = typeof target?.sender === "string" && target.sender !== "" ? target.sender : null;
+        sender = docs.isMatrixId("@", target?.sender) ? target.sender : null;
         if (sender !== null) excerpt = excerptOf(target);
       }
     } catch (error) {
@@ -363,7 +365,7 @@ export function createMouth({
     }
     const t = now();
     const entry = sender !== null ? { sender, excerpt, at: t, good: t }
-      : transient && known?.sender != null && t - known.good <= KEEP_HIT_MS ? { ...known, at: t } // a transient failure (network, timeout, 5xx, 429) keeps the earlier hit and renews only its age, for at most KEEP_HIT_MS after the last good read
+      : transient && known?.sender != null && t - known.good <= KEEP_HIT_MS ? { ...known, excerpt: null, at: t } // a transient failure (network, timeout, 5xx, 429) keeps the earlier hit's sender and renews only its age, for at most KEEP_HIT_MS after the last good read; never its excerpt (a re-read happens only after TARGET_TTL_MS, so the excerpt is already as old as a good read may be, and a renewed entry must not serve it again)
       : { sender: null, excerpt: null, at: t }; // any other failure (403, 404, wrong room, no sender), or a hit kept too long, is the failure entry
     authors.set(key, entry);
     if (authors.size > DELIVERED_INDEX) authors.delete(authors.keys().next().value);
@@ -372,17 +374,18 @@ export function createMouth({
   const authorOf = async (roomId, eventId) => (await targetOf(roomId, eventId)).sender;
   // The event a message replies to: its own m.in_reply_to, unless that is a thread's fallback pointer (is_falling_back), which
   // names no one. nil when the target cannot be read or sits in another room.
+  const DEFER = Symbol("defer"); // replyToOf's answer when the round's read cap is spent: the delivery waits for the next round, as at the pending cap
   async function replyToOf(roomId, event) {
     if (!replyLink || formOf(event.content) !== "message") return null;
     const r = event.content["m.relates_to"]?.["m.in_reply_to"];
     const id = r?.event_id;
     if (typeof id !== "string" || id === "" || event.content["m.relates_to"]["is_falling_back"] === true) return null;
     if (!cachedTarget(`${roomId} ${id}`)) {
-      if (replyReads >= REPLY_READS_PER_ROUND) { say(`reply target ${id} not read: round cap`); return null; }
+      if (replyReads >= REPLY_READS_PER_ROUND) { say(`reply target ${id} not read: round cap, the rest of the round waits for the next`); return DEFER; }
       replyReads++;
     }
     const { sender, excerpt } = await targetOf(roomId, id);
-    return sender === null ? null : { eventId: id, sender, excerpt };
+    return sender === null || !docs.isMatrixId("$", id) ? null : { eventId: id, sender, excerpt };
   }
   async function repliesToMe(roomId, relates) {
     const targets = new Set([relates?.["m.in_reply_to"]?.event_id, relates?.rel_type === "m.thread" ? relates.event_id : undefined].filter(id => typeof id === "string" && id !== ""));
@@ -453,14 +456,15 @@ export function createMouth({
         const atUnixMs = Number.isSafeInteger(ts) && ts >= 0 ? ts : now();
         const deliverable = !baseline && !freshRooms.has(roomId) && event.sender !== selfId;
         if (deliverable && pending.length + made.length >= maxPending) { truncated = true; break rounds; }
+        const basis = deliverable ? await basisOf(roomId, event) : null;
+        const replyTo = deliverable ? await replyToOf(roomId, event) : null;
+        if (replyTo === DEFER) { truncated = true; break rounds; } // not processed: the next round meets this event again, with its read count reset
         processed.add(event.event_id);
         if (deliverable) {
           const context = await contextFor(roomId, event, round);
           const { image = null, refused = null } = await imageOf(roomId, event.content);
           if (refused !== null) say(`image refused: ${refused}`);
           const mentioned = event.content["m.mentions"]?.user_ids;
-          const basis = await basisOf(roomId, event);
-          const replyTo = await replyToOf(roomId, event);
           made.push({
             seq: ++seqCounter, roomAlias: alias, roomId, eventId: event.event_id, sender: event.sender,
             senderDisplay: displayNames.get(event.sender) ?? localpart(event.sender), text: bodyText(event.content, docs.limits.textBytes, refused), atUnixMs,