GameCult/Bifrost
Watch
9
0
Fork
You've already forked Bifrost
 0
Code Issues 7 Pull requests 4 Projects Releases Packages Wiki Activity Actions

Reply link: no pre-edit text, 60 s target cache, fallback stripped, excerpt made prompt-safe

Browse source 
Soul findings on #63: an edited target excerpts as nil; a cached target is re-read after 60 s so a
redaction shows; the target's own rich-reply fallback is dropped; control and bidi characters are
removed and line breaks collapsed before the 240 cap; the env flag parser is exported and pinned.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
This commit is contained in:
Heimdall 2026-11-11 16:05:24 +00:00
commit bc4e2eb0b1
3 changed files with 91 additions and 11 deletions
Download patch file Download diff file Expand all files Collapse all files

4
docs/bridge.md
View file

@ -206,7 +206,7 @@ A client asks for the facts by the kind it polls with: `gamecult.persona.chat_pe
| Field | Type | Meaning |
|---|---|---|
| 12 `basis` | `"direct"`, `"tag"`, `"reply"`, `"room"` or nil | Why the message is for this Persona, by what Matrix says, first that holds: `direct` the room is one of her DMs; `tag` `m.mentions.user_ids` lists her id, or the client sent no `m.mentions` at all and an `<a href>` in the HTML body is exactly her `matrix.to` link; `reply` it replies to, or is in a thread under, an event her account wrote (her Self run speaks as her account, so its words count; any member's message in a thread under one of her events is a `reply` too, and the target is read once per room and event, a failure for a minute); `room` an `@room` mention. A `notice` or an `edit` costs no reply lookup. Known limit: lookups run one at a time inside the `/sync` round (10 s each, at most two per reply), so a slow homeserver slows the round; they are not parallelised or capped per round. |
| 12 `basis` | `"direct"`, `"tag"`, `"reply"`, `"room"` or nil | Why the message is for this Persona, by what Matrix says, first that holds: `direct` the room is one of her DMs; `tag` `m.mentions.user_ids` lists her id, or the client sent no `m.mentions` at all and an `<a href>` in the HTML body is exactly her `matrix.to` link; `reply` it replies to, or is in a thread under, an event her account wrote (her Self run speaks as her account, so its words count; any member's message in a thread under one of her events is a `reply` too, and the target is read once per room and event and kept for a minute, hit or failure); `room` an `@room` mention. A `notice` or an `edit` costs no reply lookup. Known limit: lookups run one at a time inside the `/sync` round (10 s each, at most two per reply), so a slow homeserver slows the round; they are not parallelised or capped per round. |
| 13 `form` | `"message"`, `"notice"` or `"edit"` | `m.notice` is an automated line; `m.replace` is an edit of an earlier message. Only a `message` can address anyone. |
What the mouth cannot know, and so what a nil `basis` means: **unproven, not "not for her"**. Her handle and public name in the text (any casing, with or without `@`, `'s`), the names her stewardships list, a project's repo name, and whether the sender is the operator or a Mind are all matched by Huginn, not here. A reply whose target the homeserver will not give is nil and logged. The mouth also cannot see whether a message was meant for her without naming her. Every delivery is still delivered, claimed tags included: nothing is dropped in the mouth, so the Huginn rule cannot lose a message the mouth had no way to judge.
@ -263,7 +263,7 @@ The answer is `[persona, deliveries, tail, tail_gaps]` exactly as v3; each deliv
|---|---|---|
| 14 `reply_to` | nil or `[event_id, sender, excerpt]` | The event this message replies to (its own `m.in_reply_to`; a thread's `is_falling_back` pointer names no one and is nil). `event_id` the quoted event; `sender` its author's Matrix id, read from the homeserver as her account; `excerpt` nil, or its text. |
`excerpt` is nil, with `event_id` and `sender` kept, when the quoted event is redacted (`unsigned.redacted_because` present or no content) or is not a text-bearing `m.room.message`; a media event excerpts as its typed line, like a delivery's text. Otherwise it is the first 240 **Unicode code points** of that text (after the 1000-byte line bound), cut on a code point so no character is split; a combining mark counts as its own code point; there is no truncation marker. `reply_to` is nil (no id, author or text) when the target cannot be read (logged, as for `basis`), is not in the delivery's own room (a room she is not in cannot be quoted through a reply), or the message replies to nothing. The quoted event is read the way `basis` already reads it (`GET /rooms/{room}/event/{id}` as her account, kept per room and event, a failure for a minute), so a reply costs at most one read per target, shared with the `basis` check. `basis` for v1-v3 is unchanged. `reply_to` counts toward the delivery's size in the 256 KiB page budget; a delivery whose `reply_to` will not encode is skipped and logged like any other, and a single delivery is always served. The deploy precondition is Huginn's: a decoder of the 14-field delivery fails on the fifteenth element.
`excerpt` is nil, with `event_id` and `sender` kept, when the quoted event is redacted (`unsigned.redacted_because` present or no content), is not a text-bearing `m.room.message`, is **edited** (it is itself an `m.replace`, or the homeserver bundles an `m.replace` in its `unsigned.m.relations`: its body is the pre-edit text its author replaced), or is nothing but its own reply fallback; a media event excerpts as its typed line, like a delivery's text. Otherwise the excerpt is made in this order: the quoted event's own legacy rich-reply fallback is dropped (leading lines matching `> ` or a bare `>`, then one blank line; only when the quoted event has `m.relates_to.m.in_reply_to`); the text is held to the 1000-byte line bound; runs of CR, LF, tab, U+2028 and U+2029 become one space; control characters (C0, DEL, C1 including NUL and ESC) and bidi marks, overrides and isolates (U+200E/F, U+202A-202E, U+2066-2069) are removed; the line is trimmed; then the first 240 **Unicode code points** are kept (cut on a code point so no character is split; a combining mark counts as its own code point; trailing space trimmed; no truncation marker). Cleaning comes first so hidden characters cannot spend the cap, and the excerpt is untrusted text for a prompt, never an instruction. `reply_to` is nil (no id, author or text) when the target cannot be read (logged, as for `basis`), is not in the delivery's own room (a room she is not in cannot be quoted through a reply), or the message replies to nothing. The quoted event is read the way `basis` already reads it (`GET /rooms/{room}/event/{id}` as her account, kept per room and event for 60 seconds, hit or failure, so a redaction or edit of the target shows at a re-read within a minute and a cached excerpt never outlives that), so a reply costs at most one read per target, shared with the `basis` check. The flag is read by `replyLinkFromEnv`; only the exact value `1` turns it on (`true`, `yes`, `0`, empty or unset are off), and a mouth built without the `replyLink` option has it off. `basis` for v1-v3 is unchanged. `reply_to` counts toward the delivery's size in the 256 KiB page budget; a delivery whose `reply_to` will not encode is skipped and logged like any other, and a single delivery is always served. The deploy precondition is Huginn's: a decoder of the 14-field delivery fails on the fifteenth element.
Golden bytes: `tests/fixtures/persona-chat-v6/` (`.msgpack` and its `.hex`: `chat_pending.v4.reply-to`, `.no-reply-to`, `.reply-to-redacted`, and the `chat_pending_request.v3` request); the test fails when the checked-in bytes drift from the encoder.

60
tests/persona-matrix-mouth.test.mjs
View file

@ -10,7 +10,7 @@ import { createImageStore, RETAIN_MS } from "../tools/persona-chat-images.mjs";
import { personaStoreRoot } from "./image-bytes.mjs";
import * as docs from "../tools/persona-chat-documents.mjs";
import { gif, png, svg } from "./image-bytes.mjs";
import { createMouth, pageOf, bodyText, createBrakeReader, guarded, redact, roomsFor, projectRoomsVariable } from "../tools/persona-matrix-mouth.mjs";
import { createMouth, replyLinkFromEnv, pageOf, bodyText, createBrakeReader, guarded, redact, roomsFor, projectRoomsVariable } from "../tools/persona-matrix-mouth.mjs";
const TOKEN = "syt_SECRET_TOKEN_9f3";
const HS_NAME = "gamecult.test";
@ -1794,3 +1794,61 @@ test("reply link: the same quoted event is read once, and a lookup cache hit fro
assert.deepEqual(made.map(d => d.replyTo?.excerpt), ["mine", "mine"]);
assert.equal(hs.seen.filter(r => r.path.includes("/event/")).length, 1);
});
test("reply link: an edited target is quoted without its pre-edit text, whether it is the edit or has one bundled; id and author stay", async () => {
const { made } = await linked([quoting("$a", "$bundled"), quoting("$b", "$edit")], [
target("$bundled", ALICE_ID, "old wording", { unsigned: { "m.relations": { "m.replace": { event_id: "$x" } } } }),
target("$edit", ALICE_ID, "* new wording", { content: { msgtype: "m.text", body: "* new wording", "m.relates_to": { rel_type: "m.replace", event_id: "$orig" } } }),
]);
assert.deepEqual(made.map(d => d.replyTo), [{ eventId: "$bundled", sender: ALICE_ID, excerpt: null }, { eventId: "$edit", sender: ALICE_ID, excerpt: null }]);
});
test("reply link: a cached target is re-read after 60 s, so a redaction or edit in between shows; inside 60 s the cached read stands", async () => {
const clock = { t: 1_000_000 };
const mine = target("$t", ALICE_ID, "before");
const hs = await fakeHomeserver({ preload: { [ROOM]: [mine] }, sync: since => since === null ? { next_batch: "b0" } : { next_batch: `b${since}x`, ...joined({ [ROOM]: { timeline: { events: [quoting(`$q${since}`, "$t")] } } }) } });
try {
const mouth = mouthFor(hs, { replyLink: true, now: () => clock.t });
await mouth.syncOnce();
const first = await mouth.syncOnce();
mine.content = {}; mine.unsigned = { redacted_because: {} };
clock.t += 59_000;
const within = await mouth.syncOnce();
clock.t += 2_000;
const after = await mouth.syncOnce();
assert.deepEqual([first, within, after].map(m => m[0].replyTo.excerpt), ["before", "before", null]);
} finally { await hs.close(); }
});
test("reply link: the target's own reply fallback is dropped before the cap; a body of only fallback has no excerpt", async () => {
const inReply = { "m.in_reply_to": { event_id: "$earlier" } };
const withFallback = body => ({ content: { msgtype: "m.text", body, "m.relates_to": inReply } });
const { made } = await linked([quoting("$a", "$f1"), quoting("$b", "$f2"), quoting("$c", "$f3"), quoting("$d", "$f4")], [
target("$f1", ALICE_ID, "", withFallback("> <@bob:gamecult.test> what was it\n> second line\n\nthe real answer")),
target("$f2", ALICE_ID, "", withFallback("> <@bob:gamecult.test> only a quote\n\n")),
target("$f3", ALICE_ID, "", { content: { msgtype: "m.text", body: "> a literal quote with no reply\n\nkept" } }),
target("$f4", ALICE_ID, "", withFallback(`> <@bob:gamecult.test> ${"q".repeat(500)}\n\n${"w".repeat(300)}`)),
]);
assert.deepEqual(made.map(d => d.replyTo.excerpt), ["the real answer", null, "> a literal quote with no reply kept", "w".repeat(240)]);
});
test("reply link: the excerpt is safe in a prompt: control characters, bidi marks and line breaks are gone before the cap", async () => {
const probe = "a\u0000b\u202ec\r\nd\u001b[31m";
const { made } = await linked([quoting("$p", "$t1"), quoting("$q", "$t2"), quoting("$r", "$t3")], [
target("$t1", ALICE_ID, probe),
target("$t2", ALICE_ID, " x\t\t y\n\n\nz\u200f\u2066w\u2069\u0085\u007f ok "),
target("$t3", ALICE_ID, `\u202e${"\u0000".repeat(500)}${"k".repeat(300)}`),
]);
assert.deepEqual(made.map(d => d.replyTo.excerpt), ["abc d[31m", "x y zw ok", "k".repeat(240)], "controls removed first, so they do not eat the 240");
});
test("reply link: a mouth built without the option has it off, and only the exact env value \"1\" turns it on", async () => {
const hs = await fakeHomeserver({ preload: { [ROOM]: [target("$t", ALICE_ID, "words")] }, sync: since => since === null ? { next_batch: "b0" } : { next_batch: "b1", ...joined({ [ROOM]: { timeline: { events: [quoting("$q", "$t")] } } }) } });
try {
const mouth = createMouth({ persona: "eureka", homeserver: hs.url, rooms: Object.keys(ROOMS), token: TOKEN, state: memoryState(), fetch: hs.fetch, serverName: HS_NAME, brake: async () => true, log: () => {}, syncTimeoutMs: 1 });
await mouth.syncOnce();
assert.equal((await mouth.syncOnce())[0].replyTo, null);
} finally { await hs.close(); }
assert.deepEqual(["1", "true", "yes", "0", "", " 1", undefined].map(v => replyLinkFromEnv({ CHAT_PENDING_V4: v })), [true, false, false, false, false, false, false]);
assert.equal(replyLinkFromEnv({}), false);
});

38
tools/persona-matrix-mouth.mjs
View file

@ -311,18 +311,37 @@ export function createMouth({
}
// A pill is an anchor whose href is exactly her matrix.to link; a longer id, an attribute that merely contains one and a quoted reply do not count.
const pillFor = html => typeof html === "string" && [...html.replace(/<mx-reply>[\s\S]*?<\/mx-reply>/gi, "").matchAll(/<a\s[^>]*?href\s*=\s*(["'])(.*?)\1/gi)].some(([, , href]) => [selfId, encodeURIComponent(selfId)].some(id => href === `https://matrix.to/#/${id}`));
const authors = new Map(); // `${room id} ${event id}` -> { sender, excerpt, at }: read results, a failure kept for NEGATIVE_MS, bounded; the key holds the room so another room's reply cannot borrow a hit
const NEGATIVE_MS = 60 * 1000;
// The quoted text: nil when the event is redacted (the homeserver strips its content) or is not text; else the first 240 code points
// (not bytes: a cut never splits a character), from the same body line a delivery shows. Computed only when replyLink is on.
const authors = new Map(); // `${room id} ${event id}` -> { sender, excerpt, at }: read results, kept TARGET_TTL_MS, bounded; the key holds the room so another room's reply cannot borrow a hit
const TARGET_TTL_MS = 60 * 1000; // how long a read of a target (hit or failure) is kept
// The quoted text, as a prompt-safe line: nil when the event is redacted (the homeserver strips its content), is not text, or is edited
// (an edit target, or one with an m.replace bundled: the original body is the pre-edit text, which its author replaced), or is nothing but
// its own reply fallback. Else: the legacy rich-reply fallback (leading "> " lines and the blank line after) is dropped, CR/LF/tab runs
// become one space, control characters (C0, DEL, C1) and bidi marks/overrides/isolates are removed, the result is trimmed, then cut to the
// first 240 code points (not bytes, so no character is split; a combining mark is its own code point) and trimmed again. Computed
// only when replyLink is on.
const FALLBACK_LINE = /^>( |$)/;
const excerptOf = target => {
if (!replyLink || target?.unsigned?.redacted_because !== undefined || target?.type !== "m.room.message" || typeof target.content?.body !== "string") return null;
return [...bodyText(target.content, docs.limits.contextBytes)].slice(0, docs.limits.replyExcerpt).join("");
const c = target?.content;
if (!replyLink || target?.unsigned?.redacted_because !== undefined || target?.type !== "m.room.message" || typeof c?.body !== "string") return null;
if (c["m.relates_to"]?.rel_type === "m.replace" || target.unsigned?.["m.relations"]?.["m.replace"] !== undefined) return null;
let body = c.body;
if (c["m.relates_to"]?.["m.in_reply_to"] !== undefined) {
const lines = body.split("\n");
while (lines.length > 0 && FALLBACK_LINE.test(lines[0])) lines.shift();
if (lines.length > 0 && lines[0] === "") lines.shift();
body = lines.join("\n");
}
const line = bodyText({ ...c, body }, docs.limits.contextBytes)
.replace(/[\r\n\t\u2028\u2029]+/g, " ")
.replace(/[\u0000-\u001f\u007f-\u009f\u200e\u200f\u202a-\u202e\u2066-\u2069]/g, "")
.trim();
const cut = [...line].slice(0, docs.limits.replyExcerpt).join("").trimEnd();
return cut === "" ? null : cut;
};
async function targetOf(roomId, eventId) {
const key = `${roomId} ${eventId}`;
const known = authors.get(key);
if (known && (known.sender !== null || now() - known.at < NEGATIVE_MS)) return known;
if (known && now() - known.at < TARGET_TTL_MS) return known; // short for hits too: a later redaction or edit of the target must show at a re-read
let sender = null, excerpt = null;
try {
const target = await matrix("GET", `/rooms/${encodeURIComponent(roomId)}/event/${encodeURIComponent(eventId)}`, undefined, API, AbortSignal.timeout(sendTimeoutMs));
@ -881,6 +900,9 @@ export function loadRuntime(cultRoot) {
return { CultCache: cc.CultCache, SingleFileMessagePackBackingStore: cc.SingleFileMessagePackBackingStore, defineDocumentType: cc.defineDocumentType, readSingleFileStore: cc.readSingleFileStore, cultnet: cn, msgpack: mp };
}
// The v4 flag: on only for exactly "1"; unset, "0", "true", "yes" and anything else are off.
export const replyLinkFromEnv = env => env.CHAT_PENDING_V4 === "1";
function required(env, name) {
if (!env[name]) throw new Error(`${name} is required`);
return env[name];
@ -928,7 +950,7 @@ export async function main(env = process.env) {
const controlRoot = env.CONTROL_ROOT ?? "/etc/gamecult/minds";
const imageRoot = env.IMAGE_STORE_ROOT ? resolve(env.IMAGE_STORE_ROOT) : null; // unset: chat images stay the typed text line
const gate = createRoomGate(); // one for every Persona: two Minds in one room take turns
const replyLink = env.CHAT_PENDING_V4 === "1"; // default off: Huginn pins the v3 tuple bytes until it reads v4
const replyLink = replyLinkFromEnv(env); // default off: Huginn pins the v3 tuple bytes until it reads v4
const mouths = new Map(personas.map(persona => {
mkdirSync(join(stateRoot, persona), { recursive: true });
return [persona, createMouth({