Mouth: chat_pending.v4 carries the quoted event of a reply (flag, default off)

Each v4 delivery is the 14 v3 fields plus reply_to = nil or [event_id, sender, excerpt]:
the quoted author's Matrix id and the first 240 code points of its text, nil excerpt for
a redacted or non-text event, nil reply_to for an unreadable or foreign-room target.
Asked by chat_pending_request.v3, served only with CHAT_PENDING_V4=1; off, that kind is
refused as any unknown kind. v1-v3 bytes unchanged. Golden bytes in persona-chat-v6.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
This commit is contained in:
Heimdall 2026-11-11 15:54:53 +00:00
commit a2568fcd2a
13 changed files with 280 additions and 18 deletions

View file

@ -253,6 +253,20 @@ A line a delivery already shows and a room she cannot speak in change nothing he
Golden bytes: `tests/fixtures/persona-chat-v5/` (`chat_pending.v3.tail` carries two lines and no gap, `chat_pending.v3.tail-gaps` one line and two gap rooms; the earlier v3 samples gain an empty fourth element); Huginn pins the same bytes.
### Reply link (`chat_pending.v4`)
A reply to anyone but the Persona arrives bare in v3: `basis` is `reply` only when the quoted author is her account. v4 adds what the message quotes. **It is off unless the mouth runs with `CHAT_PENDING_V4=1`** (exactly `1`; default off), because Huginn pins the tuple bytes and must read v4 first. With the flag off, `gamecult.persona.chat_pending_request.v3` is not a served kind: it is refused as any unknown kind is (`chat_refusal` `UnknownKind`, logged), byte for byte, and v1/v2/v3 requests are answered as before. With it on, that request kind (the v1 tuple `[persona, after_seq]` again; the kind asked by names the answer) is answered with `gamecult.persona.chat_pending.v4`.
The answer is `[persona, deliveries, tail, tail_gaps]` exactly as v3; each delivery is the 14 v3 fields, then field 14:
| Field | Type | Meaning |
|---|---|---|
| 14 `reply_to` | nil or `[event_id, sender, excerpt]` | The event this message replies to (its own `m.in_reply_to`; a thread's `is_falling_back` pointer names no one and is nil). `event_id` the quoted event; `sender` its author's Matrix id, read from the homeserver as her account; `excerpt` nil, or its text. |
`excerpt` is nil, with `event_id` and `sender` kept, when the quoted event is redacted (`unsigned.redacted_because` present or no content) or is not a text-bearing `m.room.message`; a media event excerpts as its typed line, like a delivery's text. Otherwise it is the first 240 **Unicode code points** of that text (after the 1000-byte line bound), cut on a code point so no character is split; a combining mark counts as its own code point; there is no truncation marker. `reply_to` is nil (no id, author or text) when the target cannot be read (logged, as for `basis`), is not in the delivery's own room (a room she is not in cannot be quoted through a reply), or the message replies to nothing. The quoted event is read the way `basis` already reads it (`GET /rooms/{room}/event/{id}` as her account, kept per room and event, a failure for a minute), so a reply costs at most one read per target, shared with the `basis` check. `basis` for v1-v3 is unchanged. `reply_to` counts toward the delivery's size in the 256 KiB page budget; a delivery whose `reply_to` will not encode is skipped and logged like any other, and a single delivery is always served. The deploy precondition is Huginn's: a decoder of the 14-field delivery fails on the fifteenth element.
Golden bytes: `tests/fixtures/persona-chat-v6/` (`.msgpack` and its `.hex`: `chat_pending.v4.reply-to`, `.no-reply-to`, `.reply-to-redacted`, and the `chat_pending_request.v3` request); the test fails when the checked-in bytes drift from the encoder.
Bifrost admits an observation only when guild, channel, target runtime, canonical repository, Persona, producer, and producer runtime match a preconfigured `bifrost.persona_feedback.target_binding.v0` document in Bifrost's private provider store. That binding also owns source visibility, data classification, and the delivery-store target. Bifrost never infers public visibility from Discord identifiers. It classifies the author using its own `bifrost.discord.actor_link.v0` state: a match yields `linked_governance_feedback`; absence yields `unlinked_social_feedback`. Neither class grants adoption, work, release, or deployment authority.

View file

@ -0,0 +1 @@
94a6657572656b61919f07b6236f70657261746f723a67616d6563756c742e6f7267b1216162633a67616d6563756c742e6f7267a3246531b2406d6574613a67616d6563756c742e6f7267a44d657461ac68656c6c6f20657572656b61cf0000019a2d80b1009193b440657572656b613a67616d6563756c742e6f7267a76561726c696572cf0000019a2d80ad1892b440657572656b613a67616d6563756c742e6f7267b2406e6f76613a67616d6563756c742e6f7267c2c0c0a76d657373616765c09090

View file

@ -0,0 +1 @@
94a6657572656b61919f07b6236f70657261746f723a67616d6563756c742e6f7267b1216162633a67616d6563756c742e6f7267a3246531b2406d6574613a67616d6563756c742e6f7267a44d657461ac68656c6c6f20657572656b61cf0000019a2d80b1009193b440657572656b613a67616d6563756c742e6f7267a76561726c696572cf0000019a2d80ad1892b440657572656b613a67616d6563756c742e6f7267b2406e6f76613a67616d6563756c742e6f7267c2c0c0a76d65737361676593a3247131b340616c6963653a67616d6563756c742e6f7267c09090

View file

@ -0,0 +1 @@
94a6657572656b61919f07b6236f70657261746f723a67616d6563756c742e6f7267b1216162633a67616d6563756c742e6f7267a3246531b2406d6574613a67616d6563756c742e6f7267a44d657461ac68656c6c6f20657572656b61cf0000019a2d80b1009193b440657572656b613a67616d6563756c742e6f7267a76561726c696572cf0000019a2d80ad1892b440657572656b613a67616d6563756c742e6f7267b2406e6f76613a67616d6563756c742e6f7267c2c0c0a76d65737361676593a3247131b340616c6963653a67616d6563756c742e6f7267be63616e20736f6d656f6e65206c6f6f6b20617420746865206275696c643f9090

View file

@ -0,0 +1 @@
92a6657572656b6103

View file

@ -0,0 +1 @@
’¦eureka

View file

@ -587,3 +587,111 @@ test("tail_gaps is additive: a three-field reader finds persona, deliveries and
// Not claimed: a strict three-field decoder (rmp-serde into a three-field tuple) accepts it. It fails on the length, as it did for the tail, so Huginn defaults tail_gaps before this mouth deploys.
assert.notEqual(withGaps.length, 3);
});
// ---- chat_pending.v4: reply_to (docs/bridge.md, "Reply link"). Behind a flag, default off.
const v6Dir = resolve(root, "tests", "fixtures", "persona-chat-v6");
const quoted = { eventId: "$q1", sender: "@alice:gamecult.org", excerpt: "can someone look at the build?" };
const v4Delivery = { ...delivery, isDirect: false, basis: null, form: "message" };
const v6Samples = {
"gamecult.persona.chat_pending_request.v3": { tuple: docs.pendingRequestTuple({ persona: "eureka", afterSeq: 3 }), parse: docs.parsePendingRequest },
"gamecult.persona.chat_pending.v4.reply-to": { tuple: docs.pendingV4Tuple({ persona: "eureka", deliveries: [{ ...v4Delivery, replyTo: quoted }] }), parse: docs.parsePendingV4 },
"gamecult.persona.chat_pending.v4.no-reply-to": { tuple: docs.pendingV4Tuple({ persona: "eureka", deliveries: [v4Delivery] }), parse: docs.parsePendingV4 },
"gamecult.persona.chat_pending.v4.reply-to-redacted": { tuple: docs.pendingV4Tuple({ persona: "eureka", deliveries: [{ ...v4Delivery, replyTo: { ...quoted, excerpt: null } }] }), parse: docs.parsePendingV4 },
};
const hexOf = bytes => `${[...bytes].map(b => b.toString(16).padStart(2, "0")).join("")}\n`;
for (const [name, { tuple, parse }] of Object.entries(v6Samples)) {
test(`golden fixture: ${name} is byte-stable (msgpack and hex) and parses back`, async () => {
const bytes = Buffer.from(msgpack.encode(tuple));
if (process.env.UPDATE_FIXTURES === "1") { await mkdir(v6Dir, { recursive: true }); await writeFile(resolve(v6Dir, `${name}.msgpack`), bytes); await writeFile(resolve(v6Dir, `${name}.hex`), hexOf(bytes)); }
assert.deepEqual(bytes, await readFile(resolve(v6Dir, `${name}.msgpack`)), "wire bytes changed: Huginn decodes these fixtures");
assert.equal(await readFile(resolve(v6Dir, `${name}.hex`), "utf8"), hexOf(bytes), "the hex dump is the msgpack file");
assert.deepEqual(parse(msgpack.decode(bytes)), parse(tuple));
if (name.includes("pending.v4")) {
const decoded = msgpack.decode(bytes);
assert.equal(decoded.length, 4, "persona, deliveries, tail, tail_gaps");
assert.equal(decoded[1][0].length, 15, "the 14 v3 fields, then reply_to");
assert.deepEqual(decoded[1][0][14], name.endsWith("no-reply-to") ? null : [quoted.eventId, quoted.sender, name.endsWith("redacted") ? null : quoted.excerpt]);
}
});
}
test("v1, v2 and v3 bytes are unchanged by v4: a delivery without reply_to encodes to the v3 bytes plus one nil, and the older tuples never see the field", async () => {
const v3 = msgpack.encode(docs.pendingV3Tuple({ persona: "eureka", deliveries: [v4Delivery] }));
assert.deepEqual(docs.deliveryV3Tuple({ ...v4Delivery, replyTo: quoted }), docs.deliveryV3Tuple(v4Delivery), "reply_to never reaches a v3 tuple");
assert.deepEqual(docs.deliveryTuple({ ...v4Delivery, replyTo: quoted }), docs.deliveryTuple(v4Delivery), "nor a v2 tuple");
assert.deepEqual(docs.deliveryV4Tuple(v4Delivery).slice(0, 14), docs.deliveryV3Tuple(v4Delivery));
assert.equal(msgpack.decode(v3)[1][0].length, 14);
const golden = msgpack.decode(await readFile(resolve(v5Dir, "gamecult.persona.chat_pending.v3.unproven-notice.msgpack")));
assert.equal(golden[1][0].length, 14, "the pinned v3 fixture stays 14 fields");
});
test("reply_to tuple: an excerpt over 240 code points, an empty id or sender, or a bad shape is refused; the cap counts code points, not bytes", () => {
const ok = n => docs.deliveryV4Tuple({ ...v4Delivery, replyTo: { ...quoted, excerpt: "\u{1F600}".repeat(n) } });
assert.equal([...ok(240)[14][2]].length, 240, "240 astral characters (960 bytes) fit");
assert.throws(() => ok(241), /at most 240 code points/);
assert.throws(() => docs.deliveryV4Tuple({ ...v4Delivery, replyTo: { ...quoted, sender: "" } }), /reply_to sender/);
assert.throws(() => docs.parseDeliveryV4([...docs.deliveryV3Tuple(v4Delivery), ["$q"]]), /3-field/);
assert.throws(() => docs.parseDeliveryV4(docs.deliveryV3Tuple(v4Delivery)), /15-field/);
});
test("v4 paging: reply_to counts toward each delivery's size, and one oversize or malformed delivery cannot break the page", () => {
const mk = (n, replyTo, body = "ok") => ({ ...v4Delivery, seq: n, eventId: `$e${n}`, text: body, replyTo });
const logs = [];
const page = pageOf("eureka", [mk(1, quoted), mk(2, { ...quoted, excerpt: "x".repeat(241) }), mk(3, null)], msgpack, l => logs.push(l), true, undefined, true);
assert.deepEqual(page[1].map(t => t[0]), [1, 3], "the delivery whose reply_to will not encode is skipped, logged, and the rest served");
assert.ok(logs.some(l => l.includes("delivery 2 skipped")));
const size = msgpack.encode(pageOf("eureka", [mk(1, quoted)], msgpack, () => {}, true, undefined, true)[1][0]).length;
const bare = msgpack.encode(pageOf("eureka", [mk(1, null)], msgpack, () => {}, true, undefined, true)[1][0]).length;
assert.ok(size > bare + quoted.excerpt.length, "the quote is part of the delivery's size");
const many = Array.from({ length: 400 }, (_, i) => mk(i + 1, { ...quoted, excerpt: "y".repeat(240) }, "x".repeat(3900)));
const paged = pageOf("eureka", many, msgpack, () => {}, true, undefined, true);
assert.ok(msgpack.encode(paged).length <= docs.limits.replyBytes + 1000 && paged[1].length > 1 && paged[1].length < 400, "quotes count toward the 256 KiB budget");
assert.equal(pageOf("eureka", [mk(1, quoted, "x".repeat(4000))], msgpack, () => {}, true, undefined, true)[1].length, 1, "a single delivery is always served");
assert.equal(docs.parsePendingV3(pageOf("eureka", [mk(1, quoted)], msgpack, () => {}, true))?.deliveries.length, 1, "asked as v3, the same delivery is served without reply_to");
assert.equal(pageOf("eureka", [mk(1, quoted)], msgpack, () => {}, true)[1][0].length, 14);
});
// Dial a throwaway endpoint, send one request and return the first document answered.
async function askOnce({ replyLink, kind, tuple }) {
const mouth = { pendingAfter: () => [{ ...v4Delivery, replyTo: quoted }], tail: async () => ({ lines: [], gaps: [], rooms: [] }), speakableRooms: () => [] };
const key = "connection-key-v4";
const logs = [];
const server = await startChatEndpoint({ mouths: new Map([["eureka", mouth]]), runtime, port: 0, connectionKey: key, runtimeId: "mouth-v4", log: l => logs.push(l), ...(replyLink === undefined ? {} : { replyLink }) });
const client = dgram.createSocket("udp4");
await new Promise(ok => client.bind(0, "127.0.0.1", ok));
const connection = new runtime.cultnet.CultNetRudpSocketTransportConnection({ runtimeId: "huginn-v4", socket: client, mode: "client", remoteHost: "127.0.0.1", remotePort: server.address.port, connectionId: connectionIdFor(key), initialSequence: 1, resendDelayMs: 50, resendPollMs: 10, maxFragmentBytes: 1200 });
try {
const got = new Promise((ok, fail) => { const t = setTimeout(() => fail(new Error(`no reply; logged ${logs.join(" | ")}`)), 5000); connection.on("frame", v => { clearTimeout(t); ok(runtime.cultnet.parseCultNetMessage(msgpack.decode(v.payload), "cultnet.schema.v0").document); }); });
connection.connect();
for (let i = 0; i < 100 && !connection.connected; i++) await new Promise(r => setTimeout(r, 20));
connection.send("schema", msgpack.encode(runtime.cultnet.encodeCultNetMessageForWire(docs.envelope({ kind, recordKey: "eureka", payload: msgpack.encode(tuple), sourceRuntimeId: "huginn-v4", sourceAgentId: "eureka", sourceRole: "persona-organ" }), "cultnet.schema.v0")));
return { document: await got, logs };
} finally { server.close(); connection.close(); try { client.close(); } catch { /* closed */ } }
}
test("v4 flag off (the default): a chat_pending_request.v3 is answered exactly as any unknown kind is, and the v3 kind is unchanged", async () => {
const request = docs.pendingRequestTuple({ persona: "eureka", afterSeq: 0 });
for (const flag of [undefined, false]) {
const { document, logs } = await askOnce({ replyLink: flag, kind: docs.kinds.pendingRequestV4, tuple: request });
assert.deepEqual([document.schemaId, docs.parseRefusal(msgpack.decode(document.payload))], [docs.kinds.refusal, { persona: "eureka", refusal: "UnknownKind" }]);
assert.ok(logs.some(l => l.includes("unknown kind: gamecult.persona.chat_pending_request.v3")));
const same = await askOnce({ replyLink: flag, kind: "gamecult.persona.chat_pending_request.v9", tuple: request });
assert.deepEqual(Buffer.from(same.document.payload), Buffer.from(document.payload), "byte for byte the answer to a kind that never existed");
}
const v3 = await askOnce({ replyLink: false, kind: docs.kinds.pendingRequestV3, tuple: request });
assert.equal(v3.document.schemaId, docs.kinds.pendingV3);
assert.equal(msgpack.decode(v3.document.payload)[1][0].length, 14, "a v3 request never gets reply_to, flag or no");
});
test("v4 flag on: the v3 request kind is answered with chat_pending.v4 over RUDP, reply_to in field 14 of each delivery; v1 and v3 requests are unchanged", async () => {
const request = docs.pendingRequestTuple({ persona: "eureka", afterSeq: 0 });
const { document } = await askOnce({ replyLink: true, kind: docs.kinds.pendingRequestV4, tuple: request });
assert.equal(document.schemaId, docs.kinds.pendingV4);
const wire = msgpack.decode(document.payload);
assert.deepEqual(wire[1][0][14], [quoted.eventId, quoted.sender, quoted.excerpt]);
assert.equal(docs.parsePendingV4(wire).deliveries[0].replyTo.sender, "@alice:gamecult.org");
const v3 = await askOnce({ replyLink: true, kind: docs.kinds.pendingRequestV3, tuple: request });
assert.equal(msgpack.decode(v3.document.payload)[1][0].length, 14);
const v1 = await askOnce({ replyLink: true, kind: docs.kinds.pendingRequest, tuple: request });
assert.equal(msgpack.decode(v1.document.payload)[1][0].length, 12);
});

View file

@ -1721,3 +1721,76 @@ test("tail gaps: the read timer is cleared when the read answers, and a mouth th
assert.ok([...made].every(t => cleared.has(t)), "no read timer is left running");
} finally { globalThis.setTimeout = realSet; globalThis.clearTimeout = realClear; await hs.close(); }
});
// ---- Reply link (chat_pending.v4, docs/bridge.md "Reply link"): the quoted event's id, author and excerpt, behind replyLink (default off).
const ALICE_ID = `@alice:${HS_NAME}`;
const quoting = (id, to, extra = {}) => withContent(id, META, { body: "re", "m.relates_to": { "m.in_reply_to": { event_id: to }, ...extra } });
const target = (id, sender, body, extra = {}) => ({ ...withContent(id, sender, { body }, 1000), room_id: ROOM, ...extra });
const quotes = made => made.map(d => [d.eventId, d.basis, d.replyTo]);
async function linked(events, preload, replyLink = true) {
const hs = await fakeHomeserver({ preload: { [ROOM]: preload }, sync: since => since === null ? { next_batch: "b0" } : { next_batch: "b1", ...joined({ [ROOM]: { timeline: { events } } }) } });
try {
const mouth = mouthFor(hs, { replyLink });
await mouth.syncOnce();
return { made: await mouth.syncOnce(), hs };
} finally { await hs.close(); }
}
test("reply link: a reply to anyone carries the quoted id, the author's Matrix id and the text; basis stays reply only for her own words", async () => {
const { made } = await linked(
[quoting("$a", "$hers"), quoting("$b", "$alices"), withContent("$c", META, { body: "plain" })],
[target("$hers", SELF, "my words"), target("$alices", ALICE_ID, "can someone look at the build?")]);
assert.deepEqual(quotes(made), [
["$a", "reply", { eventId: "$hers", sender: SELF, excerpt: "my words" }],
["$b", null, { eventId: "$alices", sender: ALICE_ID, excerpt: "can someone look at the build?" }],
["$c", null, null],
]);
});
test("reply link: flag off (default) delivers no replyTo and the basis is what it always was", async () => {
const pre = [target("$hers", SELF, "my words"), target("$alices", ALICE_ID, "secret words")];
const { made } = await linked([quoting("$a", "$hers"), quoting("$b", "$alices")], pre, false);
assert.deepEqual(quotes(made), [["$a", "reply", null], ["$b", null, null]]);
});
test("reply link: the excerpt is cut at 240 code points, not bytes, and never splits a character", async () => {
const emoji = "\u{1F600}".repeat(300), ascii = "a".repeat(300), combining = "é".repeat(200);
const { made } = await linked([quoting("$e", "$t1"), quoting("$a", "$t2"), quoting("$c", "$t3"), quoting("$s", "$t4")],
[target("$t1", ALICE_ID, emoji), target("$t2", ALICE_ID, ascii), target("$t3", ALICE_ID, combining), target("$t4", ALICE_ID, "short")]);
const ex = made.map(d => d.replyTo.excerpt);
assert.deepEqual([...ex[0]].length, 240);
assert.equal(ex[0], "\u{1F600}".repeat(240), "240 astral characters, 960 bytes");
assert.equal(ex[1], "a".repeat(240));
assert.equal([...ex[2]].length, 240, "code points: a combining mark counts as one, as documented");
assert.equal(ex[3], "short");
assert.ok(ex.every(e => !e.includes("<22>")));
});
test("reply link: a redacted quoted event keeps its id and author but no text; so does a non-text one", async () => {
const { made } = await linked([quoting("$r", "$red"), quoting("$n", "$nobody"), quoting("$m", "$img")], [
target("$red", ALICE_ID, "the secret that was redacted", { content: {}, unsigned: { redacted_because: { type: "m.room.redaction" } } }),
target("$nobody", ALICE_ID, "", { content: {} }),
target("$img", ALICE_ID, "x", { type: "m.sticker", content: { body: "private caption" } }),
]);
assert.deepEqual(made.map(d => d.replyTo), [
{ eventId: "$red", sender: ALICE_ID, excerpt: null },
{ eventId: "$nobody", sender: ALICE_ID, excerpt: null },
{ eventId: "$img", sender: ALICE_ID, excerpt: null },
]);
const redactedStillText = await linked([quoting("$r", "$red")], [target("$red", ALICE_ID, "still has a body", { unsigned: { redacted_because: {} } })]);
assert.equal(redactedStillText.made[0].replyTo.excerpt, null, "a redaction marker wins even if a body is present");
});
test("reply link: a quoted event in a room she cannot read, an unreadable one and a thread fallback give no replyTo and no text", async () => {
const { made } = await linked(
[quoting("$other", "$foreign"), quoting("$gone", "$missing"), quoting("$th", "$alices", { rel_type: "m.thread", event_id: "$root", is_falling_back: true })],
[{ ...target("$foreign", ALICE_ID, "from another room"), room_id: "!elsewhere:gamecult.test" }, target("$alices", ALICE_ID, "fallback target")]);
assert.deepEqual(made.map(d => d.replyTo), [null, null, null]);
assert.ok(!JSON.stringify(made.map(d => d.replyTo)).includes("room") && made.every(d => d.replyTo === null), "no id, author or text of the foreign event travels");
});
test("reply link: the same quoted event is read once, and a lookup cache hit from the basis check serves the quote", async () => {
const { made, hs } = await linked([quoting("$a", "$hers"), quoting("$b", "$hers")], [target("$hers", SELF, "mine")]);
assert.deepEqual(made.map(d => d.replyTo?.excerpt), ["mine", "mine"]);
assert.equal(hs.seen.filter(r => r.path.includes("/event/")).length, 1);
});

View file

@ -11,6 +11,8 @@ export const kinds = Object.freeze({
pending: "gamecult.persona.chat_pending.v2",
pendingRequestV3: "gamecult.persona.chat_pending_request.v2", // the same tuple as v1; asking by this kind is asking for the v3 answer
pendingV3: "gamecult.persona.chat_pending.v3",
pendingRequestV4: "gamecult.persona.chat_pending_request.v3", // the same tuple again; served only when the mouth's reply-link flag is on, else an unknown kind
pendingV4: "gamecult.persona.chat_pending.v4",
ack: "gamecult.persona.chat_ack.v1",
read: "gamecult.persona.chat_read.v1",
sendRequest: "gamecult.persona.chat_send_request.v1",
@ -39,7 +41,7 @@ const maxContext = 20;
// Per-delivery bounds, applied where the mouth reads Matrix: the body is bounded like a send,
// a context line and a display name are smaller, and a reply is paged to a byte budget.
export const limits = Object.freeze({ textBytes: maxTextBytes, contextBytes: 1000, displayBytes: 256, mentions: 64, replyBytes: 256 * 1024, tailPerRoom: 20, tailTotal: 60 });
export const limits = Object.freeze({ textBytes: maxTextBytes, contextBytes: 1000, displayBytes: 256, mentions: 64, replyBytes: 256 * 1024, tailPerRoom: 20, tailTotal: 60, replyExcerpt: 240 });
// `value` cut to at most `maxBytes` of UTF-8 without splitting a character.
export function clip(value, maxBytes) {
@ -215,6 +217,28 @@ function form(v) {
return v;
}
// The v4 delivery: the 14 fields of v3, then reply_to: nil, or [event_id, sender, excerpt] for the event this one replies to.
// sender is the quoted author's Matrix id. excerpt is nil for a quoted event whose text must not travel (redacted, not text);
// otherwise its first replyExcerpt (240) Unicode code points. A quote the mouth could not establish is reply_to nil, never a guess.
export function deliveryV4Tuple(d) {
return [...deliveryV3Tuple(d), replyTo(d.replyTo ?? null)];
}
export function parseDeliveryV4(t) {
const fields = tuple(t, 15, "delivery_v4");
return { ...parseDeliveryV3(fields.slice(0, 14)), replyTo: parseReplyTo(fields[14]) };
}
function replyTo(r) {
if (r === null) return null;
const { eventId, sender, excerpt = null } = r;
if (excerpt !== null && (typeof excerpt !== "string" || [...excerpt].length > limits.replyExcerpt)) throw new Error(`reply_to excerpt must be a string of at most ${limits.replyExcerpt} code points, or nil`);
return [text(eventId, "reply_to event_id"), text(sender, "reply_to sender"), excerpt];
}
function parseReplyTo(t) {
if (t === null) return null;
const [eventId, sender, excerpt] = tuple(t, 3, "reply_to");
return { eventId: text(eventId, "reply_to event_id"), sender: text(sender, "reply_to sender"), excerpt: replyTo({ eventId, sender, excerpt })[2] };
}
export function parseDelivery(t) {
const [s, roomAlias, roomId, eventId, sender, senderDisplay, body, atUnixMs, context, mentions, isDirect, image] = tuple(t, 12, "delivery");
if (!Array.isArray(context)) throw new Error("delivery context must be a list");
@ -263,6 +287,18 @@ export function parsePendingV3(t) {
return { persona: text(persona, "persona"), deliveries: deliveries.map(parseDeliveryV3), tail: tail.map(parseTailLine), tailGaps: gaps };
}
// The v4 answer: the v3 answer with v4 deliveries.
export function pendingV4Tuple({ persona, deliveries, tail = [], tailGaps = [] }) {
const [p, , t, g] = pendingV3Tuple({ persona, deliveries: [], tail, tailGaps });
return [p, deliveries.map(deliveryV4Tuple), t, g];
}
export function parsePendingV4(t) {
const [persona, deliveries, tail, tailGaps] = tuple(t, 4, "chat_pending_v4");
if (!Array.isArray(deliveries)) throw new Error("chat_pending deliveries must be a list");
const rest = parsePendingV3([persona, [], tail, tailGaps]);
return { ...rest, deliveries: deliveries.map(parseDeliveryV4) };
}
// The CultNet document_put_raw envelope every chat_* document travels in.
export function envelope({ kind, recordKey, payload, sourceRuntimeId, sourceAgentId, sourceRole }) {
return {

View file

@ -122,6 +122,7 @@ export function createMouth({
persona, homeserver, serverName, rooms, token, fetch: httpFetch = globalThis.fetch, state, brake,
now = Date.now, log = () => {}, syncTimeoutMs = 30000, maxPending = MAX_PENDING, gate = createRoomGate(), sendTimeoutMs = SEND_TIMEOUT_MS,
presenceGraceMs = PRESENCE_GRACE_MS,
replyLink = false, // resolve each reply's quoted event into delivery.replyTo (the v4 answer; one more event read per reply to a non-Mind)
images = null, // this Persona's image store (createImageStore), or null: images stay the typed text line
isServed = async () => false, // (async) whether a Matrix user id is an account this process speaks for: a Mind is not a person to open a DM with
}) {
@ -310,22 +311,43 @@ export function createMouth({
}
// A pill is an anchor whose href is exactly her matrix.to link; a longer id, an attribute that merely contains one and a quoted reply do not count.
const pillFor = html => typeof html === "string" && [...html.replace(/<mx-reply>[\s\S]*?<\/mx-reply>/gi, "").matchAll(/<a\s[^>]*?href\s*=\s*(["'])(.*?)\1/gi)].some(([, , href]) => [selfId, encodeURIComponent(selfId)].some(id => href === `https://matrix.to/#/${id}`));
const authors = new Map(); // `${room id} ${event id}` -> { sender, at }: read results, a failure kept for NEGATIVE_MS, bounded; the key holds the room so another room's reply cannot borrow a hit
const authors = new Map(); // `${room id} ${event id}` -> { sender, excerpt, at }: read results, a failure kept for NEGATIVE_MS, bounded; the key holds the room so another room's reply cannot borrow a hit
const NEGATIVE_MS = 60 * 1000;
async function authorOf(roomId, eventId) {
// The quoted text: nil when the event is redacted (the homeserver strips its content) or is not text; else the first 240 code points
// (not bytes: a cut never splits a character), from the same body line a delivery shows. Computed only when replyLink is on.
const excerptOf = target => {
if (!replyLink || target?.unsigned?.redacted_because !== undefined || target?.type !== "m.room.message" || typeof target.content?.body !== "string") return null;
return [...bodyText(target.content, docs.limits.contextBytes)].slice(0, docs.limits.replyExcerpt).join("");
};
async function targetOf(roomId, eventId) {
const key = `${roomId} ${eventId}`;
const known = authors.get(key);
if (known && (known.sender !== null || now() - known.at < NEGATIVE_MS)) return known.sender;
let sender = null;
if (known && (known.sender !== null || now() - known.at < NEGATIVE_MS)) return known;
let sender = null, excerpt = null;
try {
const target = await matrix("GET", `/rooms/${encodeURIComponent(roomId)}/event/${encodeURIComponent(eventId)}`, undefined, API, AbortSignal.timeout(sendTimeoutMs));
if (target?.room_id === undefined || target.room_id === roomId) sender = typeof target?.sender === "string" ? target.sender : null; // an id from another room is not an answer about this one
if (target?.room_id === undefined || target.room_id === roomId) { // an id from another room is not an answer about this one: no sender, no text
sender = typeof target?.sender === "string" && target.sender !== "" ? target.sender : null;
if (sender !== null) excerpt = excerptOf(target);
}
} catch (error) {
say(`reply target ${eventId} not read: ${error.message}`);
}
authors.set(key, { sender, at: now() });
const entry = { sender, excerpt, at: now() };
authors.set(key, entry);
if (authors.size > DELIVERED_INDEX) authors.delete(authors.keys().next().value);
return sender;
return entry;
}
const authorOf = async (roomId, eventId) => (await targetOf(roomId, eventId)).sender;
// The event a message replies to: its own m.in_reply_to, unless that is a thread's fallback pointer (is_falling_back), which
// names no one. nil when the target cannot be read or sits in another room.
async function replyToOf(roomId, event) {
if (!replyLink || formOf(event.content) !== "message") return null;
const r = event.content["m.relates_to"]?.["m.in_reply_to"];
const id = r?.event_id;
if (typeof id !== "string" || id === "" || event.content["m.relates_to"]["is_falling_back"] === true) return null;
const { sender, excerpt } = await targetOf(roomId, id);
return sender === null ? null : { eventId: id, sender, excerpt };
}
async function repliesToMe(roomId, relates) {
const targets = new Set([relates?.["m.in_reply_to"]?.event_id, relates?.rel_type === "m.thread" ? relates.event_id : undefined].filter(id => typeof id === "string" && id !== ""));
@ -402,12 +424,13 @@ export function createMouth({
if (refused !== null) say(`image refused: ${refused}`);
const mentioned = event.content["m.mentions"]?.user_ids;
const basis = await basisOf(roomId, event);
const replyTo = await replyToOf(roomId, event);
made.push({
seq: ++seqCounter, roomAlias: alias, roomId, eventId: event.event_id, sender: event.sender,
senderDisplay: displayNames.get(event.sender) ?? localpart(event.sender), text: bodyText(event.content, docs.limits.textBytes, refused), atUnixMs,
context, image,
mentions: Array.isArray(mentioned) ? mentioned.filter(id => typeof id === "string" && id !== "" && id.length <= 255).slice(0, docs.limits.mentions) : [],
isDirect: directRooms.has(roomId), basis, form: formOf(event.content),
isDirect: directRooms.has(roomId), basis, form: formOf(event.content), replyTo,
});
if (image) images.hold(image.path, seqCounter);
}
@ -730,15 +753,16 @@ export function connectionIdFor(connectionKey) {
// The reply to a chat_pending_request: deliveries oldest first, each encoded on its own so one that
// will not encode is skipped and logged without silencing the rest, up to the reply byte budget.
// The caller acks what it has read and asks again.
// A v4 reply (v4 true, with v3) is the v3 reply with each delivery's reply_to counted in its size, like any field: one oversize delivery is still alone on its page.
// A v3 reply also carries `tail` (room lines, oldest first), trimmed oldest first to what the byte budget has left after the deliveries,
// and `tail_gaps`: the rooms of `tail.rooms`, in that order, with lines missing from the reply: those `tail.gaps` names plus every room
// whose line the byte budget or an unencodable line dropped here.
export function pageOf(persona, deliveries, msgpack, log = () => {}, v3 = false, tail = { lines: [], gaps: [], rooms: [] }) {
export function pageOf(persona, deliveries, msgpack, log = () => {}, v3 = false, tail = { lines: [], gaps: [], rooms: [] }, v4 = false) {
const page = [];
let bytes = 0;
for (const delivery of deliveries) {
let tuple, size;
try { tuple = (v3 ? docs.deliveryV3Tuple : docs.deliveryTuple)(delivery); size = msgpack.encode(tuple).length; } catch (error) {
try { tuple = (v4 ? docs.deliveryV4Tuple : v3 ? docs.deliveryV3Tuple : docs.deliveryTuple)(delivery); size = msgpack.encode(tuple).length; } catch (error) {
log(`delivery ${delivery?.seq} skipped: ${error?.message ?? error}`);
continue;
}
@ -769,7 +793,7 @@ export function pageOf(persona, deliveries, msgpack, log = () => {}, v3 = false,
// session, and a request is answered on the session it arrived on. Requests arrive as
// document_put_raw messages. A request naming a Persona this
// process does not serve is answered with a typed UnknownPersona refusal.
export async function startChatEndpoint({ mouths, runtime, port, connectionKey, runtimeId, socketFactory = createSocket, host = "127.0.0.1", log = () => {} }) {
export async function startChatEndpoint({ mouths, runtime, port, connectionKey, runtimeId, socketFactory = createSocket, host = "127.0.0.1", log = () => {}, replyLink = false }) {
if (!/^127\.\d+\.\d+\.\d+$/.test(host)) throw new Error("the chat endpoint listens on loopback only");
const { msgpack, cultnet } = runtime;
const socket = socketFactory("udp4");
@ -781,7 +805,7 @@ export async function startChatEndpoint({ mouths, runtime, port, connectionKey,
const respond = (peer, persona, kind, recordKey, tuple) => server.send(peer, "schema", msgpack.encode(cultnet.encodeCultNetMessageForWire(
docs.envelope({ kind, recordKey, payload: msgpack.encode(tuple), sourceRuntimeId: runtimeId, sourceAgentId: persona, sourceRole: "bifrost-persona-matrix-mouth" }), "cultnet.schema.v0")));
const refuseUnknown = (peer, persona) => respond(peer, persona, docs.kinds.refusal, persona, docs.refusalTuple({ persona, refusal: "UnknownPersona" }));
const served = [docs.kinds.pendingRequest, docs.kinds.pendingRequestV3, docs.kinds.ack, docs.kinds.read, docs.kinds.sendRequest, docs.kinds.sendRequestV2, docs.kinds.openDirectRequest];
const served = [docs.kinds.pendingRequest, docs.kinds.pendingRequestV3, ...(replyLink ? [docs.kinds.pendingRequestV4] : []), docs.kinds.ack, docs.kinds.read, docs.kinds.sendRequest, docs.kinds.sendRequestV2, docs.kinds.openDirectRequest];
const handle = async ({ peer, channelId, payload }) => {
try {
if (channelId !== "schema") throw new Error("chat request arrived on a foreign channel");
@ -798,13 +822,14 @@ export async function startChatEndpoint({ mouths, runtime, port, connectionKey,
const mouth = mouths.get(request.persona);
const receipt = mouth ? await mouth.openDirect(request) : { requestId: request.requestId, outcome: "UnknownPersona", roomId: null, eventId: null };
respond(peer, request.persona, docs.kinds.openDirectReceipt, receipt.requestId, docs.openDirectReceiptTuple(receipt));
} else if (document.schemaId === docs.kinds.pendingRequest || document.schemaId === docs.kinds.pendingRequestV3) {
const v3 = document.schemaId === docs.kinds.pendingRequestV3; // the kind asked by names the answer: v1 gets the 12-field v2 answer, as before
} else if (document.schemaId === docs.kinds.pendingRequest || document.schemaId === docs.kinds.pendingRequestV3 || document.schemaId === docs.kinds.pendingRequestV4) {
const v4 = document.schemaId === docs.kinds.pendingRequestV4; // only reachable with replyLink: `served` drops it otherwise
const v3 = v4 || document.schemaId === docs.kinds.pendingRequestV3; // the kind asked by names the answer: v1 gets the 12-field v2 answer, as before
const request = docs.parsePendingRequest(tuple);
const mouth = mouths.get(request.persona);
if (!mouth) return refuseUnknown(peer, request.persona);
const tail = v3 ? await mouth.tail().catch(error => { log(`tail not read: ${error?.message ?? error}`); const rooms = mouth.speakableRooms(); return { lines: [], gaps: rooms, rooms }; }) : undefined;
respond(peer, request.persona, v3 ? docs.kinds.pendingV3 : docs.kinds.pending, request.persona, pageOf(request.persona, mouth.pendingAfter(request.afterSeq), msgpack, log, v3, tail));
respond(peer, request.persona, v4 ? docs.kinds.pendingV4 : v3 ? docs.kinds.pendingV3 : docs.kinds.pending, request.persona, pageOf(request.persona, mouth.pendingAfter(request.afterSeq), msgpack, log, v3, tail, v4));
} else if (document.schemaId === docs.kinds.ack) {
const request = docs.parseAck(tuple);
const mouth = mouths.get(request.persona);
@ -903,10 +928,11 @@ export async function main(env = process.env) {
const controlRoot = env.CONTROL_ROOT ?? "/etc/gamecult/minds";
const imageRoot = env.IMAGE_STORE_ROOT ? resolve(env.IMAGE_STORE_ROOT) : null; // unset: chat images stay the typed text line
const gate = createRoomGate(); // one for every Persona: two Minds in one room take turns
const replyLink = env.CHAT_PENDING_V4 === "1"; // default off: Huginn pins the v3 tuple bytes until it reads v4
const mouths = new Map(personas.map(persona => {
mkdirSync(join(stateRoot, persona), { recursive: true });
return [persona, createMouth({
persona, homeserver, serverName, rooms: roomsByPersona.get(persona), token: tokens.get(persona), gate, isServed: async id => { await Promise.all([...mouths.values()].map(m => m.start().catch(() => {}))); return [...mouths.values()].some(m => m.userId === id); }, log: line => log(`${persona}: ${line}`),
persona, homeserver, serverName, rooms: roomsByPersona.get(persona), token: tokens.get(persona), gate, replyLink, isServed: async id => { await Promise.all([...mouths.values()].map(m => m.start().catch(() => {}))); return [...mouths.values()].some(m => m.userId === id); }, log: line => log(`${persona}: ${line}`),
images: imageRoot ? createImageStore({ root: join(imageRoot, persona), log: line => log(`${persona}: ${line}`) }) : null,
state: cultcacheState({ runtime, path: join(stateRoot, persona, "cursor.cc") }),
brake: createBrakeReader({ controlPath: join(controlRoot, persona, "control.cc"), readStore: runtime.readSingleFileStore, decode: runtime.msgpack.decode }),
@ -914,7 +940,7 @@ export async function main(env = process.env) {
}));
const listen = required(env, "CHAT_RUDP_LISTEN"); // host:port, loopback
const colon = listen.lastIndexOf(":");
await startChatEndpoint({ mouths, runtime, host: listen.slice(0, colon), port: Number(listen.slice(colon + 1)), connectionKey: chatKey, runtimeId: "bifrost-persona-mouth", log });
await startChatEndpoint({ mouths, runtime, host: listen.slice(0, colon), port: Number(listen.slice(colon + 1)), connectionKey: chatKey, runtimeId: "bifrost-persona-mouth", log, replyLink });
await Promise.all([...mouths.values()].map(mouth => mouth.run()));
});
}