diff --git a/docs/bridge.md b/docs/bridge.md index e899b6c..a68d186 100644 --- a/docs/bridge.md +++ b/docs/bridge.md @@ -206,7 +206,7 @@ A client asks for the facts by the kind it polls with: `gamecult.persona.chat_pe | Field | Type | Meaning | |---|---|---| -| 12 `basis` | `"direct"`, `"tag"`, `"reply"`, `"room"` or nil | Why the message is for this Persona, by what Matrix says, first that holds: `direct` the room is one of her DMs; `tag` `m.mentions.user_ids` lists her id, or the client sent no `m.mentions` at all and an `` in the HTML body is exactly her `matrix.to` link; `reply` it replies to, or is in a thread under, an event her account wrote (her Self run speaks as her account, so its words count; any member's message in a thread under one of her events is a `reply` too, and the target is read once per room and event, a failure for a minute); `room` an `@room` mention. A `notice` or an `edit` costs no reply lookup. Known limit: lookups run one at a time inside the `/sync` round (10 s each, at most two per reply), so a slow homeserver slows the round; they are not parallelised or capped per round. | +| 12 `basis` | `"direct"`, `"tag"`, `"reply"`, `"room"` or nil | Why the message is for this Persona, by what Matrix says, first that holds: `direct` the room is one of her DMs; `tag` `m.mentions.user_ids` lists her id, or the client sent no `m.mentions` at all and an `` in the HTML body is exactly her `matrix.to` link; `reply` it replies to, or is in a thread under, an event her account wrote (her Self run speaks as her account, so its words count; any member's message in a thread under one of her events is a `reply` too, and the target is read once per room and event and kept for a minute, hit or failure); `room` an `@room` mention. A `notice` or an `edit` costs no reply lookup. Known limit: lookups run one at a time inside the `/sync` round (10 s each, at most two per reply), so a slow homeserver slows the round; they are not parallelised or capped per round. | | 13 `form` | `"message"`, `"notice"` or `"edit"` | `m.notice` is an automated line; `m.replace` is an edit of an earlier message. Only a `message` can address anyone. | What the mouth cannot know, and so what a nil `basis` means: **unproven, not "not for her"**. Her handle and public name in the text (any casing, with or without `@`, `'s`), the names her stewardships list, a project's repo name, and whether the sender is the operator or a Mind are all matched by Huginn, not here. A reply whose target the homeserver will not give is nil and logged. The mouth also cannot see whether a message was meant for her without naming her. Every delivery is still delivered, claimed tags included: nothing is dropped in the mouth, so the Huginn rule cannot lose a message the mouth had no way to judge. @@ -253,6 +253,20 @@ A line a delivery already shows and a room she cannot speak in change nothing he Golden bytes: `tests/fixtures/persona-chat-v5/` (`chat_pending.v3.tail` carries two lines and no gap, `chat_pending.v3.tail-gaps` one line and two gap rooms; the earlier v3 samples gain an empty fourth element); Huginn pins the same bytes. +### Reply link (`chat_pending.v4`) + +A reply to anyone but the Persona arrives bare in v3: `basis` is `reply` only when the quoted author is her account. v4 adds what the message quotes. **It is off unless the mouth runs with `CHAT_PENDING_V4=1`** (exactly `1`; default off), because Huginn pins the tuple bytes and must read v4 first. With the flag off, `gamecult.persona.chat_pending_request.v3` is not a served kind: it is refused as any unknown kind is (`chat_refusal` `UnknownKind`, logged), byte for byte, and v1/v2/v3 requests are answered as before. With it on, that request kind (the v1 tuple `[persona, after_seq]` again; the kind asked by names the answer) is answered with `gamecult.persona.chat_pending.v4`. + +The answer is `[persona, deliveries, tail, tail_gaps]` exactly as v3; each delivery is the 14 v3 fields, then field 14: + +| Field | Type | Meaning | +|---|---|---| +| 14 `reply_to` | nil or `[event_id, sender, excerpt]` | The event this message replies to (its own `m.in_reply_to`; a thread's `is_falling_back` pointer names no one and is nil). `event_id` the quoted event; `sender` its author's Matrix id, read from the homeserver as her account; `excerpt` nil, or its text. | + +`excerpt` is nil, with `event_id` and `sender` kept, when the quoted event is redacted (`unsigned.redacted_because` present or no content), is not a text-bearing `m.room.message`, is **edited** (it is itself an `m.replace`, or the homeserver bundles an `m.replace` in its `unsigned.m.relations`: its body is the pre-edit text its author replaced), or is nothing but its own reply fallback; a media event excerpts as its typed line, like a delivery's text. Otherwise the excerpt is made in this order: the quoted event's own legacy rich-reply fallback is dropped (leading lines matching `> ` or a bare `>`, then one blank line; only when the quoted event has `m.relates_to.m.in_reply_to`); the text is held to the 1000-byte line bound; runs of CR, LF, tab, U+2028 and U+2029 become one space; control characters (C0, DEL, C1 including NUL and ESC) and bidi marks, overrides and isolates (U+200E/F, U+202A-202E, U+2066-2069) are removed; the line is trimmed; then the first 240 **Unicode code points** are kept (cut on a code point so no character is split; a combining mark counts as its own code point; trailing space trimmed; no truncation marker). Cleaning comes first so hidden characters cannot spend the cap, and the excerpt is untrusted text for a prompt, never an instruction. `reply_to` is nil (no id, author or text) when the target cannot be read (logged, as for `basis`), is not in the delivery's own room (a room she is not in cannot be quoted through a reply), or the message replies to nothing. The quoted event is read the way `basis` already reads it (`GET /rooms/{room}/event/{id}` as her account, kept per room and event for 60 seconds, hit or failure, so a redaction or edit of the target shows at a re-read within a minute and a cached excerpt never outlives that), so a reply costs at most one read per target, shared with the `basis` check. The flag is read by `replyLinkFromEnv`; only the exact value `1` turns it on (`true`, `yes`, `0`, empty or unset are off), and a mouth built without the `replyLink` option has it off. `basis` for v1-v3 is unchanged. `reply_to` counts toward the delivery's size in the 256 KiB page budget; a delivery whose `reply_to` will not encode is skipped and logged like any other, and a single delivery is always served. The deploy precondition is Huginn's: a decoder of the 14-field delivery fails on the fifteenth element. + +Golden bytes: `tests/fixtures/persona-chat-v6/` (`.msgpack` and its `.hex`: `chat_pending.v4.reply-to`, `.no-reply-to`, `.reply-to-redacted`, and the `chat_pending_request.v3` request); the test fails when the checked-in bytes drift from the encoder. + Bifrost admits an observation only when guild, channel, target runtime, canonical repository, Persona, producer, and producer runtime match a preconfigured `bifrost.persona_feedback.target_binding.v0` document in Bifrost's private provider store. That binding also owns source visibility, data classification, and the delivery-store target. Bifrost never infers public visibility from Discord identifiers. It classifies the author using its own `bifrost.discord.actor_link.v0` state: a match yields `linked_governance_feedback`; absence yields `unlinked_social_feedback`. Neither class grants adoption, work, release, or deployment authority. diff --git a/tests/fixtures/persona-chat-v6/gamecult.persona.chat_pending.v4.no-reply-to.hex b/tests/fixtures/persona-chat-v6/gamecult.persona.chat_pending.v4.no-reply-to.hex new file mode 100644 index 0000000..a0d45dd --- /dev/null +++ b/tests/fixtures/persona-chat-v6/gamecult.persona.chat_pending.v4.no-reply-to.hex @@ -0,0 +1 @@ +94a6657572656b61919f07b6236f70657261746f723a67616d6563756c742e6f7267b1216162633a67616d6563756c742e6f7267a3246531b2406d6574613a67616d6563756c742e6f7267a44d657461ac68656c6c6f20657572656b61cf0000019a2d80b1009193b440657572656b613a67616d6563756c742e6f7267a76561726c696572cf0000019a2d80ad1892b440657572656b613a67616d6563756c742e6f7267b2406e6f76613a67616d6563756c742e6f7267c2c0c0a76d657373616765c09090 diff --git a/tests/fixtures/persona-chat-v6/gamecult.persona.chat_pending.v4.no-reply-to.msgpack b/tests/fixtures/persona-chat-v6/gamecult.persona.chat_pending.v4.no-reply-to.msgpack new file mode 100644 index 0000000..169db8d Binary files /dev/null and b/tests/fixtures/persona-chat-v6/gamecult.persona.chat_pending.v4.no-reply-to.msgpack differ diff --git a/tests/fixtures/persona-chat-v6/gamecult.persona.chat_pending.v4.reply-to-redacted.hex b/tests/fixtures/persona-chat-v6/gamecult.persona.chat_pending.v4.reply-to-redacted.hex new file mode 100644 index 0000000..0b2da2c --- /dev/null +++ b/tests/fixtures/persona-chat-v6/gamecult.persona.chat_pending.v4.reply-to-redacted.hex @@ -0,0 +1 @@ +94a6657572656b61919f07b6236f70657261746f723a67616d6563756c742e6f7267b1216162633a67616d6563756c742e6f7267a3246531b2406d6574613a67616d6563756c742e6f7267a44d657461ac68656c6c6f20657572656b61cf0000019a2d80b1009193b440657572656b613a67616d6563756c742e6f7267a76561726c696572cf0000019a2d80ad1892b440657572656b613a67616d6563756c742e6f7267b2406e6f76613a67616d6563756c742e6f7267c2c0c0a76d65737361676593a3247131b340616c6963653a67616d6563756c742e6f7267c09090 diff --git a/tests/fixtures/persona-chat-v6/gamecult.persona.chat_pending.v4.reply-to-redacted.msgpack b/tests/fixtures/persona-chat-v6/gamecult.persona.chat_pending.v4.reply-to-redacted.msgpack new file mode 100644 index 0000000..dc3f351 Binary files /dev/null and b/tests/fixtures/persona-chat-v6/gamecult.persona.chat_pending.v4.reply-to-redacted.msgpack differ diff --git a/tests/fixtures/persona-chat-v6/gamecult.persona.chat_pending.v4.reply-to.hex b/tests/fixtures/persona-chat-v6/gamecult.persona.chat_pending.v4.reply-to.hex new file mode 100644 index 0000000..8f6304c --- /dev/null +++ b/tests/fixtures/persona-chat-v6/gamecult.persona.chat_pending.v4.reply-to.hex @@ -0,0 +1 @@ +94a6657572656b61919f07b6236f70657261746f723a67616d6563756c742e6f7267b1216162633a67616d6563756c742e6f7267a3246531b2406d6574613a67616d6563756c742e6f7267a44d657461ac68656c6c6f20657572656b61cf0000019a2d80b1009193b440657572656b613a67616d6563756c742e6f7267a76561726c696572cf0000019a2d80ad1892b440657572656b613a67616d6563756c742e6f7267b2406e6f76613a67616d6563756c742e6f7267c2c0c0a76d65737361676593a3247131b340616c6963653a67616d6563756c742e6f7267be63616e20736f6d656f6e65206c6f6f6b20617420746865206275696c643f9090 diff --git a/tests/fixtures/persona-chat-v6/gamecult.persona.chat_pending.v4.reply-to.msgpack b/tests/fixtures/persona-chat-v6/gamecult.persona.chat_pending.v4.reply-to.msgpack new file mode 100644 index 0000000..ef6ed11 Binary files /dev/null and b/tests/fixtures/persona-chat-v6/gamecult.persona.chat_pending.v4.reply-to.msgpack differ diff --git a/tests/fixtures/persona-chat-v6/gamecult.persona.chat_pending_request.v3.hex b/tests/fixtures/persona-chat-v6/gamecult.persona.chat_pending_request.v3.hex new file mode 100644 index 0000000..f9bfe7e --- /dev/null +++ b/tests/fixtures/persona-chat-v6/gamecult.persona.chat_pending_request.v3.hex @@ -0,0 +1 @@ +92a6657572656b6103 diff --git a/tests/fixtures/persona-chat-v6/gamecult.persona.chat_pending_request.v3.msgpack b/tests/fixtures/persona-chat-v6/gamecult.persona.chat_pending_request.v3.msgpack new file mode 100644 index 0000000..0c326f5 --- /dev/null +++ b/tests/fixtures/persona-chat-v6/gamecult.persona.chat_pending_request.v3.msgpack @@ -0,0 +1 @@ +eureka \ No newline at end of file diff --git a/tests/persona-matrix-mouth-runtime.test.mjs b/tests/persona-matrix-mouth-runtime.test.mjs index b9361cc..874aeb9 100644 --- a/tests/persona-matrix-mouth-runtime.test.mjs +++ b/tests/persona-matrix-mouth-runtime.test.mjs @@ -587,3 +587,111 @@ test("tail_gaps is additive: a three-field reader finds persona, deliveries and // Not claimed: a strict three-field decoder (rmp-serde into a three-field tuple) accepts it. It fails on the length, as it did for the tail, so Huginn defaults tail_gaps before this mouth deploys. assert.notEqual(withGaps.length, 3); }); + +// ---- chat_pending.v4: reply_to (docs/bridge.md, "Reply link"). Behind a flag, default off. +const v6Dir = resolve(root, "tests", "fixtures", "persona-chat-v6"); +const quoted = { eventId: "$q1", sender: "@alice:gamecult.org", excerpt: "can someone look at the build?" }; +const v4Delivery = { ...delivery, isDirect: false, basis: null, form: "message" }; +const v6Samples = { + "gamecult.persona.chat_pending_request.v3": { tuple: docs.pendingRequestTuple({ persona: "eureka", afterSeq: 3 }), parse: docs.parsePendingRequest }, + "gamecult.persona.chat_pending.v4.reply-to": { tuple: docs.pendingV4Tuple({ persona: "eureka", deliveries: [{ ...v4Delivery, replyTo: quoted }] }), parse: docs.parsePendingV4 }, + "gamecult.persona.chat_pending.v4.no-reply-to": { tuple: docs.pendingV4Tuple({ persona: "eureka", deliveries: [v4Delivery] }), parse: docs.parsePendingV4 }, + "gamecult.persona.chat_pending.v4.reply-to-redacted": { tuple: docs.pendingV4Tuple({ persona: "eureka", deliveries: [{ ...v4Delivery, replyTo: { ...quoted, excerpt: null } }] }), parse: docs.parsePendingV4 }, +}; +const hexOf = bytes => `${[...bytes].map(b => b.toString(16).padStart(2, "0")).join("")}\n`; +for (const [name, { tuple, parse }] of Object.entries(v6Samples)) { + test(`golden fixture: ${name} is byte-stable (msgpack and hex) and parses back`, async () => { + const bytes = Buffer.from(msgpack.encode(tuple)); + if (process.env.UPDATE_FIXTURES === "1") { await mkdir(v6Dir, { recursive: true }); await writeFile(resolve(v6Dir, `${name}.msgpack`), bytes); await writeFile(resolve(v6Dir, `${name}.hex`), hexOf(bytes)); } + assert.deepEqual(bytes, await readFile(resolve(v6Dir, `${name}.msgpack`)), "wire bytes changed: Huginn decodes these fixtures"); + assert.equal(await readFile(resolve(v6Dir, `${name}.hex`), "utf8"), hexOf(bytes), "the hex dump is the msgpack file"); + assert.deepEqual(parse(msgpack.decode(bytes)), parse(tuple)); + if (name.includes("pending.v4")) { + const decoded = msgpack.decode(bytes); + assert.equal(decoded.length, 4, "persona, deliveries, tail, tail_gaps"); + assert.equal(decoded[1][0].length, 15, "the 14 v3 fields, then reply_to"); + assert.deepEqual(decoded[1][0][14], name.endsWith("no-reply-to") ? null : [quoted.eventId, quoted.sender, name.endsWith("redacted") ? null : quoted.excerpt]); + } + }); +} + +test("v1, v2 and v3 bytes are unchanged by v4: a delivery without reply_to encodes to the v3 bytes plus one nil, and the older tuples never see the field", async () => { + const v3 = msgpack.encode(docs.pendingV3Tuple({ persona: "eureka", deliveries: [v4Delivery] })); + assert.deepEqual(docs.deliveryV3Tuple({ ...v4Delivery, replyTo: quoted }), docs.deliveryV3Tuple(v4Delivery), "reply_to never reaches a v3 tuple"); + assert.deepEqual(docs.deliveryTuple({ ...v4Delivery, replyTo: quoted }), docs.deliveryTuple(v4Delivery), "nor a v2 tuple"); + assert.deepEqual(docs.deliveryV4Tuple(v4Delivery).slice(0, 14), docs.deliveryV3Tuple(v4Delivery)); + assert.equal(msgpack.decode(v3)[1][0].length, 14); + const golden = msgpack.decode(await readFile(resolve(v5Dir, "gamecult.persona.chat_pending.v3.unproven-notice.msgpack"))); + assert.equal(golden[1][0].length, 14, "the pinned v3 fixture stays 14 fields"); +}); + +test("reply_to tuple: an excerpt over 240 code points, an empty id or sender, or a bad shape is refused; the cap counts code points, not bytes", () => { + const ok = n => docs.deliveryV4Tuple({ ...v4Delivery, replyTo: { ...quoted, excerpt: "\u{1F600}".repeat(n) } }); + assert.equal([...ok(240)[14][2]].length, 240, "240 astral characters (960 bytes) fit"); + assert.throws(() => ok(241), /at most 240 code points/); + assert.throws(() => docs.deliveryV4Tuple({ ...v4Delivery, replyTo: { ...quoted, sender: "" } }), /reply_to sender/); + assert.throws(() => docs.parseDeliveryV4([...docs.deliveryV3Tuple(v4Delivery), ["$q"]]), /3-field/); + assert.throws(() => docs.parseDeliveryV4(docs.deliveryV3Tuple(v4Delivery)), /15-field/); +}); + +test("v4 paging: reply_to counts toward each delivery's size, and one oversize or malformed delivery cannot break the page", () => { + const mk = (n, replyTo, body = "ok") => ({ ...v4Delivery, seq: n, eventId: `$e${n}`, text: body, replyTo }); + const logs = []; + const page = pageOf("eureka", [mk(1, quoted), mk(2, { ...quoted, excerpt: "x".repeat(241) }), mk(3, null)], msgpack, l => logs.push(l), true, undefined, true); + assert.deepEqual(page[1].map(t => t[0]), [1, 3], "the delivery whose reply_to will not encode is skipped, logged, and the rest served"); + assert.ok(logs.some(l => l.includes("delivery 2 skipped"))); + const size = msgpack.encode(pageOf("eureka", [mk(1, quoted)], msgpack, () => {}, true, undefined, true)[1][0]).length; + const bare = msgpack.encode(pageOf("eureka", [mk(1, null)], msgpack, () => {}, true, undefined, true)[1][0]).length; + assert.ok(size > bare + quoted.excerpt.length, "the quote is part of the delivery's size"); + const many = Array.from({ length: 400 }, (_, i) => mk(i + 1, { ...quoted, excerpt: "y".repeat(240) }, "x".repeat(3900))); + const paged = pageOf("eureka", many, msgpack, () => {}, true, undefined, true); + assert.ok(msgpack.encode(paged).length <= docs.limits.replyBytes + 1000 && paged[1].length > 1 && paged[1].length < 400, "quotes count toward the 256 KiB budget"); + assert.equal(pageOf("eureka", [mk(1, quoted, "x".repeat(4000))], msgpack, () => {}, true, undefined, true)[1].length, 1, "a single delivery is always served"); + assert.equal(docs.parsePendingV3(pageOf("eureka", [mk(1, quoted)], msgpack, () => {}, true))?.deliveries.length, 1, "asked as v3, the same delivery is served without reply_to"); + assert.equal(pageOf("eureka", [mk(1, quoted)], msgpack, () => {}, true)[1][0].length, 14); +}); + +// Dial a throwaway endpoint, send one request and return the first document answered. +async function askOnce({ replyLink, kind, tuple }) { + const mouth = { pendingAfter: () => [{ ...v4Delivery, replyTo: quoted }], tail: async () => ({ lines: [], gaps: [], rooms: [] }), speakableRooms: () => [] }; + const key = "connection-key-v4"; + const logs = []; + const server = await startChatEndpoint({ mouths: new Map([["eureka", mouth]]), runtime, port: 0, connectionKey: key, runtimeId: "mouth-v4", log: l => logs.push(l), ...(replyLink === undefined ? {} : { replyLink }) }); + const client = dgram.createSocket("udp4"); + await new Promise(ok => client.bind(0, "127.0.0.1", ok)); + const connection = new runtime.cultnet.CultNetRudpSocketTransportConnection({ runtimeId: "huginn-v4", socket: client, mode: "client", remoteHost: "127.0.0.1", remotePort: server.address.port, connectionId: connectionIdFor(key), initialSequence: 1, resendDelayMs: 50, resendPollMs: 10, maxFragmentBytes: 1200 }); + try { + const got = new Promise((ok, fail) => { const t = setTimeout(() => fail(new Error(`no reply; logged ${logs.join(" | ")}`)), 5000); connection.on("frame", v => { clearTimeout(t); ok(runtime.cultnet.parseCultNetMessage(msgpack.decode(v.payload), "cultnet.schema.v0").document); }); }); + connection.connect(); + for (let i = 0; i < 100 && !connection.connected; i++) await new Promise(r => setTimeout(r, 20)); + connection.send("schema", msgpack.encode(runtime.cultnet.encodeCultNetMessageForWire(docs.envelope({ kind, recordKey: "eureka", payload: msgpack.encode(tuple), sourceRuntimeId: "huginn-v4", sourceAgentId: "eureka", sourceRole: "persona-organ" }), "cultnet.schema.v0"))); + return { document: await got, logs }; + } finally { server.close(); connection.close(); try { client.close(); } catch { /* closed */ } } +} + +test("v4 flag off (the default): a chat_pending_request.v3 is answered exactly as any unknown kind is, and the v3 kind is unchanged", async () => { + const request = docs.pendingRequestTuple({ persona: "eureka", afterSeq: 0 }); + for (const flag of [undefined, false]) { + const { document, logs } = await askOnce({ replyLink: flag, kind: docs.kinds.pendingRequestV4, tuple: request }); + assert.deepEqual([document.schemaId, docs.parseRefusal(msgpack.decode(document.payload))], [docs.kinds.refusal, { persona: "eureka", refusal: "UnknownKind" }]); + assert.ok(logs.some(l => l.includes("unknown kind: gamecult.persona.chat_pending_request.v3"))); + const same = await askOnce({ replyLink: flag, kind: "gamecult.persona.chat_pending_request.v9", tuple: request }); + assert.deepEqual(Buffer.from(same.document.payload), Buffer.from(document.payload), "byte for byte the answer to a kind that never existed"); + } + const v3 = await askOnce({ replyLink: false, kind: docs.kinds.pendingRequestV3, tuple: request }); + assert.equal(v3.document.schemaId, docs.kinds.pendingV3); + assert.equal(msgpack.decode(v3.document.payload)[1][0].length, 14, "a v3 request never gets reply_to, flag or no"); +}); + +test("v4 flag on: the v3 request kind is answered with chat_pending.v4 over RUDP, reply_to in field 14 of each delivery; v1 and v3 requests are unchanged", async () => { + const request = docs.pendingRequestTuple({ persona: "eureka", afterSeq: 0 }); + const { document } = await askOnce({ replyLink: true, kind: docs.kinds.pendingRequestV4, tuple: request }); + assert.equal(document.schemaId, docs.kinds.pendingV4); + const wire = msgpack.decode(document.payload); + assert.deepEqual(wire[1][0][14], [quoted.eventId, quoted.sender, quoted.excerpt]); + assert.equal(docs.parsePendingV4(wire).deliveries[0].replyTo.sender, "@alice:gamecult.org"); + const v3 = await askOnce({ replyLink: true, kind: docs.kinds.pendingRequestV3, tuple: request }); + assert.equal(msgpack.decode(v3.document.payload)[1][0].length, 14); + const v1 = await askOnce({ replyLink: true, kind: docs.kinds.pendingRequest, tuple: request }); + assert.equal(msgpack.decode(v1.document.payload)[1][0].length, 12); +}); diff --git a/tests/persona-matrix-mouth.test.mjs b/tests/persona-matrix-mouth.test.mjs index b2ed814..49245ff 100644 --- a/tests/persona-matrix-mouth.test.mjs +++ b/tests/persona-matrix-mouth.test.mjs @@ -10,7 +10,7 @@ import { createImageStore, RETAIN_MS } from "../tools/persona-chat-images.mjs"; import { personaStoreRoot } from "./image-bytes.mjs"; import * as docs from "../tools/persona-chat-documents.mjs"; import { gif, png, svg } from "./image-bytes.mjs"; -import { createMouth, pageOf, bodyText, createBrakeReader, guarded, redact, roomsFor, projectRoomsVariable } from "../tools/persona-matrix-mouth.mjs"; +import { createMouth, replyLinkFromEnv, pageOf, bodyText, createBrakeReader, guarded, redact, roomsFor, projectRoomsVariable } from "../tools/persona-matrix-mouth.mjs"; const TOKEN = "syt_SECRET_TOKEN_9f3"; const HS_NAME = "gamecult.test"; @@ -1721,3 +1721,134 @@ test("tail gaps: the read timer is cleared when the read answers, and a mouth th assert.ok([...made].every(t => cleared.has(t)), "no read timer is left running"); } finally { globalThis.setTimeout = realSet; globalThis.clearTimeout = realClear; await hs.close(); } }); + +// ---- Reply link (chat_pending.v4, docs/bridge.md "Reply link"): the quoted event's id, author and excerpt, behind replyLink (default off). +const ALICE_ID = `@alice:${HS_NAME}`; +const quoting = (id, to, extra = {}) => withContent(id, META, { body: "re", "m.relates_to": { "m.in_reply_to": { event_id: to }, ...extra } }); +const target = (id, sender, body, extra = {}) => ({ ...withContent(id, sender, { body }, 1000), room_id: ROOM, ...extra }); +const quotes = made => made.map(d => [d.eventId, d.basis, d.replyTo]); +async function linked(events, preload, replyLink = true) { + const hs = await fakeHomeserver({ preload: { [ROOM]: preload }, sync: since => since === null ? { next_batch: "b0" } : { next_batch: "b1", ...joined({ [ROOM]: { timeline: { events } } }) } }); + try { + const mouth = mouthFor(hs, { replyLink }); + await mouth.syncOnce(); + return { made: await mouth.syncOnce(), hs }; + } finally { await hs.close(); } +} + +test("reply link: a reply to anyone carries the quoted id, the author's Matrix id and the text; basis stays reply only for her own words", async () => { + const { made } = await linked( + [quoting("$a", "$hers"), quoting("$b", "$alices"), withContent("$c", META, { body: "plain" })], + [target("$hers", SELF, "my words"), target("$alices", ALICE_ID, "can someone look at the build?")]); + assert.deepEqual(quotes(made), [ + ["$a", "reply", { eventId: "$hers", sender: SELF, excerpt: "my words" }], + ["$b", null, { eventId: "$alices", sender: ALICE_ID, excerpt: "can someone look at the build?" }], + ["$c", null, null], + ]); +}); + +test("reply link: flag off (default) delivers no replyTo and the basis is what it always was", async () => { + const pre = [target("$hers", SELF, "my words"), target("$alices", ALICE_ID, "secret words")]; + const { made } = await linked([quoting("$a", "$hers"), quoting("$b", "$alices")], pre, false); + assert.deepEqual(quotes(made), [["$a", "reply", null], ["$b", null, null]]); +}); + +test("reply link: the excerpt is cut at 240 code points, not bytes, and never splits a character", async () => { + const emoji = "\u{1F600}".repeat(300), ascii = "a".repeat(300), combining = "é".repeat(200); + const { made } = await linked([quoting("$e", "$t1"), quoting("$a", "$t2"), quoting("$c", "$t3"), quoting("$s", "$t4")], + [target("$t1", ALICE_ID, emoji), target("$t2", ALICE_ID, ascii), target("$t3", ALICE_ID, combining), target("$t4", ALICE_ID, "short")]); + const ex = made.map(d => d.replyTo.excerpt); + assert.deepEqual([...ex[0]].length, 240); + assert.equal(ex[0], "\u{1F600}".repeat(240), "240 astral characters, 960 bytes"); + assert.equal(ex[1], "a".repeat(240)); + assert.equal([...ex[2]].length, 240, "code points: a combining mark counts as one, as documented"); + assert.equal(ex[3], "short"); + assert.ok(ex.every(e => !e.includes("�"))); +}); + +test("reply link: a redacted quoted event keeps its id and author but no text; so does a non-text one", async () => { + const { made } = await linked([quoting("$r", "$red"), quoting("$n", "$nobody"), quoting("$m", "$img")], [ + target("$red", ALICE_ID, "the secret that was redacted", { content: {}, unsigned: { redacted_because: { type: "m.room.redaction" } } }), + target("$nobody", ALICE_ID, "", { content: {} }), + target("$img", ALICE_ID, "x", { type: "m.sticker", content: { body: "private caption" } }), + ]); + assert.deepEqual(made.map(d => d.replyTo), [ + { eventId: "$red", sender: ALICE_ID, excerpt: null }, + { eventId: "$nobody", sender: ALICE_ID, excerpt: null }, + { eventId: "$img", sender: ALICE_ID, excerpt: null }, + ]); + const redactedStillText = await linked([quoting("$r", "$red")], [target("$red", ALICE_ID, "still has a body", { unsigned: { redacted_because: {} } })]); + assert.equal(redactedStillText.made[0].replyTo.excerpt, null, "a redaction marker wins even if a body is present"); +}); + +test("reply link: a quoted event in a room she cannot read, an unreadable one and a thread fallback give no replyTo and no text", async () => { + const { made } = await linked( + [quoting("$other", "$foreign"), quoting("$gone", "$missing"), quoting("$th", "$alices", { rel_type: "m.thread", event_id: "$root", is_falling_back: true })], + [{ ...target("$foreign", ALICE_ID, "from another room"), room_id: "!elsewhere:gamecult.test" }, target("$alices", ALICE_ID, "fallback target")]); + assert.deepEqual(made.map(d => d.replyTo), [null, null, null]); + assert.ok(!JSON.stringify(made.map(d => d.replyTo)).includes("room") && made.every(d => d.replyTo === null), "no id, author or text of the foreign event travels"); +}); + +test("reply link: the same quoted event is read once, and a lookup cache hit from the basis check serves the quote", async () => { + const { made, hs } = await linked([quoting("$a", "$hers"), quoting("$b", "$hers")], [target("$hers", SELF, "mine")]); + assert.deepEqual(made.map(d => d.replyTo?.excerpt), ["mine", "mine"]); + assert.equal(hs.seen.filter(r => r.path.includes("/event/")).length, 1); +}); + +test("reply link: an edited target is quoted without its pre-edit text, whether it is the edit or has one bundled; id and author stay", async () => { + const { made } = await linked([quoting("$a", "$bundled"), quoting("$b", "$edit")], [ + target("$bundled", ALICE_ID, "old wording", { unsigned: { "m.relations": { "m.replace": { event_id: "$x" } } } }), + target("$edit", ALICE_ID, "* new wording", { content: { msgtype: "m.text", body: "* new wording", "m.relates_to": { rel_type: "m.replace", event_id: "$orig" } } }), + ]); + assert.deepEqual(made.map(d => d.replyTo), [{ eventId: "$bundled", sender: ALICE_ID, excerpt: null }, { eventId: "$edit", sender: ALICE_ID, excerpt: null }]); +}); + +test("reply link: a cached target is re-read after 60 s, so a redaction or edit in between shows; inside 60 s the cached read stands", async () => { + const clock = { t: 1_000_000 }; + const mine = target("$t", ALICE_ID, "before"); + const hs = await fakeHomeserver({ preload: { [ROOM]: [mine] }, sync: since => since === null ? { next_batch: "b0" } : { next_batch: `b${since}x`, ...joined({ [ROOM]: { timeline: { events: [quoting(`$q${since}`, "$t")] } } }) } }); + try { + const mouth = mouthFor(hs, { replyLink: true, now: () => clock.t }); + await mouth.syncOnce(); + const first = await mouth.syncOnce(); + mine.content = {}; mine.unsigned = { redacted_because: {} }; + clock.t += 59_000; + const within = await mouth.syncOnce(); + clock.t += 2_000; + const after = await mouth.syncOnce(); + assert.deepEqual([first, within, after].map(m => m[0].replyTo.excerpt), ["before", "before", null]); + } finally { await hs.close(); } +}); + +test("reply link: the target's own reply fallback is dropped before the cap; a body of only fallback has no excerpt", async () => { + const inReply = { "m.in_reply_to": { event_id: "$earlier" } }; + const withFallback = body => ({ content: { msgtype: "m.text", body, "m.relates_to": inReply } }); + const { made } = await linked([quoting("$a", "$f1"), quoting("$b", "$f2"), quoting("$c", "$f3"), quoting("$d", "$f4")], [ + target("$f1", ALICE_ID, "", withFallback("> <@bob:gamecult.test> what was it\n> second line\n\nthe real answer")), + target("$f2", ALICE_ID, "", withFallback("> <@bob:gamecult.test> only a quote\n\n")), + target("$f3", ALICE_ID, "", { content: { msgtype: "m.text", body: "> a literal quote with no reply\n\nkept" } }), + target("$f4", ALICE_ID, "", withFallback(`> <@bob:gamecult.test> ${"q".repeat(500)}\n\n${"w".repeat(300)}`)), + ]); + assert.deepEqual(made.map(d => d.replyTo.excerpt), ["the real answer", null, "> a literal quote with no reply kept", "w".repeat(240)]); +}); + +test("reply link: the excerpt is safe in a prompt: control characters, bidi marks and line breaks are gone before the cap", async () => { + const probe = "a\u0000b\u202ec\r\nd\u001b[31m"; + const { made } = await linked([quoting("$p", "$t1"), quoting("$q", "$t2"), quoting("$r", "$t3")], [ + target("$t1", ALICE_ID, probe), + target("$t2", ALICE_ID, " x\t\t y\n\n\nz\u200f\u2066w\u2069\u0085\u007f ok "), + target("$t3", ALICE_ID, `\u202e${"\u0000".repeat(500)}${"k".repeat(300)}`), + ]); + assert.deepEqual(made.map(d => d.replyTo.excerpt), ["abc d[31m", "x y zw ok", "k".repeat(240)], "controls removed first, so they do not eat the 240"); +}); + +test("reply link: a mouth built without the option has it off, and only the exact env value \"1\" turns it on", async () => { + const hs = await fakeHomeserver({ preload: { [ROOM]: [target("$t", ALICE_ID, "words")] }, sync: since => since === null ? { next_batch: "b0" } : { next_batch: "b1", ...joined({ [ROOM]: { timeline: { events: [quoting("$q", "$t")] } } }) } }); + try { + const mouth = createMouth({ persona: "eureka", homeserver: hs.url, rooms: Object.keys(ROOMS), token: TOKEN, state: memoryState(), fetch: hs.fetch, serverName: HS_NAME, brake: async () => true, log: () => {}, syncTimeoutMs: 1 }); + await mouth.syncOnce(); + assert.equal((await mouth.syncOnce())[0].replyTo, null); + } finally { await hs.close(); } + assert.deepEqual(["1", "true", "yes", "0", "", " 1", undefined].map(v => replyLinkFromEnv({ CHAT_PENDING_V4: v })), [true, false, false, false, false, false, false]); + assert.equal(replyLinkFromEnv({}), false); +}); diff --git a/tools/persona-chat-documents.mjs b/tools/persona-chat-documents.mjs index 7dc790d..f6e6210 100644 --- a/tools/persona-chat-documents.mjs +++ b/tools/persona-chat-documents.mjs @@ -11,6 +11,8 @@ export const kinds = Object.freeze({ pending: "gamecult.persona.chat_pending.v2", pendingRequestV3: "gamecult.persona.chat_pending_request.v2", // the same tuple as v1; asking by this kind is asking for the v3 answer pendingV3: "gamecult.persona.chat_pending.v3", + pendingRequestV4: "gamecult.persona.chat_pending_request.v3", // the same tuple again; served only when the mouth's reply-link flag is on, else an unknown kind + pendingV4: "gamecult.persona.chat_pending.v4", ack: "gamecult.persona.chat_ack.v1", read: "gamecult.persona.chat_read.v1", sendRequest: "gamecult.persona.chat_send_request.v1", @@ -39,7 +41,7 @@ const maxContext = 20; // Per-delivery bounds, applied where the mouth reads Matrix: the body is bounded like a send, // a context line and a display name are smaller, and a reply is paged to a byte budget. -export const limits = Object.freeze({ textBytes: maxTextBytes, contextBytes: 1000, displayBytes: 256, mentions: 64, replyBytes: 256 * 1024, tailPerRoom: 20, tailTotal: 60 }); +export const limits = Object.freeze({ textBytes: maxTextBytes, contextBytes: 1000, displayBytes: 256, mentions: 64, replyBytes: 256 * 1024, tailPerRoom: 20, tailTotal: 60, replyExcerpt: 240 }); // `value` cut to at most `maxBytes` of UTF-8 without splitting a character. export function clip(value, maxBytes) { @@ -215,6 +217,28 @@ function form(v) { return v; } +// The v4 delivery: the 14 fields of v3, then reply_to: nil, or [event_id, sender, excerpt] for the event this one replies to. +// sender is the quoted author's Matrix id. excerpt is nil for a quoted event whose text must not travel (redacted, not text); +// otherwise its first replyExcerpt (240) Unicode code points. A quote the mouth could not establish is reply_to nil, never a guess. +export function deliveryV4Tuple(d) { + return [...deliveryV3Tuple(d), replyTo(d.replyTo ?? null)]; +} +export function parseDeliveryV4(t) { + const fields = tuple(t, 15, "delivery_v4"); + return { ...parseDeliveryV3(fields.slice(0, 14)), replyTo: parseReplyTo(fields[14]) }; +} +function replyTo(r) { + if (r === null) return null; + const { eventId, sender, excerpt = null } = r; + if (excerpt !== null && (typeof excerpt !== "string" || [...excerpt].length > limits.replyExcerpt)) throw new Error(`reply_to excerpt must be a string of at most ${limits.replyExcerpt} code points, or nil`); + return [text(eventId, "reply_to event_id"), text(sender, "reply_to sender"), excerpt]; +} +function parseReplyTo(t) { + if (t === null) return null; + const [eventId, sender, excerpt] = tuple(t, 3, "reply_to"); + return { eventId: text(eventId, "reply_to event_id"), sender: text(sender, "reply_to sender"), excerpt: replyTo({ eventId, sender, excerpt })[2] }; +} + export function parseDelivery(t) { const [s, roomAlias, roomId, eventId, sender, senderDisplay, body, atUnixMs, context, mentions, isDirect, image] = tuple(t, 12, "delivery"); if (!Array.isArray(context)) throw new Error("delivery context must be a list"); @@ -263,6 +287,18 @@ export function parsePendingV3(t) { return { persona: text(persona, "persona"), deliveries: deliveries.map(parseDeliveryV3), tail: tail.map(parseTailLine), tailGaps: gaps }; } +// The v4 answer: the v3 answer with v4 deliveries. +export function pendingV4Tuple({ persona, deliveries, tail = [], tailGaps = [] }) { + const [p, , t, g] = pendingV3Tuple({ persona, deliveries: [], tail, tailGaps }); + return [p, deliveries.map(deliveryV4Tuple), t, g]; +} +export function parsePendingV4(t) { + const [persona, deliveries, tail, tailGaps] = tuple(t, 4, "chat_pending_v4"); + if (!Array.isArray(deliveries)) throw new Error("chat_pending deliveries must be a list"); + const rest = parsePendingV3([persona, [], tail, tailGaps]); + return { ...rest, deliveries: deliveries.map(parseDeliveryV4) }; +} + // The CultNet document_put_raw envelope every chat_* document travels in. export function envelope({ kind, recordKey, payload, sourceRuntimeId, sourceAgentId, sourceRole }) { return { diff --git a/tools/persona-matrix-mouth.mjs b/tools/persona-matrix-mouth.mjs index c9767c3..ffc042f 100644 --- a/tools/persona-matrix-mouth.mjs +++ b/tools/persona-matrix-mouth.mjs @@ -122,6 +122,7 @@ export function createMouth({ persona, homeserver, serverName, rooms, token, fetch: httpFetch = globalThis.fetch, state, brake, now = Date.now, log = () => {}, syncTimeoutMs = 30000, maxPending = MAX_PENDING, gate = createRoomGate(), sendTimeoutMs = SEND_TIMEOUT_MS, presenceGraceMs = PRESENCE_GRACE_MS, + replyLink = false, // resolve each reply's quoted event into delivery.replyTo (the v4 answer; one more event read per reply to a non-Mind) images = null, // this Persona's image store (createImageStore), or null: images stay the typed text line isServed = async () => false, // (async) whether a Matrix user id is an account this process speaks for: a Mind is not a person to open a DM with }) { @@ -310,22 +311,62 @@ export function createMouth({ } // A pill is an anchor whose href is exactly her matrix.to link; a longer id, an attribute that merely contains one and a quoted reply do not count. const pillFor = html => typeof html === "string" && [...html.replace(/[\s\S]*?<\/mx-reply>/gi, "").matchAll(/]*?href\s*=\s*(["'])(.*?)\1/gi)].some(([, , href]) => [selfId, encodeURIComponent(selfId)].some(id => href === `https://matrix.to/#/${id}`)); - const authors = new Map(); // `${room id} ${event id}` -> { sender, at }: read results, a failure kept for NEGATIVE_MS, bounded; the key holds the room so another room's reply cannot borrow a hit - const NEGATIVE_MS = 60 * 1000; - async function authorOf(roomId, eventId) { + const authors = new Map(); // `${room id} ${event id}` -> { sender, excerpt, at }: read results, kept TARGET_TTL_MS, bounded; the key holds the room so another room's reply cannot borrow a hit + const TARGET_TTL_MS = 60 * 1000; // how long a read of a target (hit or failure) is kept + // The quoted text, as a prompt-safe line: nil when the event is redacted (the homeserver strips its content), is not text, or is edited + // (an edit target, or one with an m.replace bundled: the original body is the pre-edit text, which its author replaced), or is nothing but + // its own reply fallback. Else: the legacy rich-reply fallback (leading "> " lines and the blank line after) is dropped, CR/LF/tab runs + // become one space, control characters (C0, DEL, C1) and bidi marks/overrides/isolates are removed, the result is trimmed, then cut to the + // first 240 code points (not bytes, so no character is split; a combining mark is its own code point) and trimmed again. Computed + // only when replyLink is on. + const FALLBACK_LINE = /^>( |$)/; + const excerptOf = target => { + const c = target?.content; + if (!replyLink || target?.unsigned?.redacted_because !== undefined || target?.type !== "m.room.message" || typeof c?.body !== "string") return null; + if (c["m.relates_to"]?.rel_type === "m.replace" || target.unsigned?.["m.relations"]?.["m.replace"] !== undefined) return null; + let body = c.body; + if (c["m.relates_to"]?.["m.in_reply_to"] !== undefined) { + const lines = body.split("\n"); + while (lines.length > 0 && FALLBACK_LINE.test(lines[0])) lines.shift(); + if (lines.length > 0 && lines[0] === "") lines.shift(); + body = lines.join("\n"); + } + const line = bodyText({ ...c, body }, docs.limits.contextBytes) + .replace(/[\r\n\t\u2028\u2029]+/g, " ") + .replace(/[\u0000-\u001f\u007f-\u009f\u200e\u200f\u202a-\u202e\u2066-\u2069]/g, "") + .trim(); + const cut = [...line].slice(0, docs.limits.replyExcerpt).join("").trimEnd(); + return cut === "" ? null : cut; + }; + async function targetOf(roomId, eventId) { const key = `${roomId} ${eventId}`; const known = authors.get(key); - if (known && (known.sender !== null || now() - known.at < NEGATIVE_MS)) return known.sender; - let sender = null; + if (known && now() - known.at < TARGET_TTL_MS) return known; // short for hits too: a later redaction or edit of the target must show at a re-read + let sender = null, excerpt = null; try { const target = await matrix("GET", `/rooms/${encodeURIComponent(roomId)}/event/${encodeURIComponent(eventId)}`, undefined, API, AbortSignal.timeout(sendTimeoutMs)); - if (target?.room_id === undefined || target.room_id === roomId) sender = typeof target?.sender === "string" ? target.sender : null; // an id from another room is not an answer about this one + if (target?.room_id === undefined || target.room_id === roomId) { // an id from another room is not an answer about this one: no sender, no text + sender = typeof target?.sender === "string" && target.sender !== "" ? target.sender : null; + if (sender !== null) excerpt = excerptOf(target); + } } catch (error) { say(`reply target ${eventId} not read: ${error.message}`); } - authors.set(key, { sender, at: now() }); + const entry = { sender, excerpt, at: now() }; + authors.set(key, entry); if (authors.size > DELIVERED_INDEX) authors.delete(authors.keys().next().value); - return sender; + return entry; + } + const authorOf = async (roomId, eventId) => (await targetOf(roomId, eventId)).sender; + // The event a message replies to: its own m.in_reply_to, unless that is a thread's fallback pointer (is_falling_back), which + // names no one. nil when the target cannot be read or sits in another room. + async function replyToOf(roomId, event) { + if (!replyLink || formOf(event.content) !== "message") return null; + const r = event.content["m.relates_to"]?.["m.in_reply_to"]; + const id = r?.event_id; + if (typeof id !== "string" || id === "" || event.content["m.relates_to"]["is_falling_back"] === true) return null; + const { sender, excerpt } = await targetOf(roomId, id); + return sender === null ? null : { eventId: id, sender, excerpt }; } async function repliesToMe(roomId, relates) { const targets = new Set([relates?.["m.in_reply_to"]?.event_id, relates?.rel_type === "m.thread" ? relates.event_id : undefined].filter(id => typeof id === "string" && id !== "")); @@ -402,12 +443,13 @@ export function createMouth({ if (refused !== null) say(`image refused: ${refused}`); const mentioned = event.content["m.mentions"]?.user_ids; const basis = await basisOf(roomId, event); + const replyTo = await replyToOf(roomId, event); made.push({ seq: ++seqCounter, roomAlias: alias, roomId, eventId: event.event_id, sender: event.sender, senderDisplay: displayNames.get(event.sender) ?? localpart(event.sender), text: bodyText(event.content, docs.limits.textBytes, refused), atUnixMs, context, image, mentions: Array.isArray(mentioned) ? mentioned.filter(id => typeof id === "string" && id !== "" && id.length <= 255).slice(0, docs.limits.mentions) : [], - isDirect: directRooms.has(roomId), basis, form: formOf(event.content), + isDirect: directRooms.has(roomId), basis, form: formOf(event.content), replyTo, }); if (image) images.hold(image.path, seqCounter); } @@ -730,15 +772,16 @@ export function connectionIdFor(connectionKey) { // The reply to a chat_pending_request: deliveries oldest first, each encoded on its own so one that // will not encode is skipped and logged without silencing the rest, up to the reply byte budget. // The caller acks what it has read and asks again. +// A v4 reply (v4 true, with v3) is the v3 reply with each delivery's reply_to counted in its size, like any field: one oversize delivery is still alone on its page. // A v3 reply also carries `tail` (room lines, oldest first), trimmed oldest first to what the byte budget has left after the deliveries, // and `tail_gaps`: the rooms of `tail.rooms`, in that order, with lines missing from the reply: those `tail.gaps` names plus every room // whose line the byte budget or an unencodable line dropped here. -export function pageOf(persona, deliveries, msgpack, log = () => {}, v3 = false, tail = { lines: [], gaps: [], rooms: [] }) { +export function pageOf(persona, deliveries, msgpack, log = () => {}, v3 = false, tail = { lines: [], gaps: [], rooms: [] }, v4 = false) { const page = []; let bytes = 0; for (const delivery of deliveries) { let tuple, size; - try { tuple = (v3 ? docs.deliveryV3Tuple : docs.deliveryTuple)(delivery); size = msgpack.encode(tuple).length; } catch (error) { + try { tuple = (v4 ? docs.deliveryV4Tuple : v3 ? docs.deliveryV3Tuple : docs.deliveryTuple)(delivery); size = msgpack.encode(tuple).length; } catch (error) { log(`delivery ${delivery?.seq} skipped: ${error?.message ?? error}`); continue; } @@ -769,7 +812,7 @@ export function pageOf(persona, deliveries, msgpack, log = () => {}, v3 = false, // session, and a request is answered on the session it arrived on. Requests arrive as // document_put_raw messages. A request naming a Persona this // process does not serve is answered with a typed UnknownPersona refusal. -export async function startChatEndpoint({ mouths, runtime, port, connectionKey, runtimeId, socketFactory = createSocket, host = "127.0.0.1", log = () => {} }) { +export async function startChatEndpoint({ mouths, runtime, port, connectionKey, runtimeId, socketFactory = createSocket, host = "127.0.0.1", log = () => {}, replyLink = false }) { if (!/^127\.\d+\.\d+\.\d+$/.test(host)) throw new Error("the chat endpoint listens on loopback only"); const { msgpack, cultnet } = runtime; const socket = socketFactory("udp4"); @@ -781,7 +824,7 @@ export async function startChatEndpoint({ mouths, runtime, port, connectionKey, const respond = (peer, persona, kind, recordKey, tuple) => server.send(peer, "schema", msgpack.encode(cultnet.encodeCultNetMessageForWire( docs.envelope({ kind, recordKey, payload: msgpack.encode(tuple), sourceRuntimeId: runtimeId, sourceAgentId: persona, sourceRole: "bifrost-persona-matrix-mouth" }), "cultnet.schema.v0"))); const refuseUnknown = (peer, persona) => respond(peer, persona, docs.kinds.refusal, persona, docs.refusalTuple({ persona, refusal: "UnknownPersona" })); - const served = [docs.kinds.pendingRequest, docs.kinds.pendingRequestV3, docs.kinds.ack, docs.kinds.read, docs.kinds.sendRequest, docs.kinds.sendRequestV2, docs.kinds.openDirectRequest]; + const served = [docs.kinds.pendingRequest, docs.kinds.pendingRequestV3, ...(replyLink ? [docs.kinds.pendingRequestV4] : []), docs.kinds.ack, docs.kinds.read, docs.kinds.sendRequest, docs.kinds.sendRequestV2, docs.kinds.openDirectRequest]; const handle = async ({ peer, channelId, payload }) => { try { if (channelId !== "schema") throw new Error("chat request arrived on a foreign channel"); @@ -798,13 +841,14 @@ export async function startChatEndpoint({ mouths, runtime, port, connectionKey, const mouth = mouths.get(request.persona); const receipt = mouth ? await mouth.openDirect(request) : { requestId: request.requestId, outcome: "UnknownPersona", roomId: null, eventId: null }; respond(peer, request.persona, docs.kinds.openDirectReceipt, receipt.requestId, docs.openDirectReceiptTuple(receipt)); - } else if (document.schemaId === docs.kinds.pendingRequest || document.schemaId === docs.kinds.pendingRequestV3) { - const v3 = document.schemaId === docs.kinds.pendingRequestV3; // the kind asked by names the answer: v1 gets the 12-field v2 answer, as before + } else if (document.schemaId === docs.kinds.pendingRequest || document.schemaId === docs.kinds.pendingRequestV3 || document.schemaId === docs.kinds.pendingRequestV4) { + const v4 = document.schemaId === docs.kinds.pendingRequestV4; // only reachable with replyLink: `served` drops it otherwise + const v3 = v4 || document.schemaId === docs.kinds.pendingRequestV3; // the kind asked by names the answer: v1 gets the 12-field v2 answer, as before const request = docs.parsePendingRequest(tuple); const mouth = mouths.get(request.persona); if (!mouth) return refuseUnknown(peer, request.persona); const tail = v3 ? await mouth.tail().catch(error => { log(`tail not read: ${error?.message ?? error}`); const rooms = mouth.speakableRooms(); return { lines: [], gaps: rooms, rooms }; }) : undefined; - respond(peer, request.persona, v3 ? docs.kinds.pendingV3 : docs.kinds.pending, request.persona, pageOf(request.persona, mouth.pendingAfter(request.afterSeq), msgpack, log, v3, tail)); + respond(peer, request.persona, v4 ? docs.kinds.pendingV4 : v3 ? docs.kinds.pendingV3 : docs.kinds.pending, request.persona, pageOf(request.persona, mouth.pendingAfter(request.afterSeq), msgpack, log, v3, tail, v4)); } else if (document.schemaId === docs.kinds.ack) { const request = docs.parseAck(tuple); const mouth = mouths.get(request.persona); @@ -856,6 +900,9 @@ export function loadRuntime(cultRoot) { return { CultCache: cc.CultCache, SingleFileMessagePackBackingStore: cc.SingleFileMessagePackBackingStore, defineDocumentType: cc.defineDocumentType, readSingleFileStore: cc.readSingleFileStore, cultnet: cn, msgpack: mp }; } +// The v4 flag: on only for exactly "1"; unset, "0", "true", "yes" and anything else are off. +export const replyLinkFromEnv = env => env.CHAT_PENDING_V4 === "1"; + function required(env, name) { if (!env[name]) throw new Error(`${name} is required`); return env[name]; @@ -903,10 +950,11 @@ export async function main(env = process.env) { const controlRoot = env.CONTROL_ROOT ?? "/etc/gamecult/minds"; const imageRoot = env.IMAGE_STORE_ROOT ? resolve(env.IMAGE_STORE_ROOT) : null; // unset: chat images stay the typed text line const gate = createRoomGate(); // one for every Persona: two Minds in one room take turns + const replyLink = replyLinkFromEnv(env); // default off: Huginn pins the v3 tuple bytes until it reads v4 const mouths = new Map(personas.map(persona => { mkdirSync(join(stateRoot, persona), { recursive: true }); return [persona, createMouth({ - persona, homeserver, serverName, rooms: roomsByPersona.get(persona), token: tokens.get(persona), gate, isServed: async id => { await Promise.all([...mouths.values()].map(m => m.start().catch(() => {}))); return [...mouths.values()].some(m => m.userId === id); }, log: line => log(`${persona}: ${line}`), + persona, homeserver, serverName, rooms: roomsByPersona.get(persona), token: tokens.get(persona), gate, replyLink, isServed: async id => { await Promise.all([...mouths.values()].map(m => m.start().catch(() => {}))); return [...mouths.values()].some(m => m.userId === id); }, log: line => log(`${persona}: ${line}`), images: imageRoot ? createImageStore({ root: join(imageRoot, persona), log: line => log(`${persona}: ${line}`) }) : null, state: cultcacheState({ runtime, path: join(stateRoot, persona, "cursor.cc") }), brake: createBrakeReader({ controlPath: join(controlRoot, persona, "control.cc"), readStore: runtime.readSingleFileStore, decode: runtime.msgpack.decode }), @@ -914,7 +962,7 @@ export async function main(env = process.env) { })); const listen = required(env, "CHAT_RUDP_LISTEN"); // host:port, loopback const colon = listen.lastIndexOf(":"); - await startChatEndpoint({ mouths, runtime, host: listen.slice(0, colon), port: Number(listen.slice(colon + 1)), connectionKey: chatKey, runtimeId: "bifrost-persona-mouth", log }); + await startChatEndpoint({ mouths, runtime, host: listen.slice(0, colon), port: Number(listen.slice(colon + 1)), connectionKey: chatKey, runtimeId: "bifrost-persona-mouth", log, replyLink }); await Promise.all([...mouths.values()].map(mouth => mouth.run())); }); }