GameCult/Huginn
Watch
9
0
Fork
You've already forked Huginn
 0
Code Issues Pull requests 1 Projects Releases Packages Wiki Activity Actions

chat ingest core: room scope, credential scrub, chunker, refresh_chat, steward purge #60

Merged
eureka merged 6 commits from gamecult-chat/chat-ingest-core into main 2026-11-11 20:53:00 +00:00
Conversation 0 Commits 6 Files changed 16 +2840 -192
eureka commented 2026-11-11 19:34:36 +00:00
Owner
Copy link

Cut gamecult-chat:cut_spec:cut-chat-ingest-core.r2, on chat-search's merged head 421ace5. Nothing here talks to Matrix; the HTTP client, flags, chat-purge subcommand and unit are the next cut (chat-ingest-job). Rebased on 36dd8ce and f1d5c40 (conflicts in sources/mod.rs, index/qdrant.rs, index/fakes.rs, tests.rs resolved keeping both cuts). Soul's first pass failed; fixes F1-F7 are in the last commit.

What lands:

  • huginn_mind::redact: redact_url_userinfo MOVED from mind-body (one definition, tests moved with it), plus scrub() / ScrubKind / SCRUB_VERSION (PEM, authorization line, Bearer/Basic/token, vendor key shapes, JWT, URL userinfo). Hand-written scanners, no new dependency.
  • sources/room_scope.rs: admit_room, refuse_event, ESTATE_ALIASES, Estate; the only file that spells the bridged platform. RoomFacts has no join-rule field.
  • sources/chat_chunk.rs: greedy chunker, chat_version(), chunk_id, cut_body, embedded_text.
  • sources/chat.rs: RoomHistory port, refresh_chat, retire_chat, cursor keys, ChatRefreshed; the fold (purge filter, scrub, edit, redaction), write-then-delete and cursor-last.
  • sources/chat_purge.rs: windows, tombstones, request_purge; applied by refresh_chat.
  • SourceIndex: chat_tail, chat_chunks_holding, chat_chunks_overlapping, delete_chat (Qdrant and fake impls).
  • Forge retire no longer reads chat: / chat-purge: keys as repo names (test written first, red at base).

Verification (docker, digest-pinned rust image, --cpus 4 --memory 12g): cargo test --locked -p huginn-mind -p huginn-daemon -p mind-body -p eureka-state green (mind-body needs root-owned tmp dirs, so it ran as root; huginn-daemon's desk-door tests refuse root, so those ran as a normal user), cargo check --locked --workspace exit 0.

Deviations: request_purge has no now_ms (F7: nothing in it reads a clock; a request is a window, and the journal's ref is the job's); ChatMsg is the payload's own ChatMessage; refused carries a RefusedRoom struct.

Soul fixes: unknown-redaction rebuild and in-page edit revert, edits of redacted or unknown originals dropped (docs/chat-retrieval-map.md names both rules); scrub gains Matrix tokens, access_token values, PGP blocks, bearer:token, glued distinctive keys, zero-width stripping (SCRUB_VERSION 2); the room name is scrubbed; RoomFacts has no Default, Estate::new demands exactly the three ids, bridge sender patterns widened; a facts error keeps the room and is reported in its ChatReport; retire_chat goes on past failures; the tail is deterministic on equal ends. Removed as redundant owners: the in-fold redacted guard and the refused-room purge in refresh_chat (retirement purges it).

Not in this PR: how the Matrix port maps a pruned event (content: {}) to EventKind::Other must be pinned by the ingest-job cut.

🤖 Generated with Claude Code

Cut `gamecult-chat:cut_spec:cut-chat-ingest-core.r2`, on chat-search's merged head 421ace5. Nothing here talks to Matrix; the HTTP client, flags, `chat-purge` subcommand and unit are the next cut (chat-ingest-job). Rebased on 36dd8ce and f1d5c40 (conflicts in sources/mod.rs, index/qdrant.rs, index/fakes.rs, tests.rs resolved keeping both cuts). Soul's first pass failed; fixes F1-F7 are in the last commit. What lands: - `huginn_mind::redact`: `redact_url_userinfo` MOVED from mind-body (one definition, tests moved with it), plus `scrub()` / `ScrubKind` / `SCRUB_VERSION` (PEM, authorization line, Bearer/Basic/token, vendor key shapes, JWT, URL userinfo). Hand-written scanners, no new dependency. - `sources/room_scope.rs`: `admit_room`, `refuse_event`, `ESTATE_ALIASES`, `Estate`; the only file that spells the bridged platform. `RoomFacts` has no join-rule field. - `sources/chat_chunk.rs`: greedy chunker, `chat_version()`, `chunk_id`, `cut_body`, `embedded_text`. - `sources/chat.rs`: `RoomHistory` port, `refresh_chat`, `retire_chat`, cursor keys, `ChatRefreshed`; the fold (purge filter, scrub, edit, redaction), write-then-delete and cursor-last. - `sources/chat_purge.rs`: windows, tombstones, `request_purge`; applied by `refresh_chat`. - `SourceIndex`: `chat_tail`, `chat_chunks_holding`, `chat_chunks_overlapping`, `delete_chat` (Qdrant and fake impls). - Forge `retire` no longer reads `chat:` / `chat-purge:` keys as repo names (test written first, red at base). Verification (docker, digest-pinned rust image, --cpus 4 --memory 12g): `cargo test --locked -p huginn-mind -p huginn-daemon -p mind-body -p eureka-state` green (mind-body needs root-owned tmp dirs, so it ran as root; huginn-daemon's desk-door tests refuse root, so those ran as a normal user), `cargo check --locked --workspace` exit 0. Deviations: `request_purge` has no `now_ms` (F7: nothing in it reads a clock; a request is a window, and the journal's ref is the job's); `ChatMsg` is the payload's own `ChatMessage`; `refused` carries a `RefusedRoom` struct. Soul fixes: unknown-redaction rebuild and in-page edit revert, edits of redacted or unknown originals dropped (docs/chat-retrieval-map.md names both rules); scrub gains Matrix tokens, access_token values, PGP blocks, `bearer:token`, glued distinctive keys, zero-width stripping (SCRUB_VERSION 2); the room name is scrubbed; `RoomFacts` has no `Default`, `Estate::new` demands exactly the three ids, bridge sender patterns widened; a facts error keeps the room and is reported in its `ChatReport`; `retire_chat` goes on past failures; the tail is deterministic on equal ends. Removed as redundant owners: the in-fold `redacted` guard and the refused-room purge in `refresh_chat` (retirement purges it). Not in this PR: how the Matrix port maps a pruned event (`content: {}`) to `EventKind::Other` must be pinned by the ingest-job cut. 🤖 Generated with [Claude Code](https://claude.com/claude-code) 
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
eureka force-pushed gamecult-chat/chat-ingest-core from 6175f57440 to 979b7307ca 2026-11-11 20:28:07 +00:00 Compare
eureka merged commit b4f3a48a8f into main 2026-11-11 20:53:00 +00:00
Sign in to join this conversation.
Reviewers
No reviewers
Labels
Clear labels
No items
No labels
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
GameCult/Huginn!60
No description provided.